The follow diagram outlines a design that uses 6 NICs, makes use of logically isolated networking and does not include Fault Tolerance.
Uplink ports should all be configured to use PortFast.
Both the storage uplink ports and the virtual switch used for storage should be configured to use Jumbo Frames by specifying an MTU of 9000.
In this design trunking should be configured on all uplinks. Trunking at the physical switch will enable the definition of multiple allowable VLANs at the virtual switch layer.
The host design presumes that vmnic0 and vmnic1 are inbuilt and vmnic2 through vmnic5 are using an expansion card. The Management and Storage switches have an uplink from both an internal and expansion slot based card. In the event of the internal card failing the host and VM's will continue to function, in the event of the expansion card failing all VMs will lose network connectivity, but will retain host and storage connectivity.
* 1 - Distributed virtual switches require Enterprise Plus licenses. If you do not have Ent+ then replace these switches with standard virtual switches.
* 2 - Route based on physical NIC load is a policy only available in a dvSwitch. If you do not have Ent+ then use the default policy instead.
* 3 - VLAN tagging is absolutely required in this design in order to apply a secure segmentation of storage traffic from the rest of the environment.
* 4 - VLAN 30 is reserved in case Fault Tolerance is ever required in the design, hence VLAN 40 is used for storage.
* 5 - Datastore path selection policy is typically best set at Round Robin, but always consult the storage vendor documentation to confirm the best type to use.
vSphere 5 - 6 NIC SegmentedStorage and NoFT Design v1.1.jpg
I welcome any questions or feedback. If you would like the original Visio document please contact me and I will email these to you.