I’m preparing a VI3 environment with 2 IBM x3650 both with 4 pNIC (2 onboard + 2 with expansion card).
I’ll have 6 VM (actually) and one of them is a mail server which has to be on DMZ.
What configuration do you suggest? I’ve read a lot of docz and posts here and there and I’m ended with this possibility (due to the low pNIC availability in this setup):

VSWITCH0 (2 pNIC – vmnic0 & vmnic2):
Port Group1: Service Console (vmnic0 Active – vmnic2 Standby)
Port Group2: VMotion Network (vmnic2 Active – vmnic0 Standby)

VSWITCH1 (2 pNIC – vmnic1 & vmnic3):
Port Group1: VM Production Network (tagged with a specific VLAN ID)
Port Group2: VM DMZ Network (tagged with a specific VLAN ID)

I will then be able to offer Service Console redundancy through a standby pNIC used actually by the VMotion Network and vice-versa.
Having only 2 spare pNICs and needing some fault-tolerance (and trunking) for VM Networks I’ll have to put Production and DMZ VMs on the same VSWITCH separating them on 2 different Port Groups and using different subnets or VLANs (which is better?).

Is this safe? I know that a single pNIC only for DMZ would be a better solution but this will gave me no pNIC redundancy for both VM Networks.

Please give me any advice or suggestions on better design.

The problem has been solved by the following steps

1. Disabled the Trunking ports in the pSwitches
2. In the pSwitch I have set the VLAN3 and VLAN4 member of each other.
3. Set the VLAN ID in the Service Console and VMkernel Portgroups.
4. In the pSwitch, I have set the Tagging to TagAll for both ports 5 & 6 in both pSwitches.
5. PVID are the same on both ports on both pSwitches.
6. For the second host, did the same steps entering the VLAN ID in the correspondence portgroup.

Result:

* Both hosts they can reach other.
* Both hosts they can reach the configured Default Gateway.
* Both hosts they can reach the VMkernel IP Address of each host using vmkping command.
* Test vmnic0 failure = Still I can reach other host IP Address & hostname, and the Default Gateway as well as the VMkernel IP Address of other host in the Cluster
* Test vmnic1 failure = Still I can reach other host IP Address & hostname, and the Default Gateway as well as the VMkernel IP Address of other host in the Cluster.

Now, when I tried to re-configure the VMware HA, I got an error while I configuring the VMware HA, I got an errors:

HA agent on hosts.esx.local in cluster ESX in Development Network has an error: No Active Primaries Found “Means no Primary Host found in the Cluster”
CMD startagent failed: Internal AAM Error – Agent could not start

Steps followed to overcome this problem:

1. Put the hosts in Maintenance Mode.
2. Disconnected the hosts from the Virtual Center.
3. Removed the hosts from the Virtual Center
4. Restarted the Virtual Center, just in case to get everything cleared.
5. Added both hosts to the Cluster, and enabled VMware HA. WAW, everything works as expected successfully.

Just to let you get more clear picture about what I have done;

pSwitch1, Port 5 = VLAN3
pSwitch1, Port 6 = VLAN4
IP Address: 172.16.20.0/ 24 D.Gateway: 172.16.20.1
pSwicth2, Port 5 = VLAN3
pSwitch2, Port 6 = VLAN4
IP Address: 10.1.0.0/24 D.G: 10.1.0.1

I have trunked those ports in the pSwitch:
Trunk Name: ESX
Unit: Port:
1 5
1 6
2 5
2 6

I have set the both ports PVID equally, and selected both ports as member of each other VLANs. The Trunk is successful enabled.

I have set a VLAN ID in the Service Console and VMotion Portgroups, and set pSwitch Ports to TagAll and UnTag. But, I’m unable to get any Connectivity to the Service Console nor the VMkernel between each host.

I have disabled the Trunk, and set the VLAN Ports to be a member of each other VLAN and set the Port 5 “Where the vmnic0 Connected as TagAll”. I got the connectivity back to the hosts, and VMkernel can ping each other. But vmkping cannot reach to it’s Default Gateway.

Any further help?

Thanks,

The VMotion network doesn’t need to be a routable network. It can be a completely isolated network on a seperate switch without an uplink to the Service Console network, as long as all ESX Hosts VMkernel can “vmkping” eachother. So that’s what you should test! Not a regular ping.

Start with this and then let me know the results.

I want to configure my ESX Servers to works with VLAN under Nortel Switches 4542 GT in Stack-mode.

Server Configuration:
2- DL380 G5, each with Single Port HBA, 6 pNICs, 2 pCPU Dual-Proc.
2- BL460 G1, Each with Dual-port HBA, 6 pNICs, 1 pCPU Dual-Proc.

Setup:
vSwitch0 = ESX Networks: Service Console “172.16.20.0/24″ && VMotion “10.1.0.0/24″ using VLANs.
vSwitch1= Production Network: 128.104.0.0/16
vSwitch2 = DMZ Network: 192.168.1.0/24

Private Network for ESX:
vSwitch0 with 2 pNICs connected vmnic0 & vmnic1 Teamed on the vSwitch Level.
2 Portsgroup.
1 Service Console
1 VMotion

In the portgroup Setting for S.C –> Nic Teaming is vmnic0 Active and vmnic1 Standby
In the portgroup setting for VMotion –> Nic Teaming is the vmnic1 Active and vmnic0 Standby.

vmnic0 connected to pSwitch on port configured with VLAN 2
vmnic1 connected to pSwitch on port configured with VLAN 3

Production Network:

vSwitch1 with 2 pNICs connected vmnic2 & vmnic3 Teamed on the vSwitch Level.
1 Portgroup.
Production VMs

vSwitch2 with 2 pNICs connected vmnic4 & vmnic5 Teamed on the vSwitch Level.
1 Portgroup
DMZ VMs

================================================== =============================

If
I assign an IP Address to the S.C with the same IP which is configured
on the VLAN, “Without Assigning the ((VLAN ID)) in the portgourp,
through pServer, i can reach to other ESX Host Service Console, because
both of them are on the same VLAN.

As soon as I assign the
((VLAN ID)) on the portgroup of S.C, i lost the connectivity to the
server, and I started troubleshooting the vswif0 to create another
Service Console Network in order to access it the ESX Host. “And the
same applies on the VMotion Network”.

The same goes to the
VMotion network as well. From the pSwitch, both the VLANs are reachable
to 172.16.20.0/24 Service Console, and 10.1.0.0/24 for VMotion Network.

I want the Service Console Network, can talk to the VMotion Network and vice versa to get the VMotion works.

Service Console:
IP:172.16.20.2/24
D.G: 172.16.20.1
DNS: 172.16.20.57 “This host is connected to the same VLAN where the ESX hosts connected”. It’s a VC and DNS Server.

VMkarnal:
IP:10.1.0.2/24
D.G: 10.1.0.1

From,
within the ESX Host, I’m unable to reach to the Default Gateway of the
VMotion Network using vmkping. Nor the Service Console able to reach to
the VMotion Network.

Moreover, I wanted to reach to the Service
Console Network 172.16.20.0 via 128.104.0.0 Network to do my
Administrative Task. In this case, do I have to add a Static Route in
the Service Console, in order for the VI Clients reach from Production
Network?

Further Testing:

have UnTagged the ports for both VLANs. I setup both PortGroups S.C & VMKernel without VLAN ID.
I
got one host can ping VMkernel PortGroup on another host via COS ping.
Also from the same host tried vmkping S.C IP and D.Gateway. It’s
successul.

However, from the another host I can reach the
first host S.C IP but not VMkernel. Nor the VMKernel able to reach it’s
D.Gateway.

Since the both VLANs are reachable within the pSwitch. Do I have to use a port Trunking, and assign different VLAN ID “The Trunked vLAN” in each Portgroup *S.C & VMotion*?

In additional to what i have mentioned to earlier regarding the NIC Teaming.
Both, vmnic0 & vmnic1 assigned to vSwitch0, in the NIC Teaming
Setting of vSwitch0, both vmnic0 & vmnic1 as Active/Active. And
within each PortGroup, S.C = vmnic0 Active & vmnic1 Standby. And
VMkernel = vmnic1 Active & vmnic0 Standby. Is this Setting may
confusing the VLAN to work properly?

Furter troubleshooting I’m going to make;
1. Remove the Nic Teaming from the PortGroups.
2. Configure the vSwitch0 with only vmnic0 on both hosts, assuming a pNIC failure.
3. Test both hosts can ping each other S.C and D.Gateway 172.16.20.1
4. Cofigure VMkernel with the prospetive VLAN IP schema & test vmkping whether it can reach S.C IP & it’s D.Gateway.
5. If it’s success, then will configure the same on the other host and test the connectivity between the hosts.

If not, do I have to configure a Trunking on the pSwitches and make both VLANs 3 & 4 members of the Trunked VLAN?

Further Testing:

In the pSwitch on port 5 where the vmnic0 connected, i have set the port ot TagAll.

Result:
1. I lost the connectivity to the vswif0 “Service Console IP”. But within this vSwitch0, I have VM Network Portgroup, and one of the Virtual Machines IP’s is set to the same VLAN of the Service Console, it’s reachable

This without VLAN ID specified on any PortGroup.

2. While the the pSwitch Port 5 is set to TagAll, I have specify a VLAN ID to both the Service Console & VM Network Portgroups, I got the connectivity back up on the Service Console as well as on the VM Network.

Now I have the vmnic0 connected to port 5 “VLAN 3 – ip: 172.16.20.0″ on the pSwitch and vmnic0 linked to vSwitch0. Also, vmnic1 is connected to port 6 “VLAN 4 – ip: 10.1.0.0″ on the pSwitch and vmnic1 is Linked to vSwitch0 as well.

Question: How to get the VMotion works since Service Console is setting on the different Network and VMotion on Different Network?

Do I have to specify a Static Route in the ESX Server in order for the VMKernel Network sees the Service Console Network?

Thanks,

I was going to revise the document anyway, might do it tomorrow if I can find the time.