Yellow Bricks

by Duncan Epping

Creating a nested lab

· ·

I was just building a nested lab to record some demo videos. I find myself googling for this every single time so I figured I would write about it so I can easily get it of my own website. Many have written about this before and all credits go to William Lam and Eric Gray, which are the two  main blogs I have used in the past to get this working.

After installing ESXi on my physical box I “ssh” in to it. In order to allow “nested ESXi” to boot a 64bit OS you will need to run the following:

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

After you have done that you will want to make sure you will get network connection. Go to your “VM Network” portgroup, or if you named it differently the portgroup that is used to connect the virtual ESXi hosts to. For each of the hosts do the following:

  1. Click on the host
  2. Go to “Configuration”
  3. Click on “Networking”
  4. Click “Properties” on the vSwitch
  5. Select the correct portgroup
  6. Click “Edit”
  7. Click “Security”
  8. Set “Promiscuous Mode” to “Accept”
  9. Click “Ok”
  10. Click “Close”

Now for each virtual ESXi host (note there is a “guest os” called ESXi 5 in there, use it!) that you have created do the following:

  1. Right click on the VM
  2. Click “Edit settings”
  3. Click the “Options” tab
  4. Click on “CPU/MMU virtualization”
  5. Select the 4th option “Use Intel VT-x / AMD-v…”

I am building this out to record a new of “DR of the Cloud”. In other words, 3 virtual clusters + vCloud Director + SRM + vSphere Replication + Virtual Storage Appliances… Cool stuff right.

Reader Interactions

Comments

  2. Hi Duncan,

    I build nested ESXi all the time too, for lab work / POC testing etc.

    Not sure if you’ve heard of it, but the vBrownBag crew (particularly Alastair Cooke) have been working on automating this lab build / setup. I thought your readers might also be interested in rapidly producing a functioning nested VMware vSphere 5 lab with shared storage, vMotion, HA, DRS etc.
    It was mentioned on the VMwsre community roundtable podcast a few weeks back too.

    Check it out at http://www.labguides.com/autolab

    Cheers,
    Nick Marshall

    • HI Mate, I am new in VM world and got stuck in a bit funny issue. I am trying the nested host configuration. My host are not talking to each other even after enabling “Promiscuous mode” on Vswitch0. not sure if I have to do something else to get them communicate with each other.

  3. Strange strange problem. Boot up and after it scans for devices it doesn’t find any volumes to install on, yet the guest has an 8 GB thick vmdk. I have tried different sizes, different provisioning methods, different datastores, and adding a disk after the fact.

    Then I tried accidentally initially created the VM as Other (64 bit). And there it was. Only if you initially create the VM as RedHat 6 64bit do you get this behavior.

    • The controller type will be different depending on what initial guest OS you select and most likely the one you selected is not supported by ESX

  4. Hi Duncan,

    This is indeed cool stuff 🙂 Can you please share your physical server specs?

    Thanks,
    David Dominguez

  5. I would also be interested in your lab build. There are so many options on the HCL it’s hard to decide these days.

  6. This is not a regular box. I am running nested on a proper ESXi Cluster with 4 DL380 G7 hosts. The reason I am doing that is because I can simply snapshot all and revert to before me messing things up.

  7. i can say that a nested lab is working fine here. Been using it for some months now because ive only 1 powerfull server. (dual Xeon e5-2680 64GiB = nested 4x QC 16gb ESX cluster)

    Used the same source blogs as Duncan for building nested environment, only im using it 24/7.

    Perhaps a small tip: use a opensource/free software NAS/SAN for sharing storage to your nested ESX cluster.

    Grtz!

  8. Do you know of a way to automate the updating of /etc/vmware/config in an auto deploy environment? We run our entire lab using auto deploy and vcloud director but this is one file we have yet to be able to figure out how to properly update to make vsphere on vsphere work.

    Ed.

  9. I have been using 2 custom built x64 boxes for my home lab.

    Duncan – The nested ESXi VM still can’t run x64 VMs right?

  11. I was under the impression that AMD-V or Intel-VT supported only 1 level of virtualization.

    If I understand it correctly, you have installed ESXi on a HP DL380 G7 hardware. Within that you have created a VM with ESXi and still could run 64-bit VMs within the nested ESXi.

  14. Any advice on getting snapshots of virtual ESXi guests? There was a fix posted by Eric Grey for ESXi 4.1 physical hosts, but the command doesn’t work for physical ESXi 5 hosts hosting virtual ESXi guests. The reason I need this is because I use VDR to backup, which of course takes snapshots. This essentially destroys the filesystem of the ESXi guest on reboot and you need to reinstall.

  15. What about the parameters

    monitor_control.restrict_backdoor = TRUE
    monitor_control.vt32 = TRUE

    Are they still useful ?

  17. Hey duncan, i like you blog very much,
    I have a question on how to make a lam for srm with only
    1 storage array of nettap. Did you try this?
    Thank you in advanced

  19. Ok so lets break it down,
    2 ESXi 5 connected to NetApp Storage.
    2 vCenters , 1 HQ , 1DR connected via Link Mode.

    i want to Install SRM 5 but only use 1 NetApp because i don’t really have a DR.

    everything is in 1 site with only one Storage Array, but i want somehow to simulate it .
    because if i install the SRA on VC 1 and VC 2 they will both see the same NetApp .

    Did you write maybe a Manual regarding this kind of Labs ?

  20. As an update to my query about how to set vhv.allow when utilizing auto deploy. I ended up solving this problem by developing a custom vib and offline bundle that can be added as a software depot and subsequently added into an image profile that can be used on a host. This is working suprisingly well and could also, theoretically, be used with update manager or on a host directly to ensure that even through updates the setting is reset. I have more info about the bundle and the download on my blog:
    http://blog.edmclain.com/2012/07/enabling-vhv-allow-while-utilizing-esxi-on-vsphere-auto-deploy/

    I have also written another bundle that enables the copy and paste feature for the vmrc and vsphere client that follows the same methodology. I hope somebody finds it useful.

    Ed.

  21. Could you explain why promiscuous mode needs to be enabled? It seems ARP resolution on host’s vSwitch fails for the port group without promiscuous mode.
    Why a regular VM does not need promiscuous mode?

  22. When I type:
    cat /etc/vmware/config
    I can see the line vhv.allow = “TRUE”
    Still the ESXi doesnt show up in the Guest operating system version drop down menu.
    This is for esxi 5 host under wmworkstation as on esxi on an HP DL360
    Is there anything else I should look for? I’m getting desperate since everyone gets it to work, and I fail on two machines…

    • Chris,
      I am having the same problem. ESXi does not show up in the Guest operating system version drop down menu. I am using ESXi5 trial license. Please let me know if you find workaround for it.
      Thank you in advance and appreciate your help.

  23. I created ESXi 5.0 under that i created VM’s 2 ESXi4.1 and 1 windows 2008 server r2.In win2k8 i nstalled Vcenter5.0 and add that 2 ESXi4.1 servers.

    While i am creating win2k8 64 it throwing error.

    An unexpected error was received from the ESX host while powering on VM vm-62.
    The processor does not support a required feature for this virtual machine. The virtual machine cannot be powered on

  24. Hi all – I follow the outline as specified by Duncan and William LAN and my virtual ESXi host comes up fine. However, it does not have visibility to the local ATA disks and as such I cannot perform vMotion. Anyone seen this problem and know the resolution? Require modification to the .vmx file perhaps?

    Any help appreciated,

  25. When i power on the 64bit VM. i am getting the following error message.
    Virtual machine message.
    This virtual machine is configured for 64bit guest operating system, however, 64-bit opetation is not possible.
    Longmode is disabled for this virtual machine.

  26. I have Hp Pavilion DV6 with 64bit , Intel Duel Core, 2.5 Mhz, 350 GB. Can I build Cisco Data Centric Nexus labs and using VMware tools?

    Do I need to buy another expensi i3 processor, 500GB, laptop?

    Kindly reply your answer. Thanks in advance.

  27. Hello there. I did want to add one comment to this chain.

    Adding ‘vhv.allow = “TRUE”‘ works if your physical host is ESX 5.0.
    Adding ‘vhv.enable = “TRUE”‘ is needed if your physical host is ESX 5.1.

    This held me up for several hours, so I hope this helps others out there.

  29. Important to note is that this does not work on AMD Bulldozer processors.
    I have tried both ‘vhv.allow = “TRUE”‘ and ‘vhv.enable = “TRUE”‘ on ESXi 5.0.0 1024429
    without success of getting the Guest OS option “ESXi”

    ESXi 5.0

    On ESXi 5.0, virtualized HV is prohibited by default. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. It is available on hosts that support Intel VT-x or AMD-V, but it is not recommended for systems without second level address translation (EPT or RVI), because of its poor performance without SLAT. Unfortunately, this feature does not work on AMD “Bulldozer” CPUs running ESXi 5.0.

    http://communities.vmware.com/docs/DOC-8970

    ESX install only renders “HARDWARE_VIRTUALIZATION WARNING: Hardware Virtulization is not a feature of the CPU, or is not enabled in the BIOS”. ie the cpu doesn’t support it at all. =(

    • I stand corrected (by myself ?)

      I managed to set the guest os to ESXi 5.x, but only after I initially created the machine and enterd
      “Edit virtual machine settings…”
      “Options/General Options”
      and then looked under “Guest operation system / other”. Then it’s visible.

  31. Building vCloud nested home lab on laptop like trying to sequence power on Apollo 13 return flight.

    What’s the sizing formula? for 16GB of memory. Gah!!

  32. Hey Duncan,

    I got ESX 5 nested to create vsphere 5 virtual lab. I am now trying to create esx 5.5 lab. I have single poweredge 2900 on which I put esx 5.5. It won’t let me create esx 5.5 vms as it only goes up to hardware version 8. How do I use single server to create virtual lab of esx 5.5?

  34. Sorry but where to run the given command ? echo ‘vhv.allow = “TRUE”‘ >> /etc/vmware/config
    I have vm ware installed but still there is an error !!