• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

Creating a nested lab

Duncan Epping · Jun 12, 2012 ·

I was just building a nested lab to record some demo videos. I find myself googling for this every single time so I figured I would write about it so I can easily get it of my own website. Many have written about this before and all credits go to William Lam and Eric Gray, which are the two  main blogs I have used in the past to get this working.

After installing ESXi on my physical box I “ssh” in to it. In order to allow “nested ESXi” to boot a 64bit OS you will need to run the following:

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

After you have done that you will want to make sure you will get network connection. Go to your “VM Network” portgroup, or if you named it differently the portgroup that is used to connect the virtual ESXi hosts to. For each of the hosts do the following:

  1. Click on the host
  2. Go to “Configuration”
  3. Click on “Networking”
  4. Click “Properties” on the vSwitch
  5. Select the correct portgroup
  6. Click “Edit”
  7. Click “Security”
  8. Set “Promiscuous Mode” to “Accept”
  9. Click “Ok”
  10. Click “Close”

Now for each virtual ESXi host (note there is a “guest os” called ESXi 5 in there, use it!) that you have created do the following:

  1. Right click on the VM
  2. Click “Edit settings”
  3. Click the “Options” tab
  4. Click on “CPU/MMU virtualization”
  5. Select the 4th option “Use Intel VT-x / AMD-v…”

I am building this out to record a new of “DR of the Cloud”. In other words, 3 virtual clusters + vCloud Director + SRM + vSphere Replication + Virtual Storage Appliances… Cool stuff right.

Share it:

  • Tweet

Related

Server, Various esxi, nested, vSphere

Reader Interactions

Comments

  1. Jose Luis Gomez says

    12 June, 2012 at 17:40

    Hi Duncan,

    I recorded a video about it (http://blog.e2h.net/2011/09/01/how-to-run-xenserver-5-6-on-vsphere-5-with-support-windows-virtual-machine-into-xenserver/)

    My homelab doesn’t support MMU/EPT, do you know if it is simulated over software, the performance of VMs executed in ESXi nested are worst?

    Best regards,
    José Luis Gómez

  2. Nick Marshall says

    12 June, 2012 at 21:09

    Hi Duncan,

    I build nested ESXi all the time too, for lab work / POC testing etc.

    Not sure if you’ve heard of it, but the vBrownBag crew (particularly Alastair Cooke) have been working on automating this lab build / setup. I thought your readers might also be interested in rapidly producing a functioning nested VMware vSphere 5 lab with shared storage, vMotion, HA, DRS etc.
    It was mentioned on the VMwsre community roundtable podcast a few weeks back too.

    Check it out at http://www.labguides.com/autolab

    Cheers,
    Nick Marshall

    • Alok says

      17 June, 2014 at 04:07

      HI Mate, I am new in VM world and got stuck in a bit funny issue. I am trying the nested host configuration. My host are not talking to each other even after enabling “Promiscuous mode” on Vswitch0. not sure if I have to do something else to get them communicate with each other.

  3. Dave R says

    13 June, 2012 at 19:57

    Strange strange problem. Boot up and after it scans for devices it doesn’t find any volumes to install on, yet the guest has an 8 GB thick vmdk. I have tried different sizes, different provisioning methods, different datastores, and adding a disk after the fact.

    Then I tried accidentally initially created the VM as Other (64 bit). And there it was. Only if you initially create the VM as RedHat 6 64bit do you get this behavior.

    • Anthony says

      10 December, 2012 at 22:56

      The controller type will be different depending on what initial guest OS you select and most likely the one you selected is not supported by ESX

  4. David Dominguez says

    14 June, 2012 at 03:39

    Hi Duncan,

    This is indeed cool stuff 🙂 Can you please share your physical server specs?

    Thanks,
    David Dominguez

  5. mkematt says

    14 June, 2012 at 19:56

    I would also be interested in your lab build. There are so many options on the HCL it’s hard to decide these days.

  6. Duncan says

    14 June, 2012 at 20:53

    This is not a regular box. I am running nested on a proper ESXi Cluster with 4 DL380 G7 hosts. The reason I am doing that is because I can simply snapshot all and revert to before me messing things up.

  7. Peter-Paul says

    15 June, 2012 at 19:03

    i can say that a nested lab is working fine here. Been using it for some months now because ive only 1 powerfull server. (dual Xeon e5-2680 64GiB = nested 4x QC 16gb ESX cluster)

    Used the same source blogs as Duncan for building nested environment, only im using it 24/7.

    Perhaps a small tip: use a opensource/free software NAS/SAN for sharing storage to your nested ESX cluster.

    Grtz!

  8. Ed says

    18 June, 2012 at 17:10

    Do you know of a way to automate the updating of /etc/vmware/config in an auto deploy environment? We run our entire lab using auto deploy and vcloud director but this is one file we have yet to be able to figure out how to properly update to make vsphere on vsphere work.

    Ed.

  9. Arun Raju says

    20 June, 2012 at 03:17

    I have been using 2 custom built x64 boxes for my home lab.

    Duncan – The nested ESXi VM still can’t run x64 VMs right?

  10. Duncan says

    20 June, 2012 at 06:49

    They do run 64 bit VMs!

  11. Arun Raju says

    20 June, 2012 at 06:57

    I was under the impression that AMD-V or Intel-VT supported only 1 level of virtualization.

    If I understand it correctly, you have installed ESXi on a HP DL380 G7 hardware. Within that you have created a VM with ESXi and still could run 64-bit VMs within the nested ESXi.

  12. Duncan Epping says

    20 June, 2012 at 08:55

    Yes that is correct, that is exactly what I am doing. Read William Lam’s post for more details

  13. Arun Raju says

    20 June, 2012 at 08:58

    Awesome, thx 🙂 That was what I wanted for a long time.

  14. Greg Gooch says

    22 June, 2012 at 17:36

    Any advice on getting snapshots of virtual ESXi guests? There was a fix posted by Eric Grey for ESXi 4.1 physical hosts, but the command doesn’t work for physical ESXi 5 hosts hosting virtual ESXi guests. The reason I need this is because I use VDR to backup, which of course takes snapshots. This essentially destroys the filesystem of the ESXi guest on reboot and you need to reinstall.

  15. mb says

    25 June, 2012 at 15:59

    What about the parameters

    monitor_control.restrict_backdoor = TRUE
    monitor_control.vt32 = TRUE

    Are they still useful ?

  16. PRAVIN1CA says

    29 June, 2012 at 10:05

    Simply great 🙂 Duncan. Thanks for sharing…i am exicited

  17. Tal says

    6 July, 2012 at 17:35

    Hey duncan, i like you blog very much,
    I have a question on how to make a lam for srm with only
    1 storage array of nettap. Did you try this?
    Thank you in advanced

  18. Duncan Epping says

    6 July, 2012 at 21:41

    Not sure what you are referring to Tal.

  19. Tal says

    6 July, 2012 at 23:06

    Ok so lets break it down,
    2 ESXi 5 connected to NetApp Storage.
    2 vCenters , 1 HQ , 1DR connected via Link Mode.

    i want to Install SRM 5 but only use 1 NetApp because i don’t really have a DR.

    everything is in 1 site with only one Storage Array, but i want somehow to simulate it .
    because if i install the SRA on VC 1 and VC 2 they will both see the same NetApp .

    Did you write maybe a Manual regarding this kind of Labs ?

  20. Ed says

    20 July, 2012 at 18:28

    As an update to my query about how to set vhv.allow when utilizing auto deploy. I ended up solving this problem by developing a custom vib and offline bundle that can be added as a software depot and subsequently added into an image profile that can be used on a host. This is working suprisingly well and could also, theoretically, be used with update manager or on a host directly to ensure that even through updates the setting is reset. I have more info about the bundle and the download on my blog:
    http://blog.edmclain.com/2012/07/enabling-vhv-allow-while-utilizing-esxi-on-vsphere-auto-deploy/

    I have also written another bundle that enables the copy and paste feature for the vmrc and vsphere client that follows the same methodology. I hope somebody finds it useful.

    Ed.

  21. realdreams says

    2 August, 2012 at 00:56

    Could you explain why promiscuous mode needs to be enabled? It seems ARP resolution on host’s vSwitch fails for the port group without promiscuous mode.
    Why a regular VM does not need promiscuous mode?

  22. Chris says

    15 August, 2012 at 22:37

    When I type:
    cat /etc/vmware/config
    I can see the line vhv.allow = “TRUE”
    Still the ESXi doesnt show up in the Guest operating system version drop down menu.
    This is for esxi 5 host under wmworkstation as on esxi on an HP DL360
    Is there anything else I should look for? I’m getting desperate since everyone gets it to work, and I fail on two machines…

    • Rajan says

      18 May, 2013 at 21:19

      Chris,
      I am having the same problem. ESXi does not show up in the Guest operating system version drop down menu. I am using ESXi5 trial license. Please let me know if you find workaround for it.
      Thank you in advance and appreciate your help.

  23. Anil says

    14 September, 2012 at 06:16

    I created ESXi 5.0 under that i created VM’s 2 ESXi4.1 and 1 windows 2008 server r2.In win2k8 i nstalled Vcenter5.0 and add that 2 ESXi4.1 servers.

    While i am creating win2k8 64 it throwing error.

    An unexpected error was received from the ESX host while powering on VM vm-62.
    The processor does not support a required feature for this virtual machine. The virtual machine cannot be powered on

  24. Vince says

    18 October, 2012 at 06:56

    Hi all – I follow the outline as specified by Duncan and William LAN and my virtual ESXi host comes up fine. However, it does not have visibility to the local ATA disks and as such I cannot perform vMotion. Anyone seen this problem and know the resolution? Require modification to the .vmx file perhaps?

    Any help appreciated,

  25. Vasanth says

    24 October, 2012 at 11:37

    When i power on the 64bit VM. i am getting the following error message.
    Virtual machine message.
    This virtual machine is configured for 64bit guest operating system, however, 64-bit opetation is not possible.
    Longmode is disabled for this virtual machine.

  26. Vijay says

    27 March, 2013 at 23:20

    I have Hp Pavilion DV6 with 64bit , Intel Duel Core, 2.5 Mhz, 350 GB. Can I build Cisco Data Centric Nexus labs and using VMware tools?

    Do I need to buy another expensi i3 processor, 500GB, laptop?

    Kindly reply your answer. Thanks in advance.

  27. Charles Buege says

    4 April, 2013 at 17:07

    Hello there. I did want to add one comment to this chain.

    Adding ‘vhv.allow = “TRUE”‘ works if your physical host is ESX 5.0.
    Adding ‘vhv.enable = “TRUE”‘ is needed if your physical host is ESX 5.1.

    This held me up for several hours, so I hope this helps others out there.

  28. Arun Raju says

    18 April, 2013 at 18:03

    Thank you, Charles Buege. That saved me a lotta time.

  29. B says

    22 May, 2013 at 11:33

    Important to note is that this does not work on AMD Bulldozer processors.
    I have tried both ‘vhv.allow = “TRUE”‘ and ‘vhv.enable = “TRUE”‘ on ESXi 5.0.0 1024429
    without success of getting the Guest OS option “ESXi”

    ESXi 5.0

    On ESXi 5.0, virtualized HV is prohibited by default. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. It is available on hosts that support Intel VT-x or AMD-V, but it is not recommended for systems without second level address translation (EPT or RVI), because of its poor performance without SLAT. Unfortunately, this feature does not work on AMD “Bulldozer” CPUs running ESXi 5.0.

    http://communities.vmware.com/docs/DOC-8970

    ESX install only renders “HARDWARE_VIRTUALIZATION WARNING: Hardware Virtulization is not a feature of the CPU, or is not enabled in the BIOS”. ie the cpu doesn’t support it at all. =(

    • B says

      22 May, 2013 at 13:58

      I stand corrected (by myself ?)

      I managed to set the guest os to ESXi 5.x, but only after I initially created the machine and enterd
      “Edit virtual machine settings…”
      “Options/General Options”
      and then looked under “Guest operation system / other”. Then it’s visible.

  30. Joel says

    28 July, 2013 at 01:23

    For those of us with “classic” host equipment, keep in mind that Intel EPT and AMD RVI are required (in addition to Intel-VT AMD-V ) for 64-bit guests on nested hosts. (see http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html). I only had VT and the 64-bit guests weren’t happening.

  31. Confucius Say says

    10 October, 2013 at 22:34

    Building vCloud nested home lab on laptop like trying to sequence power on Apollo 13 return flight.

    What’s the sizing formula? for 16GB of memory. Gah!!

  32. Frank Pisano says

    28 December, 2013 at 04:58

    Hey Duncan,

    I got ESX 5 nested to create vsphere 5 virtual lab. I am now trying to create esx 5.5 lab. I have single poweredge 2900 on which I put esx 5.5. It won’t let me create esx 5.5 vms as it only goes up to hardware version 8. How do I use single server to create virtual lab of esx 5.5?

  33. tt says

    15 February, 2014 at 18:06

    hi all
    i also face the same problem, let me i try your instraction and replay u

  34. candyicey says

    24 September, 2014 at 10:53

    Sorry but where to run the given command ? echo ‘vhv.allow = “TRUE”‘ >> /etc/vmware/config
    I have vm ware installed but still there is an error !!

Primary Sidebar

About the author

Duncan Epping is a Chief Technologist in the Office of CTO of the Cloud Platform BU at VMware. He is a VCDX (# 007), the author of the "vSAN Deep Dive", the “vSphere Clustering Technical Deep Dive” series, and the host of the "Unexplored Territory" podcast.

Upcoming Events

Feb 9th – Irish VMUG
Feb 23rd – Swiss VMUG
March 7th – Dutch VMUG
May 24th – VMUG Poland
June 1st – VMUG Belgium

Recommended Reads

Sponsors

Want to support Yellow-Bricks? Buy an advert!

Advertisements

Copyright Yellow-Bricks.com © 2023 · Log in