This weeks VMTN podcast is about Homelabs. John Troyer asked on twitter who had a homelab and if they already posted an article about it. Most bloggers already did but I never got to it. Weird thing is that the common theme for most virtualization bloggers seems to be physical! Take a look at what some of these guys have in their home lab and try to imagine the associated cost in terms of cooling, power but also the noise associated with it.
- Jason Boche – EMC Celerra NS-120
- Chad Sakac – Building a home lab (check the storage he has at home!)
- Gabe – White box ESX home lab
I decided to take a completely different route. Why buy three or four servers when you can run all your ESX hosts virtually on a single desktop. Okay, I must admit, it is a desktop on steroids but it does save me a lot of (rack)space, noise, heat and of course electricity. Here are the core components of which my Desktop consists:
- Asustek P6T WS Pro
- Intel Core i7-920
- 6 x 2GB Kingston 1333Mhz
- 2 x Seagate Cheetah SAS 15k6 in RAID-0
I also have two NAS devices on which I have multiple iSCSI LUNs and NFS shares. I even have replication going on between the two devices! Works like a charm.
- 2 x Iomega IX4-200d
There’s one crucial part missing. On my laptop I use VMware Player but on my desktop I like to use VMware Workstation. Although VMware Player might just work fine, I like to have a bit more functionality at my disposal like teaming for instance.
That’s my lab. I installed 3 x ESXi 4.0 Update 1 in a VM and installed Windows 2008 in a VM with vCenter 4.0 Update 1. Attached the ESX hosts to the iSCSI LUNs and NFS Shares and off we go. Single box lab!
vSphere 4.0 Quick Start Guide
So you are running an OS – on that OS you run VMWare Workstation 7 and inside this Workstation you have 3 ESXi’s up & running?
I only have a whitebox testlab with vSphere on it. Inside this ESX, I run my vCenter as a VM. Can’t do vMotion, HA, DRS,… but don’t want to get multiple rigs because of the costs…
Thanks for posting this Duncan! This (and everything else you guys do) is a great help for the rest of us!
For my needs, a single box with a lot of ram, and HDD is good enough for me to learn and practice.
Keep up the great work!
Cool asus motherboard. I still have the 8gb limit with the asus p5m2/sas ones.
The only downside of this setup is that you can’t run 64 bits guest inside your virtualized ESX(i) servers.
IMO the only way to improve on that is to build a white box with ESX 4 on it. Then virtualize the VMs and your NAS storage. You can then run a mini lab on top of that.
But if you don’t have a full ESX 4 license, this is the next best thing.
I really need to get off my ass and build out a lab. I guess I’ve been spoiled because I get to build so many ESX environments for my customers. However now I need to get VCDX out of the way so a lab becomes more important.
I got a 2.2ghz quadcore phenom with 6gb of ram(GJ Bestbuy). I ran 2 ESX servers, a domain controller, vCenter server, and OpenFiler storage on that box in VMWare Workstation on a Linux distro. I got mostly everything I wanted running, HA/DRS, but FT doesn’t work in this situation due to no virtualization enabled in firmware/bios. Other than that it’s pretty cool for testing.
Hello Duncan,
I also have my home lab set up like this. My main desktop is running virtual ESX instances. I have 12GB of RAM. The only exception is that my storage is a VM as well
Openfiler virtual appliance providing iSCSI.
My other lab is physical with the IOMEGA box. Great little lab box.
Hi Duncan,
This is a very similar setup to what I use at home but I went for the cheaper AMD option. Is this used alongside access to a lab at VMware? Has there been anything you have had to revert to a phyical lab for rather than running on the home setup?
The main issue I have with mine is having to boot it all up when I want to use it, cant really leave mine running 24 x 7 easily (or without ear ache from the other half!) Would love to put a small server in the loft or somewhere for this reason.
Cheers
Barry
@virtualisedreal
I posted a year back my homelab equipement…but I swapped it over. On my Asus P5B I have now installed ESX 4 update 1 with local disks, the Openfiler is still there, and a desktop with W7 with Workstation 7 on it.
http://www.vladan.fr/whats-in-my-lab-at-the-moment/
But, it’s still 3 physical boxes. I’m gonna do something with it…. -:)Would like having some NAS with SSDs.. -:)
Next lab will be a Nehalem box…
Well running ESXi on such a box is nice, but would mean it has a single purpose. I would also like to be able to create videos on it. do some basic things like internet / blog writing etc. if I run ESXi I will still need to power up a second machine.
@virtualisedreal yes booting up everything takes some time. but for me it beats having 3 servers creating noise and consuming power while sitting idle.
I own very similar single box test lab
DFI LP UT X58
Intel Core i7 920 @ 3.6GHz OCed
3 x 2GB OCZ Reaper DDR3 1866MHz (going to add 6GB more of it)
2 x WD Raptor 300GB RAID-0
Only difference is that for NAS … I use one more VM of Openfiler
I just finished rebuilding my home lab. I have 6 systems in the environment.
1. Active Directory, old P4 with 2G ram
2. Firewall – pfSense 1.2.3, old P4 with 512M ram
3. vCenter – Windows 2008, SQL 2008, vCenter 4 U1. Hardware is i7920 with 12G ram, a few 15K SAS
4. ESX node 1 – Intel DG33BU mATX board, 1 onboard Intel nic, 2 addon Intel nic, 1 qLogic 4Gb FC HBA, 8G ram, 36G SATA disk
5. ESX node 2 – Intel DG33BU mATX board, 1 onboard Intel nic, 2 addon Intel nic, 1 qLogic 4Gb FC HBA, 8G ram, 36G SATA disk
6. Storage Server – OpenSolaris b130. Dual Opteron 252, 8G ram, 2*10K SCSI miror for OS, 4*10K SCSI in RAID-5 for some VM data (mail). 8*15K.7 in RAID-10 for VM and some VM data (database). 30G Intel x25-e for ZIL, 30G OCZ Vertex for Cache. 2 onboard Broadcom nic, 2 addon Intel dual-port nic. 4Gb dual-port qlogic FC HBA.
Switch:
2 HP Procurve 2824
1 Brocade 200E FC
I have another machine I haven’t configured yet, it has XEON W3520, 12G RAM, 2 onboard Intel nic and 1 qlogic 4Gb FC HBA. I’m going to build this as ESXi to host VC and SQL DB so I can free up the physical server.
@kekerode
How about disks IOs with everything virtualized. Is it still does some valuable job, or once you start 5 VMs (2ESX, 1DC, 1vCenter, 1 Openfiler VM, 1 XP client)… you just wait for the spinning never ends?
Then, having a little NAS box to spread the IOs does make sense..
Vladan
Duncan,
For JTroyer’s whitebox discussion a friend of mine has gotten inspired on how to make an ESX environment at home on the inexpensive side and put up this post per my request.
http://www.sentania.net/2010/01/home-esx-infrastructure-version-3-0/
I have not gotten the time to post my environment so I’ll just push his as something impressive at home.
I still love my water-cooled ESX 4i server: http://www.jume.nl/blog/7-vmware/124-v-water-esx4i-overclocked-watercooled
Very stable, fast and quiet.
Thank you Duncan, it was very valuable information. I have almost made a decision to buy a server but considering the noise,electricity and price you mentioned above, i’m going with your suggestion.
That’s exactly my home lab apart the shared storage where I’m using QNAP devices…
Unfortunately you can’t test everything on such environment, that’s why I have another ‘playground’ at the office for serious stuff!
Good post!
I also have a physical lab in the office which consists of:
6 x Dell R610 with 24GB
2 x EMC NS20
1 x NetApp FAS 2050
But we use that is a sandpit for the consultants… good stuff though!
I’m looking to build a little physical test lab @ work.
Could someone point me to a physical brand-name box to buy? Would a HPML115 work?
Thanks
Hi Duncan,
I have a similar components as in your setup, but have two disk not in any RAID configuration (One disk with Window 7 and the other ESX). I installed BCD bootloader but have not got this to work sucessfully. The only way I can dual boot is by changing the boot order in the BIOS – not the ideal situation. I’m able to to have a native ESX install and also a Windows install for personal stuff.
I was wondering if any of your readers of your blog had any success with dual booting?
Are you running VMs inside your virtual ESX machines, or do they just exist for testing interaction between various ESX hosts.
I just can’t imagine a VM on top of a VM.
@Vladan
Yes .. It takes time only while starting VMs. In fact only Openfiler VM is on WD Raptor RAID-0 drive, all other are on WD Black 1TB. I am going to build separate NAS rig. Maybe based Starwind iSCSI Target or Openfiler.
Here’s our setup running under my colleague’s college bedroom (Tuan) http://www.engr.ucsb.edu/~duonglt/vmware/ghettoDatacenter.html
–William
@kekerode
I’m just thinking. Would it make sense to add One SSD localy (like 128 Gigs) to the box, and use it as a iSCSI target for the VM’s?
@Duncan on what OS did you run VMware Workstation ?
Windows 7 – 64bit.
As a field sales engineer, It is critical that I have a solution I can test before I engage a customer.
A critical part of testing in a Home Lab is shared storage. There are lots of options, and I have tried a few. If power and space are a concern, a virtual storage appliance (VSA) solution is great. Alternatively, the low power and features of the new SOHO NAS Devices out there are great as well if you want to move the storage out of the server.
Physical Options:
- QNAP TS-439 PRO II – VMware Ready Certified iSCSI NAS (Dual Gigabit ports + USB) with my own 1.5TB SATA Drives. Thin provisioning! Low power w/web server, email, and more (I love mine!)
- Iomega iX4-200/200d Nas Device – Again, a nice low power ISCSI solution with everything included in the device.
- Desktop PC Running FreeNAS, OpenFiler, Star Wind, etc. – Software ISCSI/NFS solutions utilizing a cheat desktop PC, NIC’s, and spare drives.
Virtual Storage Appliance Solutions
- FalconStor Virtual Storage Appliance – Leverage local storage DAS
- HP/LeftHand Networks – Leverage local storage DAS
If I need to take my lab on the road, ditch the physical servers and move over to VMware Workstation 7 on Windows 7 64Bit on a high power laptop (I now use a MacBook Pro with 8GB RAM). I’ve been able to stand up two ESXi Hosts in Workstation 7 and point both to my iSCSI devices for a quick proof of concept. If you are not concerned about the Guest OS’s, run small instances of TinyLinux. If you have the CPU and require portability, the VSA is the way to go as well, but you’ll want to get off of those old 5400 RPM Drives and step up to a SSD.
Duncan,
Nice post, my question is that did you build this desktop yourself or did you pick it up from the shelf.
I am also trying to build my own home lab and i will rather have 1 desktop rather than multiple server
I build it myself. I would probably recommend to add a 40GB SSD to the system having worked with it for a while now and replace the SAS’s with a 7200GB SATA.