I was just reading this article on the Malaysia VMware Communities website I’ve read a couple of article on their website that didn’t make sense but this this time I’m going to respond cause it might set people on the wrong foot. Anyone is of course entitled to their own opinion and views, but please reread your article and check the facts before you publish, especially when your blog is featured on planetv12n. A short outtake of the blog:
If we refer to the current version which is ESX 3.5 u3, the maximum number of Vcpu per ESX server is only 128 per ESX Servers. Personally, I think the number of Vcpu per ESX servers is too minimal. Imagine if we do run a servers with 4 or 8 physical CPU sockets and we consolidate 40 : 1 Physical server in our virtualization environment, we will hit to the bottleneck on maximum numbers of Vcpu per ESX servers but not due to the CPU consumption
Reading this short section one might think, why reply it makes sense? No, it doesn’t make sense at all:
- The current limit isn’t 128, it’s 192 vCPU’s.
So even with a 40:1 ratio and all VMs provisioned with 4 vCPU’s you wouldn’t hit this limit. Read the max config guide, it’s the bible for virtualization consultants. - But even more important: co-scheduling and over provisioning will impact performance. With most VM’s running 2 or even 4 vCPU’s scheduling will be almost impossible even with the relaxed co-scheduling techniques ESX is using these days. In other words, please don’t use multi vCPU VMs as a standard, you can read more on c0-scheduling here.
The author asked VMware to bump up the max number of vCPU’s. Now for a VDI environment this can and will be useful I think. Again if you are hitting the number with a 16 core machine, you might need to reconsider your provisioning strategy.
I expect the number to go up… especially after watching Stephen Herrod’s keynote at VMworld Europe 2009.
Paul Shadwell says
I’m wondering if the reason your having problems with understanding these posts is that, not only are the facts wrong but the grammar is terrible. The author is obviously not a native English speaker or worse he’s passing his original text through Google’s language translator and not having it proof read.
My comment would be, if your target audience is Malaysian, why not write it in your mother language.
As so often happens, the true meaning is lost in translation.
Jason Boche says
I’ve been belly aching about this for a few months. There are a few VMWare Configuration Maximums documents floating around in Google cache when you search for “vmware configuration maximums”.
Unfortunately:
-The 1st hit Titled “VI3 configuration maximums” is the older outdated version of the document. It’s title is confusing because one would think it applies globally to all VI3 including the most current releases of ESX3.5u3 and VC2.5u4. It doesn’t.
-The 2nd hit Titled “VMware Infrastructure 3: ESX Server 3.5 Update 2, ESX Server 3i version 3.5 Update 2,
VirtualCenter 2.5 Update 2” is actually the most current document, but its title is confusing. One would think it is only valid through Update 2. Incorrect. It is the current document which also applies to Update 3.
I’ve blogged that the Configuration Maximums document is my favorite VMware document of all time, unfortunately, VMware is behind in updating it or accruately naming its title.
Jas
Duncan Epping says
Could be Paul, but even if the grammar is just bad it doesn’t make sense at all to use mostly 4 vCPU vm’s.
By the way you can find all U3 docs here:
http://www.vmware.com/support/pubs/vi_pages/vi_pubs_35u2_3i_e.html
Stu says
Once again you have beaten me to a response to something, but well said and I completely agree about the 4 vCPU machines.
We have a 32 core DL785 running in the lab at work which usually runs about 180 active vCPU’s, but there is no way in hell I would ever recommend such kit in production for 2 reasons:
1) Do you really want to run that many machines on a single piece of commodity x86 hardware?
2) In my experience memory resources sre exhausted well before CPU (which is why i made this post http://vinternals.com/2008/12/kicking-esx-host-hardware-standards-down-a-notch/). Getting an optimal ratio of CPU / RAM on a 32 core box would be a _very_ expensive proposition indeed!
Jason Boche says
Thank you for the link – I know where the docs are. With all due respect, my point is that I’ve witnessed a lot of people whose first habit in looking for something is to Google. This is where they will go wrong with Configuration Maximums. Furthermore, that does not excuse VMware from updating the titles of their documentation to current.
TimC says
“But even more important: co-scheduling and over provisioning will impact performance. With most VM’s running 2 or even 4 vCPU’s scheduling will be almost impossible even with the relaxed co-scheduling techniques ESX is using these days. In other words, please don’t use multi vCPU VMs as a standard, you can read more on c0-scheduling here.”
I guess that statement doesn’t quite add up to me. The document you link specifically says with the relaxed co-scheduler, if a vcpu is idle in an SMP VM, that vcpu will not cause any co-scheduling overhead.
“For example, an ESX Server with two physical cores may be running one vCPU each from two different VMs, if their sibling vCPUs are idling, without incurring any co-scheduling overhead.”
It would seem to me giving a VM multiple cpu’s if it can benefit from SMP will always be a no-brainer.
Stu says
TimC – it is a gross over simplification to say there is _no_ overhead with relaxed co-scheduling. There are still timer interrupts to contend with at the very least – Have a read of this document http://www.vmware.com/pdf/VI3.5_Performance.pdf for more info. Duncan was merely pointing out that if you hit the default 128 vCPU limit on a 16 way box because you tried to get 40 x 4 vCPU guests on there, you may want to rethink your provisioning strategy. Which I strongly agree with.
No one is saying that you shouldn’t provision SMP guests if they actually can benefit from it – we’re saying that if you actually could get more than 96 x 2vCPU or 48 x 4vCPU guests onto a 16 core box to hit the 192 vCPU ceiling, then SMP is clearly not necessary for the majority of those guests and a rethink of the provisioning strategy is.
Jase – I totally agree with your point about the documentation mess, however one would hope that anyone claiming to speak with authority on VMware infrastructure design would check their facts more carefully than a cursory Google search 😉
malaysiavm says
4 Vcpu is consider too much? may not really apply to some environment. If the business accept VM and not performance impact on the production system, you better think twice when you have only 1 Vcpu per VM.
Again, please read to the comment of the original post, we do serve multiple time zone in single ESX servers. 40:1 is only happen when we VMotion every VM into single host, as we require to update our ESX servers frequently. For sure we will not keep 40:1 permanent on the ESX server.
Duncan says
I haven’t got a clue what you are trying to tell me MalaysiaVM. Really no clue at all. I don’t know your environment, but your article really didn’t make sense.
@Timc: Try it sometime. add several vCPU vm’s to a host, like 2 or 3 times the amount of vCPUS vs available cores. Open ESXTOP press “S 2” and run some CPU intensive tasks on at least two or three vm’s and see what happens to “%RDY” and “%CSTP”. Especially if there’s a task running that will use SMP.
rbrambley says
All,
Just a few quick replies and clarifications. VMware design best practices (per the offical Infrastructure Design class) is that you have twice the physical cores (not sockets) of your largest vCPU VM on any single ESX host. So, if you had a 4 vCPU VM you should have an ESX host with 8 physical cores. The number of VMs with that many vCPUs you can run varies, but the point is to eliminate the scheduling conflict.and queue times. ESX only schedules a SMP VM’s access to the physical processors when all vCPUs can have access simultaneously.
In general, I agree you should not load up an ESX host or even a cluster of hosts with too many SMP VMs. Besides, it’s been proven that single vCPU VMs perform better as a rule. Look at Citrix VMs as an example.
jrenton says
We have a VDI environment where we run 5 vCPUs per physical core and the environment run OK. Each VDI is running XP with a single vCPU and 1GB memory.
However when we put one of the hosts into maintenence mode we get performance issues and CPU ready issues.
I think we have pretty much reached the limit at 5 vCPUs per core and no more that 4 should ideally be used to allow for maintenence.
This is a real world example running vSphere on a 8 node cluster of BL685G1 hosts with 4 x dual core Opteron procs and 64 GB memory per host.