At VMworld I hosted a group discussion together with Keith Farkas (HA Lead Engineer) on the topic of HA Futures. Based on this discussion group session Keith and I decided to gather more feedback from the field, this post will hopefully help us with that. Please do not hesitate to comment. I will have a couple of articles following this one, but lets get started with HA futures for the Restart Order first.
A topic that has come up at various sessions is HA restart ordering / priorities. Today HA provides four levels of restart priority: High, Medium, Low, Disabled. The thing to note with the current restart priority though is that there is no guarantee VMs are actually restarted in that order when the VMs are started on more than one host. Even when HA would restart them in the right order there is also no guarantee around when the boot cycle completes. Typically large virtual machines with for instance a database will take longer to boot than a server just running DNS. So what do we propose? We propose restart orders instead of restart priority. What does this mean, and what would we like to now from you?
There are two complementary ways of implementing this and we would like your feedback including which one you think would be most useful.
- Global Restart Order aka Bucketing
- VM to VM dependency Chains
Lets explain these two options and then I let you guys chime in.
Global Restart Order aka Bucketing is basically what you have today with “restart priorities” only it will actually enforce the restart order and it will allow for more flexibility. So with this option you could for instance create 5 buckets, and then add virtual machines to these buckets appropriately. These buckets could be: Priority 1, Priority 2 and so on. When a failure has occurred vSphere HA would then restart all VMs in the bucket “Priority 1″ first and when that bucket has finished starting (e.g., wait for VMware Tools Heartbeat to report “alive” for each VM) vSphere HA would continue with the next bucket and so on. Waiting for VMtools to report “alive” is one way to determine that a VM is “ready”. We are thinking of providing three other “wait” options — wait for an application heartbeat, wait a certain amount of time after the VM powers on, or today’s behavior, wait for the power on task to complete”.
I guess a couple of questions we have:
- How many levels would you like to see?
- Which of the wait conditions (e.g., wait on VMtools) are most useful for you?
- Suppose HA could not power on a “Priority 1″ VM. Do you want HA to stop powering on the “Priority 2″ etc VMs until it can, move to the “Priority 2″ group after a timeout, or something else?
Database VM –> Application Server –> Web Server
As you can see that would offer a significant amount of granularity, but also potentially a lot of operational complexity. How far would you like to go I guess is the question? Questions we have for you:
- Is an ordered list sufficient to express dependencies in a chain of VMs or do you need more sophistication?
- A VM with a dependent fails, do you expect HA to restart that child VM even though the previous has failed?
- What if HA could not be able to restart a VM with dependents — should HA restart these dependent VMs after a delay or only after the first VM is restarted?
That is it for now… Please chime in, as your response will help us define the future of vSphere HA.