We had an interesting discussion on one of the internal mailing lists this week. Someone asked what the general opinion was about disabling Tech Support. Of course some said disabling should not be a problem, but many also disagreed. The reason for this is simple: Support.
When Tech Support is disabled it removes the option to login to the console with “unsupported“. Please keep in mind that the console is the only way to get direct command line access to ESXi as SSH is disabled by default. This also means that in order to get access to the console you will need access to the physical host, or the IP KVM switch / DRAC / ILO for that matter. Hosts are usually located in a secured environment which removes the need for limiting console access in my opinion.
I can still imagine that people have a different opinion, but if you look at it from a support perspective you might change your mind. Troubleshooting an issue can get really complicated when there is no Tech Support access. I guess in a high secure environment you could treat ESXi as a stateless appliance and just install a new version when it fails. Personally I would prefer to find the root cause and try to prevent the same problem from occurring again.
Of course you can enable Tech Support again when needed but a reboot is required. This might cause the symptoms of the problem you were facing to disappear. It’s my recommendation to Keep Tech Support enabled.
[edit] Of course Alan “the king of powershell” Renouf jumped on this topic immediately and created a couple of lines of script which show you the current setting, disable it for all hosts or enable it for all hosts. Thanks Alan! [/edit]
Actually my thought is that VMware should enhance the BusyBox, aka Tech Support Mode, with the same command set as for a regular ESX4 and move it out of the ‘unsupported’ mode. I don’t need another COS that’s for sure, but still I need a versatile environment, at the physical host level, for day to day admin works!
100% agree! Was the guy who started this discussion named: BOFH ?
by any chance..?
Aleks
- – -
The best line from the article, bar none: “If someone has physical access to your datacenter and knows your root password you will have a problem no matter what.”
REALLY?
Duncan, Thanks for keeping things in perspective. At some point, you’re going to have to support the machine locally. Have you ever had to pull a vmnic out of a vSwitch because someone made a networking change that rendered the Management interface inaccessible? Good luck doing that any other way…
For sites with a strict auditing policy I can see doing it, enabling SSH to a limited set of IP’s might be more acceptable in those situations. There are various laws and policies that don’t allow you to assume that the staff is going to do the right thing, you must make every reasonable effort to assure they can’t and audit those areas where they might be able to not do so. I would personally hate to work in said environment as it means there’s too much process getting in the way of actually getting the job done, but they certainly do exist.
A good example was an IT security guy for some government agency who posted in the daily rant thread about a coworker allowing a vendor to use gotomeeting or something similar to shadow him. For most of us this is normal and necessary to get help troubleshooting hard problems. They had a strict policy about not sharing credentials which apparently included giving someone the ability to perform an action as you, and there were possible legal reprecussions for failing to follow said policy including possible felony charges!
SSH as disabled by default on ESXi
Understood, just saying that enabling SSH and disabling tech support might be necessary in some environments since AFAIK SSH sessions are auditable on ESXi. Of course maybe the ESXi default shell should be set to sudo with logging enabled =)
Physical console access is not always possible as far as i’m concerned so I would generally prefer to have SSH enabled for emergencies. You may be able to work around this if you had drac or ilo enabled on the host.
You forgot to take into consideration that many quality servers like HP and Dell have lights out administration tools that give you remote console access, like iLO2 or DRAC. So I could be on the moon yet still remotely login to a server console and use tech support mode. So thinking that console access requires physical access is not always the case!
Huh, I did take that into consideration but after changing the article slightly yesterday a whole sentence did disappear.
LOL,
This is like crippling ‘vm-support’ or disable the ‘Export Diagnostic Data…’ in vCenter. Because you’ll be gathering valuable information which can be used to hack onto the system. Uhm, oh yeah, we have access to the console already to run the command – so why not use that?
. Or like having an inbound firewall on ports which are not listened to…
Anyway: “How many levels of security do you want?!?”
And please don’t start about what kind of info is in “vm-support”… Extract it yourself, and look through it. Some find mac info, wwn info, ldap settings, etc. forms a liability. But to get the information, you need to have sufficient rights…
I sure hope I didn’t give people bad ideas now
I would just like to add my 2 cents worth and adding that disabling the Tech Support mode would be a bad idea. We have had situations on production ESXi servers becoming disconnected from vCenter and not accessible by any other means except via the Tech Support mode and having to log in and restart the management services on the CLI to regain control of the host without causing outages of VMs on the host. We don’t have SSH enabled.
I think people will be happy for tech support to go, if vmware provides equivalent commands in ESXi. Just for example VMKPING, what is equivalent command I do not know and many would try to find it from google.It would be tough for people to move to ESXi unless they get more and more information on doing same thing in ESXi as all of us would know from google on how to do it on ESX classic.
Other day I was troubleshooting vMotion 10% failure problem. When I hit the google, everyone talks about VMKPIng. What I would without tech support. So that gap needs to be filled first then.
The fact remains, that anything the VMware support team can do when we call in for help, we should be able to do on our own. The only reason we don’t is because we’ve missed something or we don’t know something.
If we do call for help, the second time the problem occurs, we should not need to call in.
The VMware support folks aren’t magic and they’re not writing code. What they are is familiar with the product. Hiding the “tech support mode”, or even making us type “unsupported” is nothing more than insulting.
Please don’t go out of your way to make our lives more difficult by continuing to remove tools ability to support our environments.
Though it aspires to be as such, ESXi has NOT yet reached appliance status.
I have a hard time understanding what you are trying to say.
“The VMware support folks aren’t magic and they’re not writing code. What they are is familiar with the product”
Isn’t that always the reason for calling support? Even if you had direct access to the console, which is by the way described in a KB article, you would still need to know where to look and what to look for. No it is not magic but they are THE experts on this. I’d rather call support and let them figure it out than fiddle around myself and end up with an unusable host.
I agree that “unsupported” is poorly chosen, and hopefully this will be changed in a future version. I also hope that some sort of “audit” trail can be enabled in the next version, which according to this KB http://kb.vmware.com/kb/1003677 is the main reason for “hiding” it.
Duncan,
Really, I’m echoing some of your post. I’m also stressing some of it well beyond your original intention.
The point that I’d really like to drive home to VMware is that, yes we want to use your support. We pay for it. When I say that tech support team doesn’t write code, they don’t. The execute some commands, read some logs and make some suggestions. Basically, they do what any seasoned ESX admin should be able to do. That’s all I’m saying.
Basically what I am getting at is that those who purchase ESX should be able to do anything the tech support guys can do without a contract violation or without compromising support. They fire open the console, take a look at the logs, issue some commands and try to protect the uptime of running virtual machines. We share a common goal.
That said, I really take offense to this line from the KB:
“Unsupported unless used in consultation with VMware Tech Support.”
This seems like the manager panic that occurred back in NT 3.1 when we started changing registry values to fix the settings that Microsoft had configured as default, but were not sufficient for our environment. There was a huge concern about changing the registry. Today, changing registry keys is considered routeine for any seasoned administrator.
A similar thing may be more common now with “schema extensions” in active directory. It’s a directory. Plan the changes out, what’s the big deal?
Red Hat is fully in favor of granting us access to our Red Hat Linux servers. There has never been a question about who is in control of the system and that support would be offered.
“Tech support mode”, should be fully supported. It should be allowed and endorsed by VMware and granted to ESX admins without a second thought. There is a reason that SSH is coming back to ESXi. The fact of the matter is that the system admins actually do need to touch ESX and do productive work for their environments.
ESXi has a great potential for the future, but it is not quite an appliance at this stage. Until it really gets there, VMware should not treat it as such. Even then, I might question how wise it is if there is even a 1% greater chance of rescuing that one VM which is still running.
We are passionate about uptime. http://xkcd.com/705/
Again I don’t agree. If any seasoned ESX admin should be able to do what support does then there wouldn’t be need for support. These guys know how to analyze log files, these guys are experts in troubleshooting and these guys are subject matter experts. There’s huge difference between support and sys admins. They are there for a reason.
Now again, I also believe that the term “unsupported” is poorly chosen and that it should be open for everyone without any restrictions.
Keep in mind that when you do fully open the console, the appliance “status” is gone for good.
The console with open access without restriction is agreed. That is, without a doubt, the single most important thing (to me) in this thread.
As long as there are situations when your only option is going down to the physical console to solve a problem, console access should be granted to admins. When my blade enclosure has a coms breakdown I can only do a greceful stop of my Vms to move them to another ESX (thanks to SAN shared sorage) by using the console access to the ESX.
VMWare can’t possibly guarantee that there wont ever be such situations.