Last couple of months I noticed something which led me to believe that automation is scary… And I am sure that my friends William Lam, Luc Dekens and Alan Renouf are on top of their desk right now and yelling at me but when they finished reading this post I am sure they agree.
I have been blogging for a while and when I started blogging, more than 6 years back, I posted a couple of simple scripts I wrote. The scripts allowed you to do simple things like checking the available disk space, finding snapshots, committing snapshots, find unregistered VMs, clean unregistered VMs etc. Basically the kind of scripts I used when I was a consultant to clean up the mess I typically found a couple of months after a new environment was deployed.
Many people downloaded these scripts and ran them in their environment, back then I didn’t think much of it to be honest… I mean I wrote some simple scripts and people were using those to their advantage right? Well… yes and no. Yes I did write scripts, and yes they were using them to their advantage and yes they were simple for sure. (I am not a scripting god.) So what is the problem then?
The problem is simple, 6+ years after I wrote those scripts I still receive questions about how to use them. Now note this: Those scripts were written when ESX was still the core hypervizor to use. Those scripts were written to run within the service console. Those scripts were written for ESX version 2.5.x and 3.x. Today we use ESXi and majority of folks will be on version 5.x. More than 6 years have passed, but people are still downloading them and putting them in places where they don’t belong and try to run them; without thinking about.
That is why I think Automation is Scary! (Yes in this case capital S is warranted.) If you are looking to automate operational procedures / tasks. (Automation is NOT a replacement for proper operational procedures by the way!) If you are looking to create or re-use (simple) reporting scripts, or scripts which execute simple tasks for you make sure you:
- Read the script, and make sure you understand every line of code
- Test the script in a test environment
- Check for which version the script was written and validate it against your version
- Don’t trust ANYONE blindly, and when changes are introduced in your environment or to the script go to step 1 again!
Automation isn’t really scary of course, automation can make your life easier. Just make sure you don’t become that person who has to restore 80 VMs because that script which was cleaning up unregistered VMs unintentionally deleted production VMs… Make sure you understand what your scripts are doing and assess the potential risk.
virtualrol says
I agree with you duncan, automation is the next step to go to the cloud
Drexciya says
Good points there.
I think that the problem is that people are quite reluctant to delve deeper into scripting. Whereas this is something any Linux admin should know, it’s not that common at all in a Windows environment. Microsoft should more or less educate Windows admins better and keep in mind that they’re not programmers! (they’re trying to do so in the recent course material, but I’m not convinced whether that will help) So don’t make the training material or examples too difficult or too much focused on programming “things”.
We saw the same thing with VBScript, which took quite some time to get adopted on a larger scale. Only after a few years there were decent resources and good books (I remember a good book which hits the ground running and is very much catering to the non-programming crowd).
As a trainer I notice the same thing, when I show some simple scripts during a VMware class. And there’s hardly an audience for a PowerCLI course. We did do some workshops, a bit like the things you do at VMWorld, but that’s it. Maybe we need to do more basic workshops to remove the fear of doing some scripting.
Steve says
Every power cli session I have seen at VMworld is packed. In my environment we have made a huge push for powershell and the return on investments are huge (probably no shocker there to the Linux folks). The scary thing from Duncan’s article to me are the people downloading them and using them, and not knowing what they do. You should be able to at least understand what the script is doing. A “Remove-VM” command probably shouldn’t be in there. I have junior admins downloading scripts from the internet and not checking them, and I constantly have to remind them to check what they download. Just because it’s not compiled doesn’t mean it’s safe to execute.
Yes there is code signing protection but the admin population has got to get away from just chucking scripts at problems without understanding what it’s doing.
Juan Pablo Reyes Sepúlveda says
Great Post Duncan. Never it is bad remember the importance of “Read, Understandi Test” before execute any things.
Mark Gabbs says
Sometimes, it amazes me about what people use from the net without reading the details…
Thanks for putting this up, as you have a huge following, and this message will reach a huge audience
Randy Streu says
Very true…has been for 20+ years. This is why even with the DevOps push, we still need to understand that we need proper change management, backup/roll back plans, etc. In fact, they may be even more important today given the consumerization of IT aspect (people expect higher SLAs now…).
Edward Grigson says
As Randy says, this has been the case for a long time! So what happens a few years from now when (if?) automation moves up a gear with products like vCAC? More complex code, integrations and orchestration – yet the same problem is likely to occur where people try to use canned ‘example’ code. Automation has great potential, but with great power comes great responsibility… 🙂
Chris says
Not sure if all automation is scary. Here is my take on it: http://www.automate-it.today/?p=52
Warren Legg says
Hi,
I agree particularly with Randy and Greg. There is a qualitative difference in “automation” between (a) the personal use of scripts by Admins to expediate their day-to-day lives, and (b) the implementation of complex workflow-orchestration and monitoring code to realise policy-based business logic.
In case (b), the config. management, QA, development and testing processes need to be defined and realised in a highly controlled form. In reality, the development and delivery of the automation code needs to be managed exactly as a software house with the requisite controls to preclude (no, mitigate;)) software defined weapons of mass destruction…!
Randy Streu says
Exactly Warren. Too many Admins use the “wild west” approach versus a disciplined approach. The latter approach is scalable and safe…I am a huge automation proponent (I’m lazy). When used appropriately, it is far superior to manual steps for many reason (repeatable, scalable in large environments, built-in knowledge capture by SMEs, etc.)…
Matthew Povey says
When I still a sysadmin at a major British Broadcaster I wound up writing a short batch file (ADSI was young and PowerShell had yet to be conceived, let alone born) that checked whether a service was running and restarted it if it had failed. This was necessary because a brain-dead radio playout system a very large contractor had installed was incapable of keeping running for more than a few hours at a time.
I wrote a bunch of comments at the head of the batch file explaining how this was a ‘horrible fscking hack to keep this PoS running’ (I can’t remember the exact wording but there was lots of swearing as it was written late at night by a very annoyed me). I wrote a second batch file to set it up to run on a schedule. That got things running reasonably well and I hoped it would give the very large company involved time to sort their stuff out.
But that’s not how these things go. Years later (six or seven – maybe more), I was chatting to a former colleague who explained that not only was the batch file (including its scheduler file) still around but it had been rolled into the official build of the software – unchanged and including all my colourful language.
Automation is definitely scary and organizations need to take steps to avoid the kind of zombie-like qualities that my batch file developed. More importantly though, be bloody careful what you write in comments – your words will probably be there longer than you were.
Fred Peterson says
The problem is people somehow get into jobs that they are unqualified for. Its one thing to not know how to do something and Google for it and then pick apart the code you find to understand it…its quite another to just assume what you copy and paste should just work without any thought. There are a lot of educated idiots out there who think just because they have a degree they should have a job.
Espen Nilsen says
This goes with any piece of software you find on the internet. Open source or not.