Yellow Bricks

by Duncan Epping

tips

5 Tips for preparing your VCDX Defense

· ·

After the VCDX defenses Boston I had a chat with Craig Risinger, also known as 006 ;-). We discussed some of the things we’d seen on the panels and came to the conclusion that it wouldn’t hurt to reiterate some of the tips we’ve given in the past.

  1. It’s OK to change your actual project documents. See the following points for examples. This isn’t really about what you actually happened to do on a particular project with its own unique set of circumstances. It’s about showing what you can do.This is your portfolio to convince potential customers you can do their design, whatever they might need. It’s about proving you could work with a customer to establish requirements and design an architecture that meets them.
  2. Include everything the Application says is mandatory. Don’t be surprised if you have to write some new documents or sections. For example, maybe a Disaster Recovery plan wasn’t important in your project, but it will be to another customer or in another project, so you should show you know how to create one.
  3. Explain any bad or debatable decisions. Did your customer insist on doing something that’s against best practices? Did you explain what was wrong with it? Say how you would have preferred to do things and why. Even if you just made a mistake back then, that’s OK if you can show that you’ve learned and understand the error you made. If you are using VMware’s best practices make sure you know why it is a best practice and why it met your customer’s requirements.
  4. Show you can design for large scale. It’s OK if your actual project was for a small environment, but show that you can think big too. What would you have done for a bigger customer, or for a customer who wanted to start small but be able to scale up easily? What would you need to do to add more VMs, more hosts, more storage, more networking, more vCenter servers, more roles and division of duties, a stronger BC/DR plan in the future? How would that change your design, if at all?
  5. Architect = Knowledge + Reasoning. The VCDX certification isn’t just about knowing technical facts; it’s about being able to apply that knowledge to meet goals. In the defense session itself, be prepared to discuss hypothetical scenarios and alternative approaches, to decide on a design, and to explain the reasons for your choices. Show you know how to consider the pros and cons of different approaches.

There are also many other useful collections of advice for pursuing a VCDX certification, we highly recommend reading them as they will give you an idea of the process. Here’s just a sample:

Craig Risinger (VCDX006) & Duncan Epping (VCDX007)

Network loss after HA initiated failover

· ·

I had a discussion with one of my readers last week and just read this post on the VMTN community which triggered this article.

When you create a highly available environment take into account that you will need to have enough vSwitch ports available when a failover needs to occur. By default a vSwitch will be created with 56 ports and in general this is sufficient for most environments. However when two of your hosts fail in a 10 host cluster you might end up with 60 or more VMs running on a single host. If this would happen several VMs would not have a vSwitch port assigned.

The most commonly used command when creating an automated build procedure probably is:

esxcfg-vswitch -a vSwitch1

This would result in a vSwitch named “vSwitch1” with the default amount of 56 ports. Now it is just as easy to set it up with 128 ports for instance:

esxcfg-vswitch -a vSwitch1:128

Always design for a worst case scenario. Also be aware of the overhead, some ports are reserved for internal usage. You might want to factor in some additional ports for this reason as for instance in the example above you will have 120 ports available for your VMs and not the 128 you specified.

Changing the directory of your vSphere vCenter log files

· ·

Something that a lot of people haven’t looked in to or just don’t think about is relocating the log files of vCenter, I wrote a short article 2 years ago and thought it was time to reiterate it. By default (Windows 2003) log files are stored in “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs”, and for Windows 2008 log files are stored in “C:\ProgramData\VMware\VMware VirtualCenter\Logs”.

As you can imagine the C:\ partition is not the ideal place for storing log files. I would personally recommend to use a separate drive for logfiles so avoid it from flooding any OS or Program related drives. You could pick a small size based on the expected log size and if needed increase the amount of logs that are stored and the size of the log file.

Changing this is pretty simple. Open “vpxd.cfg” and add the following line in between <log> and </log>

<directory>D:\VMware\Logs</directory>

Changing the amount of log files stored and the size is also pretty basic, in this example vCenter will store 10 logfiles which are max 10MB each:

<maxFileSize>10485760</maxFileSize>
<maxFileNum>10</maxFileNum>

Keep in mind that you will need to restart the vCenter Service after these changes before they take effect!

Best Practices: running vCenter virtual (vSphere)

· ·

Yesterday we had a discussion on running vCenter virtual on one of the internal mailinglists. One of the gaps identified was the lack of a best practices document. Although there are multiple for VI3 and there are some KB articles these do need seem to be easy to find or complete. This is one of the reasons I wrote this article. Keep in mind that these are my recommendations and they do not necessarily align with VMware’s recommendations or requirements.

Sizing

Sizing is one of the most difficult parts in my opinion. As of vSphere the minimum requirements of vCenter have changed but it goes against my personal opinion on this subject. My recommendation would be to always start with 1 vCPU for environments with less than 10 hosts for instance. Here’s my suggestion:

  • < 10 ESX Hosts
    • 1 x vCPU
    • 3GB of memory
    • Windows 64Bit OS(preferred) or Windows 32Bit OS
  • > 10 ESX Hosts but < 50 ESX Hosts
    • 2 x vCPU
    • 4GB of memory
    • Windows 64Bit OS(preferred) or Windows 32Bit OS
  • > 50 ESX hosts but < 200 ESX Hosts
    • 4 x vCPU
    • 4GB of memory
    • Windows 64Bit OS(preferred) or Windows 32Bit OS
  • > 200 ESX Hosts
    • 4 x vCPU
    • 8GB of memory
    • Windows 64Bit OS(requirement)

My recommendation differ from VMware’s recommendation. The reason for this is that in small environments(<10 Hosts) there’s usually more flexibility for increasing resources in terms of scheduling down time. Although 2 vCPUs are a requirement I’ve seen multiple installations where a single vCPU was more than sufficient. Another argument for starting with a single vCPU would be “Practice What You Preach”. (How many times have you convinced an application owner to downscale after a P2V?!) I do however personally prefer to always use a 64Bit OS to enable upgrades to configs with more than 4GB of memory when needed.

vCenter Server in a HA/DRS Cluster

  1. Disable DRS(Change Automation Level!) for your vCenter Server and make sure to document where the vCenter Server is located (My suggestion would be the first ESX host on the cluster).
  2. Make sure HA is enabled for your vCenter Server, and set the startup priority to high. (Default is medium for every VM.)
  3. Make sure the vCenter Server VM gets enough resources by setting the shares for both Memory and CPU to “high”.
  4. Make sure other services and servers on which vCenter depends are also starting automatically, with a high priority and in the correct order like:
    1. Active Directory.
    2. DNS.
    3. SQL.
  5. Write a procedure to boot the vCenter / AD / DNS / SQL manually in case of a complete power outage occurs.

Most of these recommendations are pretty obvious but you would be surprised how many environments I’ve seen where for instance MS SQL had a medium startup priority and vCenter a high priority. Or where after a complete power outage no one knows how to boot the vCenter Server. Documenting standard procedures is key here; especially know that with vSphere vCenter is more important than ever before.

Source:
http://kb.vmware.com/kb/1009080
http://kb.vmware.com/kb/1009039
ESX and vCenter Server Installation Guide
Upgrade Guide

VCDX Tips from VCDX 001 John Arrasjid

· ·

John Arrasjid created a Twitter account and has been tweeting VCDX Tips over last week or so. These tips are valuable for everyone doing the VCDX Track and going for the defense. That’s  the reason for archiving them on a blog post as tweets are bound to get lost in space while a blog post gets indexed by google. This is the list of Tips for you VCDX Defense / Application Form so far, but knowing John he will keep updating it for a while and so will I:

  1. Know you design inside out. Why, how, and impact are important! Understand the requirements, justification, and consequences.
  2. a) Consider lessons learned from Enterprise Architecture and Services Oriented Architecture for virtual cloud design.
    b)     See Thomas Erl books (SOA Design Patterns, and SOA Principles of Service Design), and Computing by Rittinghouse and Ransome.
    note: books=support of cloud architecture design 4 VCDX, not to help pass test. SOA is a key piece for VMware based clouds.
  3. Practice what you preach and learn from others. Architects listen first. Don’t assume the answer before the discussion starts!
  4. Scenarios for VCDX defenses test journey to solution, not necessarily the final answer. Whiteboard, talk and ask questions.
  5. Troubleshooting scenarios – think of the architecture and implementation approach to resolution. Logs, design, SC commands.
  6. a) Application/Design Choice=(What decision was required? Don’t include simplistic choices that are standard ie: redundant NICs)
    b)     Include architecture decisions like use of active/standby layout in single vswitch design.
  7. Application/Justification=What compelling requirement/constraint/assumption/risk mitigation drove decision.
  8. Application/Impact=How did the decision affect other areas of the design w/r to technology/risk/schedule/skills/budget.
  9. Recruit peers to review VCDX design and application. Focus on architects and specialists 1st, implementers 2nd. Red ink please.
  10. Try a dry run on the defense. 75 minutes w/peers to grill you on design and choices made. Practice breeds confidence!
  11. a) Longer is not necessarily better. We have reviewed some designs over 500 pages total (8 docs) and some under 200 pages total.
    b)     Focus on content not length. Focus on customer requirements.
  12. a) What makes for a good design?
    b)     See other tips. Key answer = meeting requirements with appropriate technology to also match schedule and budget.
    c)     Understanding and documenting the reasons for each decision including meeting requirements, constraints, and technology.
  13. a) Blank application or missing areas in the application=immediate rejection. Application is a requirement.
    b)     Incomplete application form or referring the reviewer to the document where the form should be completed=rejection.
    c)     Application requires 5 requirements/constraints/assumptions. and four answers (3 parts each) in each design choice area.
  14. a) Core components of a VCDX submitted design include a design document with requirements and an architectural solution.
    b) This design will include supporting information for the design choices. This is why the application mirrors what we look for.
    c) The design also should include an assembly and configuration type document and a validation/test plan.
    d) In the design you must touch the following areas: Network, storage, vCenter, ESX, VMs, security, disaster recovery, monitoring, upgrades, capacity planning, standard procedures, and areas related to the requirements.
  15. Current docs must be in English. Other languages will come as we grow the number of VCDX. If u translate, consider a reviewer
  16. Start early on application. Plan 8-24 hours to collect and update docs per blueprints and fill out app. Remember ‘no blanks!’
  17. Use VCDX application 2 identify new design patterns & new best practices in your design. Provide justification & impact.
  18. Considerations for design: SOA, Cloud, Enterprise Architecture, extending the boundaries. Think Elastic Sky… 🙂
  19. a) Example decision for ESX Clusters: N+2 Redundancy
    b)     Justification: Customer requirement of N+2 redundancy in cluster.
    c)     Impact: Lower utilization of resources in cluster to accommodate N+2 failure. Impact to ROI & TCO.
  20. Design Document must include requirements and an implementation to support them. It is not about diagrams alone.
  21. Validation/Test Plan typically will include test of virtualization platform/mgmnt tests, infrastructure, and adv features.
  22. Design Document includes conceptual model mapping requirements, logical design, and physical design.
  23. Installation & Configuration Guide includes details on assembly of the physical design components. Test Plan can be combined.
  24. Preparation for defense – Create presentation outlining design and include reference diagrams for discussion during defense.
  25. Getting started – take a design and red ink pen, then review for blueprint items and a match of decisions to requirements.
  26. Sleep well night before after studying design and decisions carefully, then take the test with confidence, best foot forward.
  27. Consider extensions to design to meet all blueprint requirements. This is allowed but must be defensible.
  28. a) Plan to include requirements/constraints/assumptions in a dedicated document or within the architecture design document.
    b) Requirements table example: REQ-001 Customer requires that disaster recovery site supports the same SLAs as primary site.
    c) Constraint table example: CON-001 Customer has budget to support recovery but not at the same SLA levels.
    d) Assumptions table example: To quote Benny Hill “Never assume or you make an *** out of you and me.” Note follows…
    e) Assumptions must be confirmed for risk mitigation.
    f) Assumptions table: ASS-001 To mitigate risk, REQ-001&CON-001, client resolves conflict. The design will match resolution.
    g) All architects will, at some point, have conflicts in pool of requirements/constraints/assumptions.
    h) A VCDX must be able to address conflicts in their design during discussions and obtain signoff of the final decision.
    i) Ensure that the decisions made around conflicts include justification, approval, and impact to other areas of the design.
  29. Validation/test plans should include functionality/performance/utilization considerations for testing on regular basis.
  30. Panelists spend approximately 4-8 hours reviewing application and submitted docs. We look for complete design as specified.
  31. Design documents do not have to be based on VMware deliverables but must meet content coverage. Blueprints & app provide detail.
  32. Having a good design without understanding all areas presented will limit scoring. Know the technology as you will be asked.
  33. Short answers without solid demonstration of technology or choices must be expanded to score.
  34. Panelists cannot provide positive scores in missing areas. Explaining what is missing does NOT score extra points.
  35. Cheat sites for vcdx won’t help.
  36. (@ywiskere) LISTEN to panel. When they ask to explain how you came to the design, do that and don’t just explain the facts.
  37. Sites that introduce design strategies and design patterns for virtualization that are supported by VMware will help.
  38. Liars claiming to be VCDX will be revealed as liars. Customers beware. If they lie on this then they will lie to you.
  39. HP blade virtual connect can utilize redundant back-end 10G uplinks, allowing creative perf and redundancy in vswitch design.
  40. Official VCDX certified individuals will, in the future, be listed by VMware so that customers can validate integrity.40. If the design references strategies from Mr. Hanky, do NOT submit. 🙂 (South Park anyone?)
  41. Don’t believe duplicate tip #s. Go with first one. 🙂
  42. “Panelists are experts in various virtualization areas. Don’t make technology claims that are incorrect.” – Duncan Epping
  43. Submitting and defending a design with “best practices”, know the background behind the best practice. You will be questioned!
  44. Example of bad design – Utilizing two NICs for vSphere FT that are shared with vMotion, Service Console, and VM traffic. BAD!
  45. Best practices for majority may not be best practice for some. Think first! Justify & show impact to support change.
  46. For partners especially – default VMware templates are only a baseline and do not constitute the minimal cutoff for VCDX.
  47. Customer defaults may be ok, but provide guidance in your discussions as well as recommendations for change in your design.
  48. Reminder – ‘Just because’ will cost points. Know justifications & impact of choice. If bad, provide recommendations.
  49. Defense presentation outlines your background, project background, key points, and reference materials.
  50. VCDX Presentation should not include birthday, age, religion, or other areas that are outside of virtualization project.
  51. a) vSphere designs have been accepted for the VCDX defense if they meet the criteria defined in the blueprints.
    b) The differentiator between VCDX on VI3.x versus VCDX on vSphere 4.x is likely to be based on the test.
    c) Note – for VCDX on vSphere 4 to work, the Enterprise test and the Design test must both be upgraded to vSphere 4. ETA TBD.
    d) All aspects of a design are reviewed and validated. If you include vSphere features, you WILL be questions on them also.
  52. Enterprise Exam is a prerequisite to validate advanced skills and troubleshooting required by architects. Experience counts!
  53. Design exam tests architecture skills that include both technology and process. How do multiple parts fit together?
  54. Top VCDX candidates spent extra time on their application and design to fill in missing areas for complete scoring.
  55. Designs can adapt and modify best practices. Provide solid support for this and provide examples from experience.
  56. Design Exam – consider lessons learned from VCP and Enterprise tests. Know your weak spots and learn them.
  57. Design Exam – Integration is essential in a design. Know how to tie related items and understand their relationships.
  58. D.E. – covers design exam tips for future posts.
  59. D.E. Consider security, DR, DRS, Resource Pools, and other items that influence placement of disk/net/CPU/RAM/VM in cluster.
  60. D.E. (Tip from @jasonboche) – Pace yourself. Some areas require more time than others. Some have more weight than others.
  61. I plan on adding to the tips. If you have unique questions on VCDX I will post answers.
  62. Troubleshooting scenarios include multiple paths that can be taken. Think of this as a murder mystery role play. Journey=#1.
  63. Variation on an idea from Hugo Strydom “A VCDX should have a balanced lifestyle; family, training, exercise, work.”
  64. Add to tip 63 – Maintain a good mental state when working/presenting. Clear thinking in difficult/stressful situations.
  65. a) If planning on doing February VCDX defense at PEX, start planning now. Think of 4 major decisions made in each category.
    b)     Storage, Networking, Cluster, Resources, Security, Provisioning, BC/DR, next steps, and other areas tied to design.
    c)     Include the full design details, the justification on why a decision was made, and the impact as described previously.
    d)     Create presentation with the topology map(s), provide an overview of the design, and then include support info for defense.
    e)     Based on feedback, the VCDX Preparation Workshop will likely be offered at PEX. Mock defenses will be help.
    f) Prerequisites include VCP, pass on Enterprise Test, and acceptance into program. Completion of design test gives priority.
  66. Get through the Design Test asap if you plan on submitting for a defense at Partner Exchange in February.
  67. For those asking about testing at PEX 2010, applications will be available in ~2 weeks.
  68. Update on PEX2010 – Development of the workshop is close to finished. Two hour sessions on Monday. Outline coming.
  69. Free 1/2 day VCDX workshops next month (Sydney/Melbourne/Singapore). Contact me offline. Space limited to partners.
  70. The VCDX Preparation Workshop has been approved for Partner Exchange ’10. Sunday from 1-3pm, February 7th.
  71. If you are coming to PEX 2010, the VCDX workshop is on Sunday and VCDX Panel Defenses are Monday and Friday.Ensure that you have 5 each of requirements, constraints, and assumptions.
  72. Ensure you have 4 design decisions for each of these categories: Storage, Network, Clusters, BCDR, Security, VM Config, Updates, Dependencies, Monitoring, Implementation & Next Steps, Installation Procs, Standard Procs such as P2V and VM Deployment and Testing/Validation. Also include two risks.
  73. For decisions, be complete and do not use acronyms. Focus on the outline of the decision, the design choice, the justification, and the Impact to other areas of the design and the project. Do not list the justification under the impact. Think about how
  74. The decision affects other design choices, budget, scheduling, technology choices, etc. Be as complete as possible.
  75. When you get the application, start moving forward with completing it per my previous posts. Time is of the essence!
  76. The application form specifies for item 7 “Disaster Recovery and Fault Tolerance Considerations”. It does NOT say VMware FT!
  77. VMware FT could be a consideration for a vSphere deployment maybe but the application specifies very clearly Fault Tolerance !FT.
  78. Please use full sentences and check for grammar when submitting your application for clarity when the panel reviews.
  79. VCDX Panel Defenses at PEX’10 – If you need to rework your application or design, you will have until January 15th I believe.
  80. Times up. End of day today is the deadline for VCDX Application rework for Partner Exchange defenses.
  81. Remember that the application must be 100% filled out. Justification != Impact. Application does not replace documentation.
  82. Based on feedback we will provide example ‘Table of Contents’ for the documentation required.
  83. VMware standard for design docs include: Architecture Design Document, Validation/Test Plan, Implementation Guide, (cont.)
  84. (cont) Blueprints (Visio), Next Steps, Operations Guide, and other docs where relevant to the project.
  85. VCDX submitted docs missing areas that are in the application cannot be scored on those areas, thus risking loss of points.
  86. Applications with incomplete sentences, acronyms without definition, same justification & impact entries risk rejection.
  87. Best wishes for those testing at Partner Exchange! Best wishes for those testing at future events.
  88. Upcoming VCDX testing anticipated for Munich and Sydney in Q2, and at VMworld 2010 and VMworld EMEA 2010.
  89. Those translating their design documents to English, please have someone validate the technical translation.
  90. bad translation example: “The ESX waiter will be implemented on a four-way quad-core waiter.” (Not all translators are equal)
  91. Practice defending your design in advance. Pick someone that will dig deep on your design choices and the docs you submit.
  92. Once again, justification does not equal impact. Impact can be both positive and negative.
  93. PEX2010 VCDX Preparation Workshop booked solid. Just opened more seating. Those on waiting list should get in.
  94. a) Rework of a VCDX application has a deadline to meet VMware req. If rework does not meet requirements, then it is rejected.
    b) Given that we were overloaded with resubmissions by candidates we must limit future candidates to one resubmit of their app.
    c) Why? Because of the time required to review an application and design documents.
    d) Same applies for resubmit of a design that does not meet all requirements. Be sure to reread instructions before resubmit.
  95. VCDX documentation requirements reflect VMware’s stance on what is required for fully delivering a design.
  96. Partner Exchange 2011 is the final chance to become a VCDX3

Who said twitter is a waste of time? There’s valuable info to be found there!