Scaling & Eliminating Waste in Your DevOps Environment
by Tony Albanese
March 28, 2014
| 7 min read
This is the third post in a series to help your engineering team transition into a DevOps organizational model. Here we’ll discuss how to scale and eliminate waste from your engineering environment. Click here to start at the beginnings of the series, Why You Need to Establish a DevOps Culture.
Eliminating waste from your processes highly relies on an understanding of what your operations people need and what your developers need. Having this granular understanding will help close the gaps between these two teams and make them one. As every team differs, we can’t tell you exactly what these needs are. But we can help you form the strategy to tackle these challenges within your culture.
Find Out What Everyone Really Wants
You will need to establish a shared vision between your developers and operations team that aligns with your company’s values and mission. To do this you will want to open up a dialogue between key stakeholders on each team. You will not want to make assumptions for what the other team needs or generalize their pain. Doing so, you’ll misinterpret each others’ organizational needs.
Once you get the conversation going you’ll be able to find some common ground. Instead of focusing on conflicting interest, identify your common goal and start to align teams for a singular purpose. Once you are setting up a plan to obtain a common goal you’ll most likely find that you don’t have many, if any, conflicting interests. After all, you all work for the same company with a shared mission and values.
Use the Scientific Method
We’ve all learned about the scientific method in grade school. But its core principles are easily forgotten when your deadlines are looming.
Plan. Transitioning to a DevOps model is a high level plan. In order to scale and eliminate waste during your transition you will need to focus your efforts on a few areas at a time. Common areas that may be causing problems in your organization are communication, tooling, expertise or team skepticism. You will want to identify which of these common problems are negatively impacting your team and start thinking about how to fix them. It would be beneficial to grab some initial measurements that you will be able to compare later. Even if you don’t have systems in place for measurements now, any ad hoc measurements you can take will be helpful to compare to later.
Do. Now it’s time to start doing stuff. Take what you have hypothesized during the planning phase and take a leap. For example, if you hypothesized that using continuous integrations in your development and product environments will decrease the time it takes to deploy new code, then it is time to implement your continuous integration system. Or if you learned that your team is lacking in specific skills in your initial assessment, you may need to hire (or contract) people with the expertise required.
Measure Results. After you’ve implemented your new tool you will need to see if your changes made a difference. Is your development team still having issues while deploying new code? If so, has there been fewer issues, or is everything the same?
Respond. Using your metrics you can make a decision to act. If your plan didn’t work as you thought it work use the data you collected to adjust your hypothesis and try something else. Or amend what you have already implemented to address any new, unexpected challenges.
Continuously Improve. Finally, you will want to repeat this over and over for everything in your organization that needs to be addressed to create a DevOps culture. It won’t be quick or easy, you may not even see results right away, but it will be worth it.
Hire & Train with Your DevOps Culture in Mind
Finding the right people with the right mindset is crucial to scaling your DevOps culture. You will need to get your Human Resources team and your recruiters up to speed on the type of people you are looking for. You will want to hire people who thrive in or are excited to join a team with a DevOps philosophy. This means looking for people with both the specific skills you need for your team and the cultural values you are looking for. Though this may be tough, they’re out there.
When your new hires join your team, it’s imperative that you train and mold them to be a valuable player on your team and to work towards the common goal of achieving a DevOps culture. At PagerDuty, we inject failure into our systems every Friday. Not only is this a great way to test the resilience of our systems, but Failure Friday gets our devs and ops team in a room together allowing them to cross train and have a complete understanding of our processes beyond their designated role.
And we’re not alone. We recently spoke with our friends at Sumo Logic about how they utilize training to ramp up their new developers and acclimate them with their culture as many of them have not come from a DevOps environment and not used to being on-call.
SumoLogic will pair their new developers with seasoned employees to share on-call responsibility to assist with blurring the lines between the traditional developer and operations engineering roles. They utilize an immersive shadowing training during business hours until the developer is up to speed on both their system and their way of life.
It’s imperative that you include your DevOps culture into your on-boarding process as you scale because people are resistant to change. Not everyone is cut out for this. It’s stressful and it takes people out of their comfort zones. But if you are proactive in your approach and mold people early on in your organize you will be able to evolve your engineering culture into something to brag about.
Assign a DevOps Champion
Making the decision to enter into a cultural transition is only the first step. Committing to it is an entirely different and aggressive beast. To keep everyone on track you may want to assign a DevOps Champion to answer any questions and help make the transition as easy as possible.
Your appointed champion should make themselves available so anyone can ask questions or address concerns. Your DevOps Champion needs to personify your company’s goals and be an aid to everyone involved. They need be approachable, a good listen and someone who wants to make everyone feel valued and included. Champions are good communicators that can explain the benefits for internal teams of all sizes and the company at large.
Your champion should encourage pride and sense of ownership for all employees, helping to empower them. They should also implement a “no asshole rule” to help alleviate the tension during a transition and reduce any negative that may be present from disgruntled employees. It also helps if your champion isn’t an asshole themselves.
You can’t half-ass a transition to a DevOps culture, you need to go all in order to make a difference. Once you get everyone on board, many of your challenges will just be getting started. Keep an open mind, breathe and remember to keep everyones needs at the forefront of your mind, while solving issues logically and for the future.
Lastly, keep in mind that even for highly performing teams this won’t happen overnight. For organizations with over a thousand employees it may take years, with one hundred or more it may take a year, and even for teams with as few as 10 employees it can take several months to implement. Just remember it will be worth it.
Update 4/10/14 – Continuing reading the transitioning to DevOps series:
