This blog post contains the slides along with a loose transcript and additional resources from my technical talk on DevOps and Continuous Delivery concepts given at my alma mater, the University of Virginia, to the M.S. in Management of Information Technology program on November 2nd and 4th of 2017.
Links to learn more about the concepts presented in this talk can be found in the sidebar and at the bottom of this page.
Hey folks, my name is Matt Makai. I am a software developer at Twilio and the creator of Full Stack Python, which over 125,000 developers read each month to learn how to build, deploy and operate Python-based applications.
You've talked about using the Agile software development methodology on your teams, but what's the purpose? Why does Agile development matter to you and your organization?
Agile matters because it allows you to ship more code, faster than traditional "waterfall" methodology approaches.
Shipping is a common allegory in software development nowadays, because code that is not in production, in the hands of your users, doesn't create value for anyone.
If code is not running in production, it's not creating value. New code created by your Agile development teams every couple of weeks does not create more value until it is executing in production.
Shipping code is so important to high functioning companies that the maritime theme is used across all sorts of projects, including in the Docker logo.
As well as in the Kubernetes logo in the form of a ship steering wheel.
Here is a super high-level diagram of the ideal scenario we need for Agile development teams. Create working code and get it shipped as soon as possible into production.
Facebook's internal motto used to be "Move fast and break things." They thought that if you aren't breaking things then you aren't moving fast enough.
And eventually if you're constantly shipping to production and you do not have the appropriate processes and tools in place, your applications will break. The breakage has nothing to do with the Agile methodology itself.
Your team and organization will come to a fork in the road when you end up with a broken environment.
Traditionally, organizations have tried to prevent breakage by putting more manual tools and processes in place. Manual labor slows... down... your... ability... to... execute.
This is one path provided by the fork in the road. Put your "Enterprise Change Review Boards" in place. Require production sign-offs by some Executive Vice President who has never written a line of code in his life. Put several dozen "technical architects" in a room together to argue over who gets to deploy their changes to production that month.
The manual path is insanity. Eventually the best developers in your organization will get frustrated and leave. Executives will ask why nothing ever gets done. Why does it take our organization three years to ship a small change to a critical application?
Some development teams try to get around the manual production challenges by shipping everything to a development environment. The dev environment is under their control.
But what's the huge glaring problem in this situation?
If you are not shipping to production, then you are not creating any value for your users. The teams have made a rational decision to ship to development but the organization still suffers due to the manual controls.
The problems we are talking about are created by the Agile methodology because they become acute when your development team is producing code at high velocity. Once code is created faster, you need a way to reliably, consistently put the code into production so that it can create value for its users.
DevOps and Continuous Delivery are the broad terms that encompass how to reliably ship code to production and operate it when the code is running in production.
We are going to use the terms "DevOps" and "Continuous Delivery" a lot today, so let's start by defining what they mean. In fact, the term "DevOps" has already accumulated a lot of buzzword baggage, so we'll start by defining what DevOps is not.
First,DevOps is not a new role. If you go hire a bunch of people and call them "DevOps engineers" then sit them in the middle of your developers and system admin/ops folks, you are going to have a bad time. You just added a new layer between the two groups you need to pull closer together.
Second, DevOps is not a specific tool or application. You do not need to use Docker or Puppet to do DevOps in your organization. The processes that make DevOps work are made much easier by some tools such as cloud platforms where infrastructure is transient, but even those platforms are not required to do DevOps right.
Third, DevOps is not tied to a specific programming language ecosystem. You do not need to use Node.js or Ruby on Rails. You can still use DevOps in a COBOL- or J2EE-only organization.
With those misconceptions out of the way, let's talk about what DevOps IS. First, at the risk of being way too obvious, DevOps is the combination of the two words Development and Operations. This combination is not a random pairing, it's an intentional term.
Second, DevOps means your application developers handle operations. Not necessarily all operations work, but ops work that deals with the code they write and deploy as part of their sprints. The developers also will likely become intimately familiar with the underlying infrastructure such as the web application servers, web servers and deployment code for configuration management tools.
Third, DevOps allows your organization to be more efficient in handling issues by ensuring the correct person is handling errors and application failures.
We are not going to go through Continuous Delivery (CD) by defining what it is not, but there are a couple bits to say about it. First, CD is a collection of engineering practices aimed at automating the delivery of code from version control check-in until it is running in a production environment.
The benefit of the automation CD approach is that your organization will have far greater confidence in the code running in production even as the code itself changes more frequently with every deployment.
Facebook's original motto changed a few years ago to "Move Fast and Build Things" because they realized that breaking production was not a byproduct of moving fast, it was a result of immature organizational processes and tools. DevOps and Continuous Delivery are why organizations can now deploy hundreds or thousands of times to production every day but have increasing, not decreasing, confidence in their systems as they continue to move faster.
Let's take a look at a couple of example scenarios that drive home what DevOps and CD are all about, as well as learn about some of the processes, concepts and tools that fall in this domain.
Here is a beautiful evening picture of the city I just moved away from, San Francisco.
The company I work for, Twilio is located in San Francisco. If you ever fly into the SFO airport and catch a ride towards downtown, you will see our billboard on the right side of the road.
Twilio makes it easy for software developers to add communications, such as phone calling, messaging and video, into their applications. We are a telecommunications company built with the power of software that eliminates the need for customers to buy all the expensive legacy hardware that they used to have to acquire. As a telecomm company, we can never go down, or our customers are hosed and then our business is hosed.
However, we have had challenges in our history that have forced us to confront the fork in the road between manual processes and moving faster via trust in our automation.
In August 2013, Twilio faced an infrastructure failure.
First, some context. When a developer signs up for Twilio, she puts some credit on their account and the credit is drawn upon by making phone calls, sending messages and such. When credit runs low we can re-charge your card so you get more credit.
There was a major production issue with the recurring charges in August 2013. Our engineers were alerted to the errors and the issue blew up on the top of Hacker News, drawing widespread atttention.
So now there is a major production error... what do we do?
(Reader note: this section is primarily audience discussion based on their own experiences handling these difficult technical situations.)
One step is to figure out when the problem started and whether or not it is over. If it's not over, triage the specific problems and start communicating with customers. Be as accurate and transparent as possible.
The specific technical issue in this case was due to our misconfiguration of Redis instances.
We know the particular technical failure was due to our Redis mishandling, but how do we look past the specific bit and get to a broader understanding of the processes that caused the issue?
Let's take a look at the resolution of the situation and then learn about the concepts and tools that could prevent future problems.
In this case, we communicated with our customers as much about the problem as possible. As a developer-focused company, we were fortunate that by being transparent about the specific technical issue, many of our customers gained respect for us because they had also faced similar misconfigurations in their own environments.
Twilio became more transparent with the status of services, especially with showing partial failures and outages.
Twilio was also deliberate in avoiding the accumulation of manual processes and controls that other organizations often put in place after failures. We doubled down on resiliency through automation to increase our ability to deploy to production.
What are some of the tools and concepts we use at Twilio to prevent future failure scenarios?
If you do not have the right tools and processes in place, eventually you end up with a broken production environment after shipping code. What is one tool we can use to be confident that the code going into production is not broken?
Automated testing, in its many forms, such as unit testing, integration testing, security testing and performance testing, helps to ensure the integrity of the code. You need to automate because manual testing is too slow.
Other important tools that fall into the automated testing bucket but are not traditionally thought of as a "test case" include code coverage and code metrics (such as Cyclomatic Complexity).
Awesome, now you only deploy to production when a big batch of automated test cases ensure the integrity of your code. All good, right?
Err, well no. Stuff can still break in production, espcially in environments where for various reasons you do not have the same exact data in test that you do in production. Your automated tests and code metrics will simply not catch every last scenario that could go wrong in production.
When something goes wrong with your application, you need monitoring to know what the problem is, and alerting to tell the right folks. Traditionally, the "right" people were in operations. But over time many organizations realized the ops folks ended up having to call the original application developers who wrote the code that had the problem.
A critical piece to DevOps is about ensuring the appropriate developers are carrying the pagers. It sucks to carry the pager and get woken up in the middle of the night, but it's a heck of a lot easier to debug the code that your team wrote than if you are a random ops person who's never seen the code before in her life.
Another by-product of having application developers carry the "pagers" for alerts on production issues is that over time the code they write is more defensive. Errors are handled more appropriately, because otherwise you know something will blow up on you later on at a less convenient time.
Typically you find though that there are still plenty of production errors even when you have defensive code in place with a huge swath of the most important parts of your codebase being constantly tested.
That's where a concept known as "chaos engineering" can come in. Chaos engineering breaks parts of your production environment on a schedule and even unscheduled basis. This is a very advanced technique- you are not going to sell this in an environment that has no existing automated test coverage or appropriate controls in place.
By deliberately introducing failures, especially during the day when your well-caffeinated team can address the issues and put further safeguards in place, you make your production environment more resilient.
We talked about the failure in Twilio's payments infrastructure several years ago that led us to ultimately become more resilient to failure by putting appropriate automation in place.
Screwing with other people's money is really bad, and so is messing with people's lives.
Let's discuss a scenario where human lives were at stake.
To be explicit about this next scenario, I'm only going to talk about public information, so my cleared folks in the audience can relax.
During the height of U.S forces' Iraq surge in 2007, more improvised explosive devices were killing and maiming soldiers and civilians than ever before. It was an incredible tragedy that contributed to the uncertainty of the time in the country.
However, efforts in biometrics were one part of the puzzle that helped to prevent more attacks, as shown in this picture from General Petraeus' report to Congress.
One major challenge with the project was a terrible manual build process that literally involved clicking buttons in an integrated development environment to create the application artifacts. The process was too manual and the end result was that the latest version of the software took far too long to get into production.
We did not have automated deployments to a development environment, staging or production.
Our team had to start somewhere, but with a lack of approved tools, all we had available to us was shell scripts. But shell scripts were a start. We were able to make a very brittle but repeatable, automated deployment process to a development environment?
There is still a huge glaring issue though: until the code is actually deployed to production it does not provide any value for the users.
In this case, we could never fully automate the deployment because we had to burn to a CD before moving to a physically different computer network. The team could automate just about everything else though, and that really mattered for iteration and speed to deployment.
You do the best you can with the tools at your disposal.
What are the tools and concepts behind automating deployments?
Source code is stored in a source control (or version control) repository. Source control is the start of the automation process, but what do we need to get the code into various environments using a repeatable, automated process?
This is where continuous integration comes in. Continuous integration takes your code from the version control system, builds it, tests it and calculate the appropriate code metrics before the code is deployed to an environment.
Now we have a continuous integration server hooked up to source control, but this picture still looks odd.
Technically, continuous integration does not handle the details of the build and how to configure individual execution environments.
Configuration management tools handle the setup of application code and environments.
Those two scenarios provided some context for why DevOps and Continuous Delivery matter to organizations in varying industries. When you have high performing teams working via the Agile development methodology, you will encounter a set of problems that are not solvable by doing Agile "better". You need the tools and concepts we talked about today as well as a slew of other engineering practices to get that new code into production.
There are many other practices you will need as you continue your journey. You can learn about all of them on Full Stack Python.
Additional resources to learn more about the following topics can be found on their respective pages:
Fix errors in your Python code before your users see them by monitoring with Rollbar.
Easily build maps, search and navigation into your Python applications with Mapbox.
Scout monitors the performance of your Python apps, identifying slow queries, memory bloat, and more. Free during Tech Preview.
Upgrade your Python skills by reading Real Python's awesome programming email newsletter.