Continuous Integration

Continuous integration automates the building, testing and deploying of applications. Software projects, whether created by a single individual or entire teams, typically use continuous integration as a hub to ensure important steps such as unit testing are automated rather than manual processes.

Why is continuous integration important?

When continuous integration (CI) is established as a step in a software project's development process it can dramatically reduce deployment times by minimizing steps that require human intervention. The only minor downside to using CI is that it takes some initial time by a developer to set up and then there is some ongoing maintainence if a project is broken into multiple parts, such as going from a monolith architecture to microservices.

Automated testing

Another major advantage with CI is that testing can be an automated step in the deployment process. Broken deployments can be prevented by running a comprehensive test suite of unit and integration tests when developers check in code to a source code repository. Any bugs accidentally introduced during a check-in that are caught by the test suite are reported and prevent the deployment from proceeding.

The automated testing on checked in source code can be thought of like the bumper guards in bowling that prevent code quality from going too far off track. CI combined with unit and integration tests check that any code modifications do not break existing tests to ensure the software works as intended.

Continuous integration example

The following picture represents a high level perspective on how continuous integration and deployment can work.

One potential way for continuous integration to work with source control and a deployment environment.

In the above diagram, when new code is committed to a source repository there is a hook that notifies the continuous integration server that new code needs to be built (the continuous integration server could also poll the source code repository if a notification is not possible).

The continuous integration server pulls the code to build and test it. If all tests pass, the continuous integration server begins the deployment process. The new code is pulled down to the server where the deployment is taking place. Finally the deployment process is completed via restarting services and related deployment activities.

There are many other ways a continuous integration server and its deployments can be structured. The above was just one example of a relatively simple setup.

Open source CI projects

There are a variety of free and open source continuous integration servers that are configurable based on a project's needs.

Note that many of these servers are not written in Python but work just fine for Python applications. Polyglot organizations (ones that use more than a single language and ecosystem) often use a single CI server for all of their projects regardless of the programming language the application was written in.

  • Jenkins is a common CI server for building and deploying to test and production servers. Jenkins source code is on GitHub.

  • Go CD is a CI server by ThoughtWorks that was designed with best practices for the build and test & release cycles in mind. Go CD source code is on GitHub.

  • Bazel is a build tool that works with CI tools to organize large code bases and provide consistency with a well-defined, automated build process.

  • BuildBot is a continuous integration framework with a set of components for creating your own CI server. It's written in Python and intended for development teams that want more control over their build and deployment pipeline. BuildBot source code is on GitHub.

  • TeamCity is JetBrains' closed source CI server that requires a license to use.

Jenkins CI resources

Jenkins is commonly used as a continuous integration server implementation for Python projects because it is open source and programming language agnostic. Learn more via the following resources or on the dedicated Jenkins page.

General continuous integration resources

What do you want to work on next for your deployment?

How do I log errors that occur in my application?

What can I do to mitigate security vulnerability in my web app?

What is Docker and how does it fit with Python deployments?

Matt Makai 2012-2022