Sponsored by Google, this article is the third ina three-part series on CI/CD with Kubernetes.
Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.
GitHub has changed the way development teams manage code changes, bringing in improved collaboration from “commit” all the way to “merge.” Having started with Jenkins-centric continuous integration, many organizations are now trying to optimize and improve on sub-optimal pipelines by taking a closer look at their GitHub workflows.
Weaveworks, the organization that originally coined the term GitOps describes it as “ Operations by Pull request .” The most valuable part of GitOps is that it enables “environments as code,”which means that GitOps lets you define policies and environment configuration alongside your code.
While there’s a lot to love about this approach, there are also aspects of GitOps that are not necessarily suitable in an enterprise context. Elaborating on this, Andrew Philips , a product manager in Google Cloud Platform’s DevOps division explains that there are primarily two challenges with common implementations of GitOps: “First, running your release processes and approvals largely through source code repositories isn’t ideal in larger organizations. Second, it’s not always best to choose a ‘pull’ approach to make environment as code real; a ‘push’ approach can be more suitable in many cases.” These issues crop up with organizational complexity, and this is where simple implementations of GitOps fall short.
As a result, it is imperative to retain the best parts of GitOps and replace certain parts of it with alternatives more suited to your context. In this post, we discuss a better way to think about GitOps with Kubernetes so that you can apply a set of workable patterns, keeping in mind your own organization’s complexities and constraints.Dealing with Enterprise Complexity
Large organizations typically have multiple development teams each with its own unique practices. They must empower each team with the necessary autonomy to build, deploy and maintain the code and applications the team owns in order to maintain velocity and quality. At the same time, organizations must prioritize and plan for the scaling of their applications and clusters.
There are two aspects to scaling a CI/CD pipeline. The first is when your organization scales to include new teams and new team members. The second is when your application’s usage grows and needs to be supported by additional resources. Additional Kubernetes clusters and larger environments mean more complexity and a greater risk of failure during rollout. The more your application scales the greater the cost of a failure becomes. There is more at stake for an application in production that has a million queries per second than one that has just ten.
These two goals of autonomy and scale seem to be at odds with each other. This is where GitOps fits, as it benefits both without sacrificing one or the other.
Cloud-native DevOps features like Continuous Integration/Continuous Delivery (CI/CD), containers, and container orchestration can accelerate time-to-market and reduce infrastructure & ops burden. Get an overview on DevOps and CI/CD workflows past and present in the free “CI/CD with Kubernetes” e-book, sponsored by Google.
“GitOps provides a set of best practices to organize code and configuration for applications and environments in an automated way,” Dan Lorenc , software engineering manager at Google Cloud, said.
If infrastructure as code was centered around instances, GitOps centers around environment configuration. In a GitOps model, all changes and updates to your production environment happen through changes to code repositories. Instead of manually configuring an environment, GitOps enforces changes to the environment as a by-product of code changes. This way, GitOps creates a link between your repositories and environments and enables “environments as code.”
“The power of integrating repositories and environments is that you can now extend your access controls, testing policies and authorization from your code to your environment configuration,” Lorenc said. This helps to unify your CI/CD pipeline while still enforcing development patterns that adapt to your organization.One Repository Per Team
GitHub makes it possible to maintain and deploy from multiple code repositories, but with every additional repository, the management overhead increases. In an organization with multiple development teams, the dilemma is whether to create as many repositories as possible or fewer ones. In a scenario with multiple repositories, each team would have separate repositories for development, staging and production. This results in the problem of repository sprawl.
Google’s Andrew Philips recommends an alternative. He advises, “As far as possible, it makes sense to have fewer repositories than many. Instead of creating additional repositories you could create three branches within that repository for each environment ― for example, one branch for dev, one for staging and another for prod. This way, each environment has its own commit history and config, so you’ll still have the benefits of isolation while avoiding the maintenance overhead.”
It’s often the case that organizations end up with a very granular and unnecessarily complex system that is easy to create but hard to maintain. Phillips’ advice goes against this trend and offers a pragmatic approach to managing and scaling code repositories.
By following this model, you’ll likely end up with one repository per team, which is better than multiple repositories for every team, but in a large organization with many teams, this is still hard to manage. Taking the idea of fewer repositories further, you can have a shared repository for each environment. For example, a staging cluster that five teams deploy to can have its config hosted in a single repository. This way you get the best of both worlds ― separation of concerns and ease of management.‘Push’ Over ‘Pull’
Another important consideration with GitOps is how runtimes are synced when there are numerous developers contributing code multiple times a day. GitOps is often described as operating on a “pull model” in which a developer opens a pull request, someone approves the request, and after the merge a process running in the live cluster pulls in the changes. This is problematic as the team becomes aware of failures only after the deployment is complete. When this happens teams can roll back the commit, but it gets recorded in the commit history which is then no longer a clean record of good states.Another issue with relying on a pull model to auto