Anatomy of Infrastructure Lifecycle: How Terrateam Enhances the DevOps CI/CD Pipeline
Overview
In today's software world, making great software is all about working smarter, not harder. One big part of that is DevOps - a fancy word for a set of practices that help us make software better and faster. At the core of this is CI/CD - a way to ensure our code is good and deliver it quickly. This philosophy of DevOps not only includes IAC (Infrastructure as a Code), but lots of other concepts as well, concepts that check for the potential security branches on the infrastructure you provide via IAC, and write your policy for your organization's best practices as well.
In this blog, we'll take you on a journey into the world of DevOps. We'll show you that Infrastructure as Code (IAC) is just a small part of how a successful organization operates, other aspects are Security checkers such as Checkov, policy maker and checker Open Policy Agent(OPA), and many more.
In this blog, we will consider three major tools that are Terraform, Checkov, and OPA(Open Policy Agent).
Let’s get started with the awesome tools that can help you with making your DevOps life easy.
Workflow
The workflow of a typical DevOps lifecycle goes as follows:
We start by writing Infrastructure as Code (IAC) and committing the code to a Git-based platform for collaborative access with other developers.
Next, we review the code to ensure it meets security and policy requirements with the help of Checkov.
Once the code is ready, we seek approval using OPA (Open Policy Agent).
After receiving approval, we proceed to the production environment.
Now first things first, what do we do as DevOps engineers at an organization to write and maintain our cloud-based infrastructure such as S3 buckets or provisioning Virtual Machines, to do so, most of us take help from tools such as Terraform to write the resource provisioning, Checkov to make the configuration secure, OPA to maintain the organization rules, these tools are mentioned below in a more elaborative manner.
IAC(Infrastructure as a Code)
- Let’s start with IAC(Infrastructure as a Code) tools such as Terraform, Pulumi, Ansible, AWS CloudFormation, Azure Resource Manager, Google Cloud Deployment Manager, Chef, Puppet, etc.
Statistics of IAC usage by organizations around the world.
The image is taken from here.
- The top 5 tools from the list are bound with a single cloud provider, which means they have a scope of just a single cloud provider and can help you manage only those cloud providers for which they are built. And later in the list are the IAC tools that can operate on multiple cloud platforms.
In this section, we'll discuss Terraform, which is renowned as the most popular tool for managing infrastructure across various cloud platforms.
- In 2023, more than 27,903 companies worldwide have started using Terraform as a tool to manage their configurations. The majority of these companies, around 12,495 of them, are located in the United States. That's about 41.50% of all Terraform users.
Terraform is like a tool made by people at HashiCorp. It's a great tool in the world of infrastructure handling. It helps us to use the codes to create and manage our infrastructure resources.
- Terraform can work on many cloud providers, such as AWS, Azure, and GCP this makes it “cloud-agnostic” which means you can use the same piece of configuration for all types of Cloud Providers(yes of course with some minor changes, such as the name of the resource, tags, and value blocks).
Security and Vulnerability Checks
- Next is to scan these configuration files using other DevOps tools such as Checkov, Code Climate Velocity, Sonatype Lift, Qwiet AI, GuardRails, Codacy, Moderne, Sider Scan, FossID, etc.
Here we will talk about one of the most common security scanning tools: Checkov
Checkov has been incredibly successful since its launch in December 2019, with over 1.2 million downloads by 2021. During that time, it expanded its policies to over 200, becoming the most comprehensive open-source IaC scanner. In 2023, it has amassed an astounding 82 million downloads.
With Checkov 2.0, a cutting-edge graph-based framework is introduced, allowing it to scan infrastructure as code (IaC) while considering dependencies in intricate distributed environments. It's worth noting that the latest available version is Checkov 3.0.
Checkov is like a safety inspector for the digital world. It checks the "blueprints" of your cloud-based buildings (like those made with Terraform) to make sure they're safe and follow the rules.
Key Features
It looks at different cloud systems like AWS, Azure, and Google Cloud.
It comes with a bunch of rules to check if your "blueprints" follow best practices.
You can even make your own rules to fit your needs.
It can be part of your CI/CD pipeline to check for problems automatically.
In brief, Checkov helps keep your digital creations such as infrastructure configurations safe and following the pre-written rules, which is super important in the world of DevOps.
Where Does Checkov Fit in the Workflow?
Checkov can provide the following things:
Check for vulnerabilities in your configuration files.
Helps in troubleshooting these vulnerabilities.
Check for potential security breaches such as opening the port CIDR of 0.0.0.0/0 for a highly confidential database, and provide these checks before they are applied to the infrastructure.
Therefore, Checkov majorly helps in maintaining the highest level of security possible on your infrastructure and helps in avoiding crashes and security breaches that might happen later in the future, it prevents your infrastructure from disasters in the future. Precaution is better than cure.
Rulebook/Policy Checks
- Next up is writing a custom policy to check if the infrastructure that you are building is up to the required trademark or not, and for that we are required to write a custom policy for each resource that is being made via Terraform configurations. For this, we have a bunch of DevOps tools such as OPA(Open Policy Agent), PowerDMS, Kofax Capture, iContracts, Mitratech PolicyHub, SecurityPal, Alcatel-Lucent ClearPass Policy Management System, Blissbook, DocRead and many more.
Here we will talk about OPA(Open Policy Agent) the most widely open-source Policy/Rulebook checker
Open Policy Agent (OPA) is like a rulebook for computer programs. It helps decide what actions are allowed or not allowed in software systems. You can write these rules in a language that's easy to understand.
OPA uses Rego as a rule writing language to verify each rule you write.
Here's what you should know about Rego:
It's simple and easy to read.
You write rules to say when something is okay or not.
It's good at understanding data and checking if it matches your rules.
It's safe and won't do anything harmful.
You can test and fix your rules to make sure they work.
Here's why OPA is cool
It can be used with different parts of your software.
It checks if actions or configurations follow the rules you've set.
You can change the rules as needed.
It works quickly, even with big systems.
You can reuse and share your rulebook.
People use OPA for things like deciding who can access what, making sure software is secure, and following rules for things like privacy or regulations. It's a handy tool to keep your software in check!
Complete DevOps Workflow
Let’s understand the workflow step-by-step.
Code
In this step of the workflow, we write all our configuration files including:
YAML files for pipeline run
Terraform configuration files for infrastructure provisioning
Other programming language files that are required for infrastructure policy checks such as Rego files, and Python files for Lambda function(if you are creating one).
All the files are pushed to a Git-based repository.
Git-repository
Here comes the most crucial part of a workflow.
Here we have to decide and use the best Git-based platform that is required by the organization to work upon such as Bit-Bucket, GitHub, GitLabs, and many more.
Each platform has its way of writing a CI/CD pipeline, so choose wisely.
-
Terrateam is one of the many plugins on GitHub, but this works like a helping hand if your daily job is writing Terraform configurations and running them through a GitHub actions pipeline.
Terrateam installed on your GitHub account helps to run a pipeline on each PR(Pull Request) raised on your main repository, and gives a complete overview of what will happen if you apply the configuration that is raised in the PR, and with just a single comment on the PR will apply the changes to the infrastructure.
Terrateam itself comes with a lot of plugins that you need to invoke via its workflow YAML file.
Terraform init
Those who use Terraform in their daily life, they understand this step very well.
Terraform works with its API to configure the infrastructure on cloud platforms, to be able to work with an API of its own Terraform requires downloading some files needed, and with the command “terraform init” Terraform downloads those files to help to connect to its APIs.
-
Since Checkov is a tool that finds security breaches and vulnerabilities in your Terraform configuration files, its check is crucial to take place before any other commands are executed.
To learn more about Checkov visit here.
Terraform plan
It lets you see what changes Terraform is going to make to your cloud setup before it makes them.
It checks your plans and tells you what will happen without actually doing anything.
It's like a safety net to catch any mistakes or unexpected changes.
You can use it to show others what you're planning to do before making changes.
-
- In this step of the workflow, OPA runs its Rego policies on the planned outputs and checks whether the requirements match or not.
Validate OPA results
Now a developer will validate if the results of the OPA policy check matches the requirement and rules set by the organization or not.
This step is manual work, once we get the green signal, we can move to the next step.
Staging
The next step is Staging, this step comes after the approval from the “Validate OPA results” step.
Staging goes by the name, and stages all the changes that are going to be made to the infrastructure of the organization, this includes any destruction that can be harmful or not to check whether any cost optimization is done on the configuration or not, and many more.
QA/Testing
This step comes after the approval from the “Staging” step.
Now we will test whether all the configuration changes are required or not, and do the newly created resources are cost-optimized or not.
Whether the destruction of any resource harms any other resources in its functionality.
If any step from 8,9,10 fails or finds any error, we go back to the Code part which is step 1, and follow the feedback from the Testing team and redo all the checks.
Terraform apply
- On getting all the approval from previous stages the step moves forward to apply all the changes to the infrastructure and see the changes in the production.
Production
- This is the real environment for which we are doing all the checks and getting everything cleared from each step, to get the best and optimized result for the production environment.
How does Terrateam fit in this Workflow?
Now the question is what does Terrateam do for us?
Here is the answer to the question!
Terrateam is like a helper for GitHub. It listens to GitHub's messages and does two main things:
It has a brain (backend) that understands the messages from GitHub and decides what to do.
It has a worker (Terrateam runner) that does the tasks the brain tells it to do.
So, Terrateam is like a teamwork tool for GitHub, with a brain and a worker.
In Terrateam, you can do two things with Terraform:
Plan
- This is like making a plan before you do something, for example before implementing the configuration to live we analyze if anything important is not destroyed, what new resources are going to be built also what previous resources are going to be updated, all this information is a part of the Plan.
Apply
- This is when you actually do the thing, after a successful and approved Plan we build/update/destroy infrastructure, this is part of the Apply.
Terrateam makes a difference between these two by calling them "Plan" and "Apply" when you do them yourself, and "Autoplan" and "Autoapply" when they happen automatically because of something you did, like making changes or updates.
So, Terrateam helps you make plans and do things with Terraform, whether you start them or they happen automatically by triggering the pipeline.
Terrateam has all the tools as a plugin, we just need to invoke each tool whenever needed via a Terrateam workflow file, tools include Checkov, OPA, Terragrunt, and many more.
To learn more about how to do so, visit here.
Conclusion
In conclusion, DevOps is an essential process to develop a good Infrastructure and can lead to building a great software system can be built on these infrastructures, DevOps and its tools build a basic building block on which robust software can be built by an organization.
Tools like Terraform, Checkov, and Open Policy Agent(OPA) help us to ensure our infrastructure is reliable and secure.
Terrateam, our trustworthy helper, streamlines the process, making CI/CD (Continuous Integration and Continuous Deployment) smoother especially for GitHub and Terraform users.
The workflow we followed in this blog
The DevOps journey involves writing code, scanning for security, and applying policies.
Terraform, the IAC tool, lets us build and manage infrastructure efficiently across various cloud providers.
Checkov acts as a safety inspector, identifying vulnerabilities in our configurations.
OPA, our rulebook, keeps our systems in check, ensuring they follow the right policies.
Here are some of our related blogs: