Our Journey to Infrastructure as Code

Datetime:2016-08-23 03:27:47          Topic:          Share

At MuleSoft, we are committed to customer success by investing in product and engineering operational excellence. The faster we can innovate/produce new features, the better equipped our customers are at enabling IT and transforming their businesses. One of the ways we’ve been able to move faster is by migrating to Infrastructure as Code ( IaC ).

What Is IaC?

IaC is the process of managing and provisioning infrastructure and its configuration through templates using code. This basically means you can create a working environment (virtual machines, subnets, load balancers, databases, etc.) just by defining them in a configuration file. You can think of it as a blueprint for your resources and how they are interconnected. This approach exposes the workflow to the whole team and creates a lot more transparency for getting work done, which allows us to iterate and modify things much quicker.

Before IaC: Configuration Management

We are not going to cover the time before Configuration Management because it is too painful to remember, but let’s say that the DevOps team began its quest for a Configuration Management tool around 2012. The landscape back then was very different to what it is now, but the major players are still standing:

Puppet and Chef were the most popular choices, but we went with SaltStack because it was a very promising project with heavy development and it was written in Python (which was a huge advantage for us because we’d need to add custom functionality).

We were very happy with Salt for a while, but as our infrastructure began to grow, we needed something more: It was great for managing every aspect of an instance, but everything outside the virtual machine itself still needed to be handled manually.

The same goes if you wanted to replicate an environment (for example you were promoting things from dev to qa), you’d have to create those once again, one by one, by hand. And if something needed to be modified, it would be a manual process which then had to be backported to the other environments.

Enter Infrastructure as Code

Manually created infrastructure and server configuration can be extremely time-consuming, hard to troubleshoot, and error prone. IaC makes it more transparent, giving you a single source of truth for your infrastructure definition, and the ability to do changes in a consistent way.

Well-defined pieces of IaC are really easy to iterate, modify, and clone whenever necessary, and having all the resources that you manage in one place makes it easier to read and understand how things are tied up; this is especially true for new people joining the team.

DOCUMENTATION IS ALWAYS IMPORTANT!!

How We Use SaltStack and Terraform

SaltStack was (and still is) our main tool for managing hosts in AWS EC2. In our use case, if we need a new application service, we create an instance with some metadata that identifies the specific service it will provide, and SaltStack will take care of provision it accordingly.

But what would happen if on top of that instance you needed an ELB? or if you wanted to select its Security Groups? You would have to do that manually, which doesn’t seem like a problem… but we are talking about thousands of hosts.

Terraform is a tool developed by HashiCorp, that allows you to declare resources and their desired configuration, which is then applied to your live infrastructure. It has a variety of interesting features such as multi-vendor support and dependency management.

Our DevOps team presenting at the HashiCorp meetup at MuleSoft HQ in San Francisco in July 2016.

Keeping the Pace

With IaC, we’re saving significant time and resources that we can now apply to building creative new features and iterating on our product to continue to better serve our customers. We are constantly researching new ways of making our infrastructure faster, simpler and more reliable, speeding and improving the way the code gets to production. Configuration Management and Infrastructure as Code are just two of the steps we are implementing in that direction