Sally Lehman is a production engineer at Auth0 , which provides a way for software developers to integrate authentication into their apps and APIs. She is a longtime expert in continuous delivery, having initiated CD pipelines at both GitHub and GoDaddy. At PuppetConf 2016, Sally will talk about writing templates to simplify management of configuration files in Puppet . Her talk covers the templating options available in Puppet, and advice on when and how to use them.
We asked Sally a few questions about her experience with continuous delivery pipelines, Puppet, and a few more questions just for fun. Here's what she had to say.
Puppet: How long have you been using Puppet?
Sally Lehman: I started modifying other people’s Puppet templates around six years ago. Coincidentally, that’s when I started developing an interest in Puppet; I started developing my own code a few years later.
Puppet: When and how did you first start with Puppet?
Sally: Somewhere around 2010-2011, Railsmachine had implemented configuration management using Moonshine for our infrastructure at Mad Mimi, and turned the management of some that code to me. I first started using Puppet by modifying the .erb templates to manage our mail server infrastructure. The ability to roll out long and repetitive configuration files using a comparatively simple template was what intrigued me about Puppet, along with the ability to ensure a certain state of our mail servers in spite of changes someone may have made directly.
Puppet: You built the continuous delivery pipeline at GoDaddy. Did you also do this at GitHub?
Sally: I didn’t build the CI/CD pipeline at GitHub, but I was using it a lot while I was there and occasionally delving into internals to connect up a new application. I saw how it streamlined the codepath from development to production. CI/CD removes the hoops and friction typically involved in that pipeline. Seeing how that worked and being inspired by it is what has caused me to spend time building CI/CD infrastructure at GoDaddy, and is also why I’m focusing on our pipeline at Auth0. I really love to make shipping code a more enjoyable and less stressful process for myself and for other developers.
Puppet: Everyone seems to want to do continuous delivery, but it also seems difficult to implement. What do you think blocks organizations/teams most often from getting there?
Sally: I don’t know of any team that hasn’t at least started some form of CI/CD and implemented a piece or two of it. There are so many good tools out there right now that those who haven’t are either under severe corporate restrictions, or are too burned out and exhausted from putting out fires to find cycles to implement it. For many teams, they may have implemented pieces but not a pipeline that allows for an absolute minimum of user initiated actions from committing a piece of code, until that code has been deployed. That includes cases when a rollback is necessary.
Starting from there, I think what blocks further implementation is simply not having experienced a streamlined process firsthand, and the fear that automation will lead to bad code getting out to production easier … i.e., wanting to minimize change-related incidents. When you have used a streamlined code pipeline, you will start to see how it cuts down on incident length and severity because one change per deploy makes it much easier to diagnose what has gone wrong and fix it. You also see how it makes it that much easier to fix and improve code that otherwise would be left behind to rot and cause problems later.
The fear that automation will lead to bad code getting out comes from good intentions but it puts the wrong kinds of roadblocks in the way that aren’t ideal for catching bad code. There is a lot of noise about "change-related incidents," for example. Manual deployment process and busywork catches problems, but it won’t catch as many as when you give a developer quick feedback on unit tests, allow them to focus on their code rather than the process of getting their code out, first in test and then incrementally in production. There’s already a ton of anxiety that goes into shipping code, and focusing on adding guardrails will increase quality, productivity, and morale.
Puppet: What should a team tackle or implement first if they want to do continuous delivery?
Sally: It’s not intuitive, but I think there is a strong argument for making a set of development, test and production environments that are repeatable and reproducible with a tool such as Puppet. This will give you a consistent and predictable environment with the dependencies that you need to work on deployment automation using a specialized deployment automation tool.
Puppet: What are the most common problems you see when teams are trying to improve their configuration management?
Sally: The codebase for configuration management itself becomes large, unorganized, and unwieldy quickly. Investing your time in learning best practices for how to organize your code at a conference like PuppetConf will save a lot of tears and refactoring down the road. There are a ton of structures that work fine and seem to be the correct way to do things at the beginning, but very few techniques that will scale, be manipulable, and recognizable by others.
Configuration management also requires upkeep. It’s important to have people tasked to maintenance; it will decay like any untended software.
Puppet: Is there a mindset or approach to configuration management you think is most helpful?
Sally: For common modules, look for what people have already built and build on top of them rather than starting from scratch. You will learn more quickly what works and what does not. Read up on how best to structure your Puppet code before you start putting modules together in a project.
Puppet: What's a fun fact about you that we might not know, but would like to?
Sally: Most of my friends and family know because I won’t stop talking about it, but I’ve been training in Olympic weightlifting for about a year and a half now. I’m really enjoying being physically strong and learning mental toughness from lifting progressively heavier things, particularly since joining the Nashville Weightlifting Club here. I’m looking forward to doing my first meets later this year.
Puppet: What's the most interesting thing you've read recently?
Sally: James Turnbull published "The Art of Monitoring" not long ago. It’s a very accessible and well done piece of work. I recommend it!
Puppet: Go ahead, tease us: What can we look forward to in your talk about templating in Puppet?
Sally: My goal this year is to present a simple, attractive, and complete tutorial for developing templates so the concepts for doing so are pleasant to learn for people just starting out with Puppet. If you’re just starting out, there are many new things to assimilate; lowering the bar to template development is one way to do that. Doing this lends energy towards figuring out the more difficult structural decisions ahead.
Aliza Earnshaw is the managing editor at Puppet.