Introduction to Puppet

Managing software state on hundreds of thousands of servers can be a real challenge. What web server should be installed? What web server configuration should looks like? Which users should have access to certain machine or group of machines? Now multiply those questions and many more by the number of servers you have and you end up in a total madness.

Fortunately the year is 2014+ so welcome to the world where tools for various automations are all over the open source world and different people use different sets depending on their needs, use case and environment requirements.

Meet Puppet. A software framework that allows you to easily make rapid changes across your infrastructure and enforce consistency with minimal efforts in four simple steps.

  • Define what the final state of your machines and groups of machines should look like with simple rules
  • Simulate enforcement of the rules to see how it goes
  • Enforce the rules so the machine can enter the desired software state
  • Receive reports about the changes, runtime etc.

So how to get started with it so it can make your operations life easier? Well we should know how its basics works first, so we can take proper decisions and avoid wrong ones bite us later.

  • Puppet runs/executes the so-called manifests files (recipes). Think of them like simple programs.
  • Manifest files describe the final stage of the system with clear, easy to use and understand syntax.
  • Group of manifest files are called modules
  • Puppet have set of functions you can use to make your life easier
  • Puppet is cool so it offers you functionality to write your own custom functions, classes and facts

Basic flow of Puppet operation

So how the very basic puppet execution flow of recipes looks like? Here it is.

Q: Hey Puppet! I want to eat meatballs. I have written recipe for meatballs over here. Could you please read my recipe and prepare me meatballs please?
A1: OK. Let me see if I can read it.
A2: I can read it and dish can be cooked as it has valid ingredients.
A3: Let me check if dish is already cooked and it looks exactly like you wanted it.
A4a: Dish is already cooked exactly to your requirements so just serving it.
A4b: Cooking dish according to your requirements and serving it.

Receipts that will be executed on the machines can be stored either locally or on a central server. Those are the main two modes in which puppet operates:

  • Running puppet receipts files stored on the same node (single node setup)
  • Pulling puppet receipts files stored on a central machine (master or multi-master setup)

Let us first cover the single node setup as it is easier for understanding for our example.

Puppet with a Kyup container

The only thing you need to get started is a running Kyup container instance. If you already have a running container and you want to use Puppet on it you are well prepared as Puppet comes pre-installed with each distribution image.

In case you want to start new container check the How to Create your First Container  steps. When your container is started go to “Containers” section in your user area and you will see all details about your instance. What we need will be the IP address so you can ssh into your container:

Installing the MySQL Puppet modules

Once you are in, suppose that you want to install mysql and manage your mysql instances with puppet. You can do that in various ways. For example you can create your own class or module that will handle the job or you can use third party modules for mysql management available at https://forge.puppetlabs.com. There are tons of freely available modules which help you manage different services and save your precious time for writing your own.

For the purpose of this tutorial we'll use Puppetlabs' mysql module available here:

https://forge.puppetlabs.com/puppetlabs/mysql

To get started to use this puppet module the only thing you need to do is to install it as simple as this command:

You should see the following output of the above command

The next thing you need is to define the rules for your MySQL instance. On which port it will be running, which will be the root MySQL password and many other options you need.

Creating the MySQL Puppet recipe

If you remember the intro in this article to achieve “peace” we need to define the final state of the system in easy to use language in the so called manifest that later can be tested and executed to achieve desired level of consistency.

So what is the desired level of consistency depends on your needs but for this article we will take the following as an example:

− We want MySQL installed on our machine
− We want to set custom MySQL root password
− We want MySQL to be constantly running
− We want MySQL max connections to be set to max_connections
− We want MySQL key_buffer_size set to 512M
− We want want to create daily backups of all our MySQL databases
− We want to store those backups secure in custom folder
− We want to archive those backups
− We want to retain 7 copies of our backups

We want too much? Luckily we do not have to write everything from scratch but we can rather use the definitive language that puppet provides along with the classes and functions offered by puppetlabs-mysql to achieve all our requirements in several lines.

By default puppet main recipe file is stored at:

It is always a good practice to follow the best practices so you can define your requirements in that default file by opening it in your favorite text editor.

root@c2710:~# vi /etc/puppet/manifests/init.pp

And here is an example manifest file that covers the forehand listed requirements:

Clear enough? More options that this module provides are listed on its home page here

https://forge.puppetlabs.com/puppetlabs/mysql

Syntax check and execution of the recipe

The next step is to check the syntax of your receipt:

Looks good. You can move further to see what actions will puppet make without actually make any changes to your system. That can be achieved with the so called noop (no operation) switch passed to the puppet apply command.

Finally if you are ok with the changes that are about to be performed on your system you can tell to puppet to apply all changes required to achieve the desired level of consistency defined in your init.pp file:

Prior applying this puppet manifest/recipe this container was almost naked with minimal set of packages installed on it. MySQL was not installed. After the recipe was applied that changed just as expected.

Verifying the result

So if you are like me and you don't trust the system you can always verify the final results:

Check 1: Is MySQL running:

Check 2: Proper cron entry for the backups is set

Check 3: Backup script executes without errors

Check 4: You will also want to verify if backup is actually being created in the expected directory (backup your systems, monitor your backups, test your backups)

Guarantee the consistency of the setup

But wait. What if mysql crashed or someone alter the cron file by hand and remove the backup script. Your changes will be lost and machine will enter in unknown or undesired state. The goal here is to achieve desired level of consistency right?

While running in stand alone mode you can set puppet to run via cronjob at interval suitable to you. Let's say 30 minutes.

This will read your puppet manifest file and all its includes, requirements etc. and enforce the policy defined in there.

If each machine keep its manifests files locally (puppet stand alone mode) whenever you change manifests code in your revision control system and tested in details (you do both RCS and tests right?) those should be distributed across all machines manually. If you are using central puppet server you can sync your changes to the central server only and all puppet clients will pull the changes from there.

Final considerations

  • Puppet itself is not a silver bullet that automatically fixes everything for you. It is a great tool and if used properly it can save you a lot of time and troubles. It can be a bit tricky to get started with
  • Tasks to convert old systems to be managed with puppet should be executed with great care and deep understanding of how those systems works as well as how exactly certain puppet modules, classes, components
  • Using puppet client-master with more than 10-20 nodes requires some efforts to scale. Possible solutions are using multiple-masters with DNS round robin, load balancing or scaling puppet with git.