IT operations have changed drastically in the past few years. Virtualization, cloud, business needs, and emerging technologies have accelerated the pace of how systems are provisioned, configured, and managed.

The manual setup of a growing number of operating systems is no longer a sustainable option. At the same time, in-house custom solutions to automate the installation and the management of systems cannot scale in terms of required maintenance and development efforts.

For these reasons, configuration management tools such as Puppet, Chef, CFEngine, Rudder, Salt, and Ansible (to mention only the most known open source ones) are becoming increasingly popular in many infrastructures.

They show infrastructure as code, that allows, in systems management, the use of some of the same best practices in software development for decades, such as maintainability, code reusability, testability, or version control.

Once we can express the status of our infrastructure with versioned code, there are powerful benefits:

With these kinds of tools, we can have a system provisioned from zero to production in a few minutes, or we can quickly propagate a configuration change over our whole infrastructure automatically.

Their power is huge and has to be handled with care; as we can automate massive and parallelized setups and configurations of systems; we might automate distributed destructions.

When dealing with Puppet's DSL, most of the time, we use resources as they are single units of configuration that express the properties of objects on the system. A resource declaration is always composed of the following parts:

Inside a catalog, for a given type, there can be only one title; there cannot be multiple resources of the same type with the same title, otherwise we get an error like this:

Error: Duplicate declaration: <Type>[<name>] is already declared in file <manifest_file> at line <line_number>; cannot redeclare on node <node_name>.

Resources can be native (written in Ruby), or defined by users in Puppet DSL.

These are examples of common native resources; what they do should be quite obvious:

  file { 'motd':
    path    => '/etc/motd',
    content => "Tomorrow is another day\n",
  }

  package { 'openssh':
    ensure => present,
  }

  service { 'httpd':
    ensure => running, # Service must be running
    enable => true,    # Service must be enabled at boot time
  }

For inline documentation about a resource, use the describe subcommand, for example:

puppet describe file

From the previous resource examples, we can deduce that the Puppet DSL allows us to concentrate on the types of objects (resources) to manage and doesn't bother us on how these resources may be applied on different operating systems.

This is one of Puppet's strong points, resources are abstracted from the underlying OS, we don't have to care or specify how, for example, to install a package on Red Hat Linux, Debian, Solaris, or Mac OS, we just have to provide a valid package name. This is possible thanks to Puppet's Resource Abstraction Layer (RAL), which is engineered around the concept of types and providers.

Types, as we have seen, map to an object on the system. There are more than 50 native types in Puppet (some of them applicable only to specific OSes), the most common and used are augeas, cron, exec, file, group, host, mount, package, service, and user. To have a look at their Ruby code, and learn how to make custom types, check these files:

ls -l $(facter rubysitedir)/puppet/type

For each type, there is at least one provider, which is the component that enables that type on a specific OS. For example, the package type is known for having a large number of providers that manage the installation of packages on many OSes, which are aix, appdmg, apple, aptitude, apt, aptrpm, blastwave, dpkg, fink, freebsd, gem, hpux, macports, msi, nim, openbsd, pacman, pip, pkgdmg, pkg, pkgutil, portage, ports, rpm, rug, sunfreeware, sun, up2date, urpmi, yum, and zypper.

We can find them here:

ls -l $(facter rubysitedir)/puppet/provider/package/

The Puppet executable offers a powerful subcommand to interrogate and operate with the RAL: puppet resource.

For a list of all the users present on the system, type:

puppet resource user

For a specific user, type:

puppet resource user root

Other examples that might give glimpses of the power of RAL to map systems' resources are:

puppet resource package
puppet resource mount
puppet resource host
puppet resource file /etc/hosts
puppet resource service

The output is in the Puppet DSL format; we can use it in our manifests to reproduce that resource wherever we want.

The Puppet resource subcommand can also be used to modify the properties of a resource directly from the command line, and, since it uses the Puppet RAL, we don't have to know how to do that on a specific OS, for example, to enable the httpd service:

puppet resource service httpd ensure=running enable=true

We can place the preceding resources in our first manifest file (/etc/puppetlabs/code/environments/production/manifests/site.pp) or in the form included there and they will be applied to all our Puppet managed nodes. This is okay for quick samples out of books, but in real life things are very different. We have hundreds of different resources to manage, and dozens, hundreds, or thousands of different systems to apply different logic and properties to.

To help organize our Puppet code, there are two different language elements: with node, we can confine resources to a given host and apply them only to it; with class, we can group different resources (or other classes), which generally have a common function or task.

Whatever is declared in a node, definition is included only in the catalog compiled for that node. The general syntax is:

   node $name [inherits $parent_node] {
  [ Puppet code, resources and classes applied to the node ]
}

$name is the certname of the client (by default its FQDN) or a regular expression; it's possible to inherit, in a node, whatever is defined in the parent node, and, inside the curly braces, we can place any kind of Puppet code: resources declarations, classes inclusions, and variable definitions. An example is given as follows:

node 'mysql.example.com' {

  package { 'mysql-server':
    ensure => present,
  }
  service { 'mysql':

    ensure => 'running',
  }
}

But generally, in nodes we just include classes, so a better real life example would be:

node 'mysql.example.com' {
  include common
  include mysql
}

The preceding include statements that do what we might expect; they include all the resources declared in the referred class.

Note that there are alternatives to the usage of the node statement; we can use an External Node Classifier (ENC) to define which variables and classes assign to nodes or we can have a nodeless setup, where resources applied to nodes are defined in a case statement based on the hostname or a similar fact that identifies a node.

A class can be defined (resources provided by the class are defined for later usage but are not yet included in the catalog) with this syntax:

class mysql {
  $mysql_service_name = $::osfamily ? {
    'RedHat' => 'mysqld',
    default  => 'mysql',
  }
  package { 'mysql-server':
    ensure => present,
  }
  service { 'mysql':
    name => $mysql_service_name,
    ensure => 'running',
  }
  […]
}

Once defined, a class can be declared (the resources provided by the class are actually included in the catalog) in multiple ways:

A parameterized class has a syntax like this:

class mysql (
  $root_password,
  $config_file_template = undef,
  ...
) {
  […]
}

Here, we can see the expected parameters defined between parentheses. Parameters with an assigned value have it as their default, as it is here. The case of undef for the $config_file_template parameter.

The declaration of a parameterized class has exactly the same syntax of a normal resource:

class { 'mysql':
  $root_password => 's3cr3t',
}

Puppet 3.0 introduced a feature called data binding; if we don't pass a value for a given parameter, as in the preceding example, before using the default value, if present, Puppet does an automatic lookup to a Hiera variable with the name $class::$parameter. In this example, it would be mysql::root_password.

This is an important feature that radically changes the approach of how to manage data in Puppet architectures. We will come back to this topic in the following chapters.

Besides classes, Puppet also has defines, which can be considered classes that can be used multiple times on the same host (with a different title). Defines are also called defined types, since they are types that can be defined using Puppet DSL, contrary to the native types written in Ruby.

They have a similar syntax to this:

define mysql::user (
  $password,                # Mandatory parameter, no defaults set
  $host      = 'localhost', # Parameter with a default value
  [...]
 ) {
  # Here all the resources
}

They are used in a similar way:

mysql::user { 'al':
  password => 'secret',
}

Note that defines (also called user defined types, defined resource type, or definitions) like the preceding one, even if written in Puppet DSL, have exactly the same usage pattern as native types, written in Ruby (such as package, service, file, and so on).

In types, besides the parameters explicitly exposed, there are two variables that are automatically set. They are $title (which is the defined title) and $name (which defaults to the value of $title) and can be set to an alternative value.

Since a define can be declared more than once inside a catalog (with different titles), it's important to avoid to declare resources with a static title inside a define. For example, this is wrong:

define mysql::user ( ...) {
  exec { 'create_mysql_user':
    [ … ]
  }
}

Because, when there are two different mysql::user declarations, it will generate an error like:

Duplicate definition: Exec[create_mysql_user] is already defined in file /etc/puppet/modules/mysql/manifests/user.pp at line 2; cannot redefine at /etc/puppet/modules/mysql/manifests/user.pp:2 on node test.example42.com 

A correct version could use the $title variable which is inherently different each time:

define mysql::user ( ...) {
  exec { "create_mysql_user_${title}":
    [ … ]
  }
}

We have seen that in Puppet classes are just containers of resources that have nothing to do with Object Oriented Programming classes so the meaning of class inheritance is somehow limited to a few specific cases.

When using class inheritance, the parent class (puppet in the sample below) is always evaluated first and all the variables and resource defaults sets are available in the scope of the child class (puppet::server).

Moreover, the child class can override the arguments of a resource defined in the parent class:

class puppet {
  file { '/etc/puppet/puppet.conf':
    content => template('puppet/client/puppet.conf'),
  }
}
class puppet::server inherits puppet {
  File['/etc/puppet/puppet.conf'] {
    content => template('puppet/server/puppet.conf'),
  }
}

Note the syntax used; when declaring a resource, we use a syntax such as file { '/etc/puppet/puppet.conf': [...] }; when referring to it the syntax is File['/etc/puppet/puppet.conf'].

Even when possible, class inheritance is usually discouraged in Puppet style guides except for some design patterns that we'll see later in the book.

It is possible to set default argument values for a resource type in order to reduce code duplication. The general syntax to define a resource default is:

Type {
  argument => default_value,
}

Some common examples are:

Exec {
  path => '/sbin:/bin:/usr/sbin:/usr/bin',
}
File {
  mode  => 0644,
  owner => 'root',
  group => 'root',
}

Resource defaults can be overridden when declaring a specific resource of the same type.

It is worth noting that the area of effect of resource defaults might bring unexpected results. The general suggestion is as follows:

We cannot expect a resource default defined in a class to be working in another class, unless it is a child class, with an inheritance relationship.

In Puppet, any resource is uniquely identified by its type and its name. We cannot have two resources of the same type with the same name in a node's catalog.

We have seen that we declare resources with a syntax such as:

type { 'name':
  arguments => values,
}

When we need to reference them (typically when we define dependencies between resources) in our code, the syntax is (note the square brackets and the capital letter):

Type['name']

Some examples are as follows:

file { 'motd': ... }
apache::virtualhost { 'example42.com': .... }
exec { 'download_myapp': .... }

These examples are referenced, respectively, with the following code:

File['motd']
Apache::Virtualhost['example42.com']
Exec['download_myapp']

When we install Puppet on a system, the facter package is installed as a dependency. Facter is executed on the client each time Puppet is run and it collects a large set of key/value pairs that reflect many system's properties. They are called facts and provide valuable information like the system's operatingsystem, operatingsystemrelease, osfamily, ipaddress, hostname, fqdn, macaddress to name just some of the most used ones.

All the facts gathered on the client are available as variables to the Puppet Master and can be used inside manifests to provide a catalog that fits the client.

We can see all the facts of our nodes, running locally:

facter -p

(The -p argument is the short version of --puppet, and also shows eventual custom facts, which are added to the native ones, via our modules).

In facter 1.x, only plain values were available; facter 2.x introduces structured values, so any fact can contain arrays or hashes. Facter is replaced by cFacter for 3.0, a more efficient implementation in C++ that makes an extensive use of structured data. In any case, it keeps legacy keys, making these two queries equivalent:

$ facter ipaddress
1.2.3.4
$ facter networking.interfaces.eth0.ip
1.2.3.4

External facts, supported since Puppet 3.4/Facter 2.0.1, provide a way to add facts from arbitrary commands or text files.

These external facts can be added in different ways:

Executable facts can be scripts in any language, or even binaries; the only requirement is that its output has to be formed by lines with the format key=value, like:

key1=value1
key2=value2

Structured data facts have to be plain text files with an extension indicating its format, .txt files for files containing key=value lines, .yaml for YAML files, and .json for JSON files.

Variable definition inside the Puppet DSL follows the general syntax: $variable = value.

Let's see some examples. Here the value is set as a string, a boolean, an array. or a hash:

$redis_package_name = 'redis'
$install_java = true
$dns_servers = [ '8.8.8.8' , '8.8.4.4' ]
$config_hash = { user => 'joe', group => 'admin' }

From Puppet 3.5, using the future parser or starting on version 4.0 by default Here docs are also supported, what is a convenient way of define multiline strings:

$gitconfig = $("GITCONFIG")
  [user]
    name = ${git_name}
    email = ${email}
  | GITCONFIG
file { "${homedir}/.gitconfig":
  content => $gitconfig,
}

They have multiple options. In the previous example, we set GITCONFIG as the delimiter, the quotes indicate that variables in the text have to be interpolated, and the pipe character marks the indentation level.

Here, the value is the result of a function call (which may have strings and other data types or other variables as arguments):

$config_file_content = template('motd/motd.erb')

$dns_servers = hiera(name_servers)
$dns_servers_count = inline_template('<%= @dns_servers.length %>')

Here, the value is determined according to the value of another variable (here the $::osfamily fact is used), using the selector construct:

$mysql_service_name = $::osfamily ? {
  'RedHat' => 'mysqld',
  default  => 'mysql', 
}

A special value for a variable is undef (a null value similar to Ruby's nil), which basically removes any value to the variable (can be useful in resources when we want to disable, and make Puppet ignore, an existing attribute):

$config_file_source = undef
file { '/etc/motd':
  source  => $config_file_source,
  content => $config_file_content,
}

Note that we can't change the value assigned to a variable inside the same class (more precisely inside the same scope; we will review them later). Consider a code like the following:

$counter = '1'
$counter = $counter + 1

The preceding code will produce the following error:

Cannot reassign variable counter

The new parser used as default in Puppet 4 has better support for data types, what includes optional type-checking. Each value in Puppet has a data type, for example, strings are of the type String, booleans are of the type Boolean, and types themselves have their own type Type. Every time we declare a parameter, we can enforce its type:

class ntp (
    Boolean $enable = true,
    Array[String] $servers = [],
  ) { … }

We can also check types with expressions like this one:

$is_boolean =~ String

Or make selections by the type:

$enable_real = $enable ? {
  Boolean => $enable,
  String  => str2bool($enable),
  Numeric => num2bool($enable),
  default => fail('Illegal value for $enable parameter')
}

Types can have parameters, String[8] would be a string of at least 8 characters-length, Array[String] would be an array of strings and Variant[Boolean, Enum['true', 'false']] would be a composed value that would match with any Boolean or with members of the enumeration formed by the strings true and false.

When an ENC is used for the classification of nodes, it returns the classes to include in the requested node and variables. All the variables provided by an ENC are at top scope (we can reference them with $::variablename all over our manifests).

A very popular and useful place to place user data (yes, variables) is also Hiera; we will review it extensively in Chapter 2, Managing Puppet Data with Hiera; let's just point out a few basic usage patterns here. We can use it to manage any kind of variable, whose value can change according to custom logic in a hierarchical way. Inside manifests, we can lookup a Hiera variable using the hiera() function. Some examples are as follows:

$dns = hiera(dnsservers)
class { 'resolver':
  dns_server => $dns,
}

The preceding example can also be written as:

class { 'resolver':
  dns_server => hiera(dnsservers),
}

In our Hiera YAML files, we would have something like this:

dnsservers:
  - 8.8.8.8
  - 8.8.4.4

If our Puppet Master uses Puppet version 3 or greater, then we can benefit from the Hiera automatic lookup for class parameters, that is, the ability to define values for any parameter exposed by the class in Hiera. The preceding example would become something like this:

include resolver

Then, in the Hiera YAML files:

resolver::dns_server:
  - 8.8.8.8
  - 8.8.4.4

A bunch of other variables are available and can be used in manifests or templates:

One of the parts where Puppet development can be misleading and not so intuitive is how variables are evaluated according to the place in the code where they are used.

Variables have to be declared before they can be used and this is parse order dependent, so, for this reason, Puppet language can't be considered completely declarative.

In Puppet, there are different scopes; partially isolated areas of code where variables and resource defaults values can be confined and accessed.

There are four types of scope, from general to local:

We always write code within a scope and we can directly access variables (that is just specifying their name without using the fully qualified name) defined only in the same scope or in a parent or containing one. The following are the ways, we can access top scope variables, node scope variables, and class variables:

Variables' value or resources default arguments defined at a more general level can be overridden at a local level (Puppet uses always the most local value).

It's possible to refer to variables outside a scope by specifying their fully qualified name, which contains the name of the class where the variables are defined. For example, $::apache::config_dir is a variable, called config_dir, defined in the apache class.

One important change introduced in Puppet 3.x is the forcing of static scoping for variables; this involves that a parent scope for a class can be only its parent class.

Earlier Puppet versions had dynamic scoping, where parent scopes were assigned both by inheritance (as in static scoping) and by simple declaration; that is, any class has the first scope where it has been declared as parent. This means that, since we can include classes multiple times, the order used by Puppet to parse our manifests may change the parent scope and therefore how a variable is evaluated.

This can obviously lead to any kind of unexpected problems, if we are not particularly careful on how classes are declared, with variables evaluated in different (parse order dependent) ways. The solution is Puppet 3's static scoping and the need to reference to out of scope variables with their fully qualified name.

Puppet 2.6 introduced the concept of run stages to help users manage the order of dependencies when applying groups of resources.

Puppet provides a default main stage; we can add any number or further stages, and their ordering, with the stage resource type and the normal syntax we have seen:

stage { 'pre':
  before => Stage['main'],
}

The normal syntax is equivalent to:

stage { 'pre': }
Stage['pre'] -> Stage['main']

We can assign any class to a defined stage with the stage meta parameter:

class { 'yum':
  stage => 'pre',
}

In this way, all the resources provided by the yum class are applied before all the other resources (in the default main stage).

The idea of stages at the beginning seemed a good solution to better handle large sets of dependencies in Puppet. In reality, some drawbacks and the augmented risk of having dependency cycles make them less useful than expected. A thumb rule is to use them for simple classes (that don't include other classes) and where it is really necessary (for example, to set up package management configurations at the beginning of a Puppet run or deploy our application after all the other resources have been managed).

The in operator checks whether a string is present in another string, an array, or in the keys of a hash. It is case sensitive:

if '64' in $::architecture
if $monitor_tool in [ 'nagios' , 'icinga' , 'sensu' ]

It's possible to combine multiple comparisons with and and or:

if ($::osfamily == 'RedHat') and ($::operatingsystemrelease == '5') { [ ... ] }
if (operatingsystem == 'Ubuntu') or ($::operatingsystem == 'Mint') { [ ...] }

Virtual resources define a desired state for a resource without adding it to the catalog. Like normal resources, they are applied only on the node where they are declared, but, as virtual resources, we can apply only a subset of the ones we have declared; they have also a similar usage syntax: we declare them with a single @ prefix (instead of the @@ used for exported resources), and we collect them with <| |> (instead of <<| |>>).

A useful and rather typical example involves user's management.

We can declare all our users in a single class, included by all our nodes:

class my_users {
  @user { 'al': […] tag => 'admins' }
  @user { 'matt': […] tag => 'developers' }
  @user { 'joe': [… tag => 'admins' }
[ … ]
}

These users are actually not created on the system; we can decide which ones we actually want on a specific node with a syntax like this:

User <| tag == admins |>It is equivalent to:
realize(User['al'] , User['joe'])

Note that the realize function needs to address resources with their name.

Modules have a standard structure, for example, for a MySQL module the code reads thus:

mysql/            # Main module directory

mysql/manifests/  # Manifests directory. Puppet code here.
mysql/lib/        # Plugins directory. Ruby code here
mysql/templates/  # ERB Templates directory
mysql/files/      # Static files directory
mysql/spec/       # Puppet-rspec test directory
mysql/tests/      # Tests / Usage examples directory
mysql/facts.d/    # Directory for external facts

mysql/Modulefile  # Module's metadata descriptor

This layout enables useful conventions, which are widely used in Puppet world; we must know them to understand where to look for files and classes:

For example, we can use modules and write the following code:

include mysql

Puppet will then automatically look for a class called mysql defined in the file $modulepath/mysql/manifests/init.pp:

The init.pp script is a special case that applies for classes that have the same name of the module. For sub classes there's a similar convention that takes in consideration the subclass name:

include mysql::server

It then auto loads the $modulepath/mysql/manifests/server.pp file.

A similar scheme is also followed for defines or classes at lower levels:

mysql::conf { ...}

This define is searched in $modulepath/mysql/manifests/conf.pp:

include mysql::server::ha

It then looks for $modulepath/mysql/manifests/server/ha.pp.

It's generally recommended to follow these naming conventions that allow auto loading of classes and defines without the need to explicitly import the manifests that contain them.

Module's naming conventions apply also to the files that Puppet provides to clients.

We have seen that the file resource accepts two different and alternative arguments to manage the content of a file: source and content. Both of them have a naming convention when used inside modules.

Templates, typically parsed via the template or the epp functions with syntax like the one given here, are found in a place like $modulepath/mysql/templates/my.cnf.erb:

content => template('mysql/my.cnf.erb'),

This also applies to sub directories, so for example:

content => template('apache/vhost/vhost.conf.erb'),

It uses a template located in $modulepath/apache/templates/vhost/vhost.conf.erb.

A similar approach is followed with static files provided via the source argument:

source => 'puppet:///modules/mysql/my.cnf'

It serves a file placed in $modulepath/mysql/files/my.cnf:

source => 'puppet:///modules/site/openssh/sshd_config'

This serves a file placed in $modulepath/site/openssh/sshd_config.

Notice the differences in templates and source paths. Templates are resolved in the server, and they are always placed inside the modules. Sources are retrieved by the client and modules in the URL could be a different mount point if it's configured on the server.

Finally, the whole content of the lib subdirectory in a module has a standard scheme. Note that here, we can place Ruby code that extends Puppet's functionality and is automatically redistributed from the Master to all clients (if the pluginsync configuration parameter is set to true, this is default for Puppet 3 and widely recommended in any setup):

mysql/lib/augeas/lenses/                # Custom Augeas lenses.
mysql/lib/facter/                       # Custom facts.
mysql/lib/puppet/type/                  # Custom types.
mysql/lib/puppet/provider/<type_name>/  # Custom providers.
mysql/lib/puppet/parser/functions/      # Custom functions.

Files provisioned by Puppet can be templates written in Ruby's ERB templating language or in the Embedded Puppet Template Syntax (EPP).

An ERB template can contain whatever text we need and have inside <% %> tags, interpolation of variables or Ruby code. We can access, in a template, all the Puppet variables (facts or user assigned) with the <%= tag:

# File managed by Puppet on <%= @fqdn %>
search <%= @domain %>

The @ prefix for variable names is highly recommended in all Puppet versions, and mandatory starting from 4.0.

To use out of scope variables, we can use the scope.lookupvar method:

path <%= scope.lookupvar('apache::vhost_dir') %>

This uses the variable's fully qualified name. If the variable is at top scope then run the following command:

path <%= scope.lookupvar('::fqdn') %>

Since Puppet 3, we can use this alternative syntax:

path <%= scope['apache::vhost_dir'] %>

In ERB templates, we can also use more elaborate Ruby code inside a <% opening tag, for example, to reiterate over an array:

<% @dns_servers.each do |ns| %>
nameserver <%= ns %>
<% end %>

The <% tag is used to place a line of text if some conditions are met:

<% if scope.lookupvar('puppet::db') == "puppetdb" -%>
  storeconfigs_backend = puppetdb
<% end -%>

Noticed the -%> ending tag here? When the dash is present, no line is introduced on the generated file, as it would if we had written <% end %>.

EPP templates are quite similar, they are also plain text files and they use the same tags for the embedded code, the main differences are that EPPs use Puppet code instead of Ruby, that they can receive type-checked parameters, and that they can directly access other variables where in ERBs we'd have to use lookup functions.

The parameters definition is optional but if it's included it has to be in the beginning of the file:

<%- | Array[String] $dns_servers,
      String $search_domain | -%>

To use templates in Puppet code, we have to use the template function for ERBs or the epp function for EPPs; epp can receive a hash with the values of the arguments as a second argument:

file { '/etc/resolv.conf':
  content => epp('resolvconf/resolv.conf.epp', { 
    'dns_ervers': ['8.8.8.8', '8.8.4.4'],
    'search_domain': 'example.com',
  })
}

The file is a YAML hash, where the top-level keys are Ruby symbols, with a colon (:) prefix, which may be either global or backend specific settings.

The default content for the configuration file is as follows:

---
:backends: yaml
:yaml:
  :datadir: /etc/puppetlabs/code/environments/%{environment}/hieradata
:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common
:logger: console

Using these settings, Hiera key-values are read from a YAML file with the /etc/puppetlabs/code/environments/%{environment}/common.yaml path.

The default datadir in versions before 4.0 was /var/lib/hiera.

---
:backends:
  - yaml
  - gpg

:hierarchy:
  - "nodes/%{::fqdn}"
  - "roles/%{::role}"
  - "zones/%{::zone}"
  - "common"

:yaml:
  :datadir: /etc/puppetlabs/code/environments/%{environment}/hieradata
:gpg:
  :datadir: /etc/puppetlabs/code/environments/%{environment}/hieradata
  :key_dir: /etc/puppetlabs/gpgkeys

---
:backends:
  - yaml
  - file
  - gpg

:hierarchy:
  - "%{::env}/fqdn/%{::fqdn}"
  - "%{::env}/role/%{::role}"
  - "%{::env}/zone/%{::zone}"
  - "%{::env}/common

:yaml:
  :datadir: /etc/puppetlabs/code/data

:file:
  :datadir: /etc/puppetlabs/code/data

:gpg:
  :key_dir: /etc/puppetlabs/code/gpgkeys
  :datadir: /etc/puppetlabs/code/gpgdata

With a hash, it is possible to express complex and structured data that has to be managed inside the Puppet code.

Remember that Hiera always returns the value of the first defined level keys, for example, we have a hash with nested hashes, as shown in the following code:

  network::interfaces_hash:
    eth0:
      ipaddress: 10.0.0.193
      netmask: 255.255.255.0
      network: 10.0.0.0
      broadcast: 10.0.0.255
      gateway: 10.0.0.1
      post_up:
        - '/sbin/ifconfig eth3 up'
        - '/sbin/ifconfig eth4 up'
    eth2:
      enable_dhcp: true
    eth3:
      auto: false
      method: manual
    eth4:
      auto: false
      method: manual

We can create a variable inside our Puppet code that loads it:

$int_hash = hiera('network::interfaces_hash')

Then, refer to single values inside its data structure with the following code:

$ip_eth0 = $int_hash['eth0']['ipaddress']

If we need to access this value from a template, we can directly write it in our erb file:

ipaddress <%= @int_hash['eth0']['ipaddress'] %>

create_resources('network::interface', $interfaces_hash)

With Puppet 3 Hiera is shipped directly within the core code, but the integration goes far beyond: an automatic Hiera lookup is done for each class' parameter using the $class::$argument key; this functionality is called data bindings or automatic parameter lookup.

An example is the following class definition:

class openssh (
  $template = 'openssh/sshd.config.erb',
) { . . . }

The value of $template will be evaluated according to the following logic:

To emulate a similar behavior on Puppet versions earlier than version 3, we would write something like the following:

class openssh (
 $template = hiera("openssh::template",'openssh/sshd.config.erb'),
) { . . . }

This strong integration has definitively boosted the adoption of Hiera and is changing the way Puppet code is organized and classes are declared.

Before Puppet 2.6, we could declare classes by just including them, optionally more than once in our catalog, using the following code:

include openssh

And, we could manage the behavior of the class just by setting variables and having them dynamically evaluated inside the class with something like the following code:

class openssh {
  file { 'sshd_config':
    content => template($openssh_template),
  }
}

This approach suffered the risks of having inconsistent values due to dynamic scoping of variables and parse ordering. Also, when using variables inside the module's code, it wasn't easy to understand which variables could affect the class's behavior, and there was not a public API, which is easily accessible.

The introduction of parameterized classes in Puppet 2.6 allowed classes to expose their arguments in a clear way using the following code:

class openssh (
  $template = 'openssh/sshd.config.erb',
) { . . . }

In order to pass them explicitly and consistently in a class declaration, use the following code:

class { 'openssh':
  template => 'site/openssh/sshd_config.erb',
}

But the fact that we can declare a specific class, using parameters, only once in a node's catalog presented new challenges on how our custom code had to be organized. We could not include classes wherever needed but we had to reorganize our manifests in order to avoid duplicate declarations.

From Puppet 3 onwards, it is possible to have the best of both worlds, we can use the original way to include classes:

include openssh

But we can also be sure that the template parameter is always and consistently evaluated according to the value of the Hiera key openssh::template.

Hiera-file (https://github.com/adrienthebo/hiera-file) has been conceived by Adrien Thebo to manage a kind of data that previously couldn't be stored in a sane way in Hiera—that is, plain files.

To install it, just clone the previous Git repository in our modulepath or use its gem as follows:

gem install hiera-file

We configure it by specifying a datadir path where our data files are placed:

---
:backends:
  - file
:hierarchy:
  - "fqdn/%{fqdn}"
  - "role/%{role}"
  - "common"
:file:
  :datadir: /etc/puppetlabs/code/data

Here, the key used for Hiera lookups is actually the name of a file present in .d subdirectories inside our datadir according to our hierarchy.

For example, consider the following Puppet code:

file { '/etc/ssh/sshd_config':
  ensure  => present,
  content => hiera('sshd_config'),
}

Given the previous hierarchy (first file found is returned), the code will create a /etc/ssh/sshd_config file with the content taken from files searched in these places:

/etc/puppetlabs/code/data/fqdn/<fqdn>.d/sshd_config
/etc/puppetlabs/code/data/role/<role>.d/sshd_config
/etc/puppetlabs/code/data/common.d/sshd_config

In the preceding examples, <fqdn> and <role> have to be substituted with the actual FQDN and role of our nodes.

If we want to provide an ERB template using hiera-file we can use this syntax:

file { '/etc/ssh/sshd_config':
  ensure  => present,
  content => inline_template(hiera('sshd_config.erb')),
}

This will look for an erb template and parse it from:

/etc/puppetlabs/code/data/fqdn/<fqdn>.d/sshd_config.erb
/etc/puppetlabs/code/data/role/<role>.d/sshd_config.erb
/etc/puppetlabs/code/data/common.d/sshd_config.erb

Hiera-file is quite simple to use and very powerful because it allows us to move to Hiera what is generally placed in (site) modules: plain files with which we manage and configure our applications.

Being Puppet code and data generally versioned with a SCM and distributed accordingly, it has always been an issue to decide how and where to store reserved data such as passwords, private keys, and credentials. They were generally values assigned to variables either in clear text or as MD5/SHA hashes, but the possibility to expose them has always been a concern for Puppeteers, and various more or less imaginative solutions have been tried (sometimes, the solution has been to just ignore the problem and have no solution).

gem install hiera-gpg

---
:backends:
  - gpg
:hierarchy:
  - %{env}
  - common
:gpg:
  :datadir: /etc/puppetlabs/code/gpgdata
#  :key_dir: /etc/puppet/gpg

gpg –-gen-key

gpg --list-key
/root/.gnupg/pubring.gpg
------------------------
pub   2048R/C96EECCF 2013-12-08
uid                  Puppet Master (Puppet) <al@lab42.it>
sub   2048R/0AFB6B1F 2013-12-08

---
mysql::root_password: 'V3rys3cr3T!'

gpg --encrypt -o common.gpg -r C96EECCF common.yaml

hiera mysql::root_password -d

DEBUG: <datetime>: Hiera YAML backend starting
DEBUG: <datetime>: Looking up mysql::root_password in YAML backend
DEBUG: <datetime>: Looking for data source common
DEBUG: <datetime>: [gpg_backend]: Loaded gpg_backend
DEBUG: <datetime>: [gpg_backend]: Lookup called, key mysql::root_password resolution type is priority
DEBUG: <datetime>: [gpg_backend]: GNUPGHOME is /root/.gnupg
DEBUG: <datetime>: [gpg_backend]: loaded cipher: /etc/puppet/gpgdata/common.gpg
DEBUG: <datetime>: [gpg_backend]: result is a String ctx #<GPGME::Ctx:0x7fb6aaa2f810> txt ---
mysql::root_password: 'V3rys3cr3T!'
DEBUG: <datetime>: [gpg_backend]: GPG decrypt returned valid data
DEBUG: <datetime>: [gpg_backend]: Data contains valid YAML
DEBUG: <datetime>: [gpg_backend]: Key mysql::root_password found in YAML document, Passing answer to hiera
DEBUG: <datetime>: [gpg_backend]: Assigning answer variable

gpg -o common.yaml -d common.gpg

Hiera-eyaml is currently the most recommended plugin to manage secure data with hiera. It was created as an evolution of hiera-gpg, and has some improvements over the older plugin:

Let's see how hiera-eyaml works, as it is more used and maintained than hiera-gpg.

We install gem:

gem install hiera-eyaml

We edit the hiera.yaml to configure it:

---
:backends:
  - eyaml
:hierarchy:
  - "nodes/%{fqdn}"
  - "env/%{env}"
  - common
:eyaml:
  :datadir: /etc/puppet/code/hieradata
  :pkcs7_private_key: /etc/puppet/keys/private_key.pkcs7.pem
  :pkcs7_public_key:  /etc/puppet/keys/public_key.pkcs7.pem

Now, at our disposal is the powerful eyaml command, which makes the whole experience pretty easy and straightforward. We can use it to create our keys, encrypt and decrypt files or single strings, and directly edit on-the-fly files with encrypted values:

Note that the value is in this format: ENC[PKCS7,Encrypted_Value].

Luckily, in a similar fashion, we have to generate the keys only once, since great things happen when we have to change our encrypted values in our eyaml files. We can directly edit them with the following command:

eyaml edit /etc/puppet/hieradata/common.eyaml

Our editor will open the file and we will see the decrypted values, as it will decrypt them on the fly, so that we can edit our secrets in clear text and save the file again (of course, we can do this only on a machine where we have access to the private key). The decrypted values will appear in the editor between brackets and prefixed by DEC and the encryption backend used, PKCS7 by default:

mysql::root_password: DEC(1)::PKCS7[V3ryS3cr3T!]!

This makes the management and maintenance of our secrets particularly easy. To view the decrypted content of a eyaml file, we can use:

eyaml decrypt -f /etc/puppet/hieradata/common.eyaml

Since hiera-eyaml manages both clear text and encrypted values, we can use it as our only backend if we want to work only on YAML files.

Hiera-http (https://github.com/crayfishx/hiera-http) and hiera-mysql (https://github.com/crayfishx/hiera-mysql) are other powerful Hiera backends written by Craig Dunn; they perfectly interpret Hiera's modular and extendable design and allow us to retrieve our data either via a REST interface or via MySQL queries on a database.

A quick view of how they could be configured can give an idea of how they can fit different cases. To configure hiera-http in our hiera.yaml, place something like this:

:backends:
  - http
:http:
  :host: 127.0.0.1
  :port: 5984
  :output: json
  :failure: graceful
  :paths:
    - /configuration/%{fqdn}
    - /configuration/%{env}
    - /configuration/common

To configure hiera-mysql, run the following code:

---
:backends:
  - mysql
:mysql:
  :host: localhost
  :user: root
  :pass: examplepassword
  :database: config
  :query: SELECT val FROM configdata WHERE var='%{key}' AND environment='%{env}'

We will not examine them deeper and leave the implementation and usage details to the official documentation. Just consider how easy and intuitive the syntax to configure them and what powerful possibilities they open. They let users manage Puppet data from, for example, a web interface, without touching Puppet code or even logging in to a server and working with a SCM such as Git.

There are multiple ways to install PuppetDB. It can be installed from source, from packages, or using the Puppet Labs puppetdb module. In this book we are going to use the latter approach, so we also practice the use of community plugins. This module will be in charge of deploying a PostgreSQL server and PuppetDB.

First of all, we have to install the puppet module and its dependencies from the Puppet forge; they can be directly downloaded from their source or using Puppet:

puppet module install puppetlabs-puppetdb

Once installed in our Puppet server, it can be used to define the catalog of our PuppetDB infrastructure, it can be defined in three different ways:

We can also specify other parameters as the a version to install, what may be needed depending on the version of Puppet our servers are running:

class { 'puppetdb':
  puppetdb_version => '2.3.7-1puppetlabs1',
}

In any of these cases, once Puppet is executed, we'll have PuppetDB running in our server, by default on port 8080. We can check it by querying the version through its API:

$ curl http://localhost:8080/pdb/meta/v1/version
{
  "version" : "3.1.0"
}

If something goes wrong, we can check the logs in /var/log/puppetlabs/puppetdb/.

The configuration of PuppetDB involves operations on different files, such as:

[global]

vardir = /var/lib/puppetdb # Must be writable by Puppetdb user
logging-config = /etc/puppetdb/log4j.properties
resource-query-limit = 20000
event-query-limit = 20000

[command-processing]

threads = 4
store-usage = 102400
temp-usage = 51200
dlo-compression-threshold = 1d # Same of 24h, 1440m, 86400s

[database]

classname = org.hsqldb.jdbcDriver
subprotocol = hsqldb
subname = file:/var/lib/puppetdb/db/db;hsqldb.tx=mvcc;sql.syntax_pgs=true

classname = org.postgresql.Driver
subprotocol = postgresql
subname = //<HOST>:<PORT>/<DATABASE>
username = <USERNAME>
password = <PASSWORD>

sudo -u postgres sh
createuser -DRSP puppetdb
createdb -E UTF8 -O puppetdb puppetdb

# TYPE  DATABASE   USER   CIDR-ADDRESS  METHOD
host    all        all    10.42.42.30/32  md5

subname = //10.42.42.35:5432/puppetdb
username = puppetdb
password = <the password entered with the createuser command>

gc-interval = 60
node-ttl = 15d # Nodes not reporting for 15 days are deactivated
node-purge-ttl = 10d # Nodes purged 10 days after deactivation
report-ttl = 14d # Event reports are kept for 14 days

log-slow-statements = 10

conn-max-age = 60 # Maximum idle time
conn-keep-alive = 45 # Client-side keep-alive interval
conn-lifetime = 60 # The maximum lifetime of a connection

[jetty]

host = 0.0.0.0 # Listen on any interface (Read Note below)
port = 8080    # If not set, unencrypted HTTP access is disabled

ssl-host = 0.0.0.0
ssl-port = 8081
ssl-key = /var/lib/puppet/ssl/private_keys/puppetdb.site.com.pem
ssl-cert = /var/lib/puppet/ssl/public_keys/puppetdb.site.com.pem
ssl-ca-cert = /var/lib/puppet/ssl/certs/ca.pem

keystore = /etc/puppetdb/ssl/keystore.jks
truststore = /etc/puppetdb/ssl/truststore.jks
key-password = s3cr3t # Passphrase to unlock the keystore file
trust-password = s3cr3t # Passphrase to unlock the truststore file

If we used the Puppet Labs puppet module to install PuppetDB we can also use it to configure our puppet master so that it sends the information about the executions to PuppetDB. This configuration is done by the puppetdb::master::config class. As we have seen in other cases, we can execute this class by including it in the Puppet catalog for our server:

include puppetdb::master::config

Or by running masterless Puppet:

puppet apply -e "include puppetdb::master::config"

This will install the puppetdb-termini package, as well as set up the required settings:

PuppetDB integrates a performance dashboard out of the box; we can use it to check how the software is working in real time. It can be accessed via HTTP at the URL http://puppetdb.server:8080/pdb/dashboard/index.html if you set host = 0.0.0.0 on the PuppetDB configuration. Remember that you should limit HTTP access to unauthorized clients only, either by firewalling the host's port or setting host = localhost and having a local reverse proxy where you can manage access lists or authentication:

The PuppetDB performance dashboard

From the previous picture, the most interesting metrics are as follows:

The amount of information stored on PuppetDB is huge and precious, and while it can be queried from the command line, a visual interface can help users explore Puppet's data.

Daniele Sluijters, a community member, started a project that has quickly become the visual interface of reference for PuppetDB: Puppetboard is a web frontend written in Python that allows easy browsing of nodes' facts, reports, events, and PuppetDB metrics.

It also allows us to directly query PuppetDB, so all the example queries from the command-line we'll see in this chapter can be issued directly from the web interface.

This is a relatively young and quite dynamic project that follows PuppetDB's APIs evolution; check its GitHub project for the latest installation and usage instructions: https://github.com/nedap/puppetboard.

The URL for queries is structured like this:

http[s]://<server>:<port>/pdb/query/<version>/<endpoint>?query=<query>

Available endpoints for queries are: nodes, environments, factsets, facts, fact-names, fact-paths, fact-contents, catalogs, edges, resources, reports, events, event-counts, aggregate-event-counts, metrics, server-time, and version.

Query strings are URL-encoded JSON arrays in prefix notation, which makes them look a bit unusual. The general format is as follows:

[ "<operator>" , "<field>" , "<value>" ]

The comparison operators are: =, >=, >, < , <= and ~ (regexp matching). Some examples are as follows:

["=", "type", "Service"]
[">=", "timestamp", "2013-12-18T14:00:00"]
["~", "certname", "www\\d+\\.example\\.com"]

The expressions can be combined with and, not, and or. An example (here split over multiple lines for clarity) is as follows:

[ "and",
  ["=", "type", "File"],
  ["=", "title", "/etc/hosts" ]
]

It's possible to build complex subqueries using the in operator, the extract statement, and subqueries such as select-resources or select-facts. An example usable on the /facts endpoint to return the IPs of all the nodes that have an Apache service is as follows:

["and",
  ["=", "name", "ipaddress"],
  ["in", "certname",
    ["extract", "certname",
      ["select-resources",
        ["and",
          ["=", "type", "Service"],
          ["=", "title", "apache"] ] ] ] ] ]

Since version 3 of API, it has been possible to paginate and sort the results of queries. Each endpoint may support one or more query parameters: order-by, limit, include-total, offset, and so on.

It's quite easy to query PuppetDB directly with curl; following is the simplest example, with curl executed on HTTP on the same PuppetDB host:

curl http://localhost:8080/pdb/query/v4/nodes/web01.example.com

Note the URL to a specific endpoint (facts), the API version (v4), and the specific client certname.

When we have to use queries, we must URL encode characters such as [ and ], and for this we can use curl's –data-urlencode option. When we use it, we have to specify to use the -X GET option (otherwise a POST would be done):

curl -X GET 'http://localhost:8080/pdb/query/v4/events' –data-urlencode 'query=["=", "certname" , "puppet.example.com"]'

The response, in JSON array format (note the starting and ending square brackets [ ]), contains one or more entries like this:

[ {

  "new_value" : "{md5}be99db88f4c07058843ea356eb3469bf",

  "report" : "2331579061f83db1a35e7579a83a671f011e07fa",

  "run_start_time" : "2016-03-19T21:17:26.790Z",

  "property" : "content",

  "file" : "/etc/puppetlabs/code/environments/production/modules/puppetdb/manifests/master/routes.pp",

  "old_value" : "{md5}d13e1f5c099082afbe8a5ed9d4695beb",

  "containing_class" : "Puppetdb::Master::Routes",

  "line" : 38,

  "resource_type" : "File",

  "status" : "success",

  "configuration_version" : "1458422249",

  "resource_title" : "/etc/puppetlabs/puppet/routes.yaml",

  "environment" : "production",

  "timestamp" : "2016-03-19T21:17:39.138Z",

  "run_end_time" : "2016-03-19T21:17:36.705Z",

  "report_receive_time" : "2016-03-19T21:18:38.350Z",

  "containment_path" : [ "Stage[main]", "Puppetdb::Master::Routes", "File[/etc/puppetlabs/puppet/routes.yaml]" ],

  "certname" : "puppet.example.com",

  "message" : "content changed '{md5}d13e1f5c099082afbe8a5ed9d4695beb' to '{md5}be99db88f4c07058843ea356eb3469bf'"

} ]

When we make requests over HTTPS we have to reference the certificates' files to use:

$ curl 'https://puppetdb:8081/pdb/query/v4/facts/web01.example.com' \
  --cacert /var/lib/puppet/ssl/certs/ca.pem \
  --cert  /var/lib/puppet/ssl/certs/<node>.pem \
  --key /var/lib/puppet/ssl/private_keys/<node>.pem

Explicit commands are used (via HTTP URL-encoded POST to the /commands URL) to populate and modify data.

The available commands on PuppetDB are:

On /var/log/puppetdb/puppetdb.log, all the executed commands are shown.

When the Puppet Master receives a client's facts, it immediately submits them to PuppetDB:

2016-03-19 21:23:14,780 INFO  [p.p.command] [51ab082d-a04f-4b11-a88e-ab38adc248d7] [replace facts] web01.example.com

Then the catalog is compiled, sent to the client, and stored on PuppetDB:

2016-03-19 21:23:26,050 INFO  [p.p.command] [076e6a53-5d92-44a3-a550-05d9a99114fe] [replace catalog] web01.example.com

Finally, when the report of the Puppet run is received from the client, the Puppet Master submits it to PuppetDB:2016-03-19 21:23:31,247 INFO [p.p.command] [ceff648b-f67d-44db-89c5-e0f9d1e936c4] [store report] puppet v4.3.1 – web01.example.com

Show all the facts of all our nodes (be careful, there may be a lot!):

curl 'http://localhost:8080/pdb/query/v4/facts'

Show the IP addresses of all our nodes (a similar search can be for any fact):

curl 'http://localhost:8080/pdb/query/v4/facts/ipaddress'

Show the node that has a specific IP address:

curl 'http://localhost:8080/pdb/query/v4/facts/ipaddress/10.42.42.27'

Show all the facts of a specific node:

curl -X GET http://localhost:8080/pdb/query/v4/facts \
--data-urlencode 'query=["=", "certname", "web01.example.com"]'

The response is always a JSON array with an entry per fact. Each entry is like the following:

{ "certname": <node name>, (IE: www01.example.com)
  "name": <fact name>, (IE: operatingsystem)
  "value": <fact value> (IE: ubuntu) }

Show all the resources of type Mount for all the nodes:

curl 'http://localhost:8080/pdb/query/v4/resources/Mount'

Note that the resource type must be capitalized, as we are referring to the type, and not to an specific instance.

Show all the resources of a given node:

curl -X GET http://localhost:8080/pdb/query/v4/resources --data-urlencode \
  'query=["=", "certname", "web01.example.com"]'

Show all nodes that have Service['apache'] with ensure = running:

curl -X GET http://localhost:8080/pdb/query/v4/resources/Service \
--data-urlencode 'query=[ "and" , ["=", "title", "apache" ],
                  ["=", ["parameter", "ensure"], "running"] ]'

Same as before, using a different approach:

curl -X GET http://localhost:8080/pdb/query/v4/resources/Service/apache \
--data-urlencode 'query=["=", ["parameter", "ensure"], "running"]'

Show all the resources managed for a given node in a given manifest:

curl -X GET http://localhost:8080/pdb/query/v4/resources/ --data-urlencode \
  'query=["and" ["=", "file", "/etc/puppet/manifests/apache.pp"], \
                ["=", "certname", "web01.example.com"]]'

The response format of the resources endpoint shows how we can query everything about the resources managed by Puppet and defined in our manifests:

{"certname":   "<node name>", (IE: www01.example.com)
 "resource":   "<the resource's unique hash>", (IE: f3h34ds...) 
 "type":       "<resource type>", (IE: Service)
 "title":      "<resource title>", (IE: apache)
 "exported":   "<true|false>", (IE: false)
 "tags":       ["<tag>", "<tag>"], (IE: "apache", "class" ...)
 "file": "<manifest path>", (IE: "/etc/puppet/manifests/site.pp")
 "line": "<manifest line>", (IE: "3")
 "parameters": {<parameter>: <value>, (IE: "enable" : true,)
               <parameter>: <value>,
               ...}}

Show all the (not deactivated) nodes:

curl 'http://localhost:8080/pdb/query/v4/nodes'

Show all the facts of a specific node (this is a better alternative than the earlier example):

curl 'http://localhost:8080/pdb/query/v4/nodes/www01.example.com/facts'

Show all the resources of a specific node (this is a better alternative than the earlier example):

curl 'http://localhost:8080/pdb/query/v4/nodes/www01.example.com/resources'

Show all the nodes with the operating system CentOS:

curl -X GET http://localhost:8080/pdb/query/v4/nodes --data-urlencode \'query=["=", ["fact","operatingsystem"], "CentOS"]'

The response format is as follows:

{"certname": <string>,
 "deactivated": <timestamp or null>,
 "expired": <timestamp or null>,
 "catalog_timestamp": <timestamp or null>,
 "facts_timestamp": <timestamp or null>,
 "report_timestamp": <timestamp or null>,
 "catalog_environment": <string or null>,
 "facts_environment": <string or null>,
 "report_environment": <string or null>,
 "latest_report_status": <string>,
 "latest_report_hash": <string>
}

When using the facts and resources sub URLs, we get replies in the same format as the relative endpoint.

Get the whole catalog (the last saved one) of a node (all the resources and edges):

curl 'http://localhost:8080/pdb/query/v4/catalogs/www01.example.com' 

Get the names (just the names, not the values) of all the stored facts:

curl 'http://localhost:8080/pdb/query/v4/fact-names'

These are mostly useful to check PuppetDB performances and operational statistics. Some of them are visible from the performance dashboard.

Get the names of all the metrics available:

curl 'http://localhost:8080/metrics/v1/mbeans'

The result shows a remarkable list of items in JMX Mbean ObjectName style:

<Mbean-doman>:type=<Type>[,name:<Name>]

An example, in URL-encoded format as returned by PuppetDB, is as follows:

"com.jolbox.bonecp:type=BoneCP" : "/metrics/mbean/com.jolbox.bonecp%3Atype%3DBoneCP"

Available metrics are about nodes' population, database connection, delivery status of the processed commands, HTTP access hits, command processing, HTTP access, storage operations, JVM statistics, and the message queue system.

The following are few examples.

The total number of nodes in the population:

curl http://localhost:8080/metrics/v1/mbeans/com.puppetlabs.puppetdb.query.population%3Atype%3Ddefault%2Cname%3Dnum-nodes

The average number of resources per node:

curl http://localhost:8080/metrics/v1/mbeans/com.puppetlabs.puppetdb.query.population%3Atype%3Ddefault%2Cname%3Davg-resources-per-node

Statistics about the time used for the command replace-catalog:

curl http://localhost:8080/ metrics/mbeans/com.puppetlabs.puppetdb.scf.storage%3Atype%3Ddefault%2Cname%3Dreplace-catalog-time

Show the summaries of all the saved reports of a given node:

curl -X GET http://localhost:8080/pdb/query/v4/reports --data-urlencode \'query=["=", "certname", "db.example.com"]' 

Search all reports for failures:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=["=", "status" , "failure"]'

Search all reports for failures on Service type:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=[ "and", ["=", "status" , "failure"], \
                  ["=", "resource-type", "Service"] ]'

Search all reports for any change to the file with title hosts:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=[ "and", ["=", "resource-type", "File"], \
                  ["=", "resource-title", "hosts" ] ]'

Search all reports for changes in the content of the file with title hosts:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=[ "and", ["=", "resource-type", "File"], \
                  ["=", "resource-title", "hosts" ], \
                  ["=", "property", "content"] ]'

Show changes to the specified file only after a given timestamp:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=[ "and", ["=", "resource-type", "File"], \
                  ["=", "resource-title", "hosts" ], \
                  [">", "timestamp", "2015-12-18T14:00:00"] ]'

Show all changes in a timestamp range:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
  'query=[ "and", [">", "timestamp", "2015-12-18T14:00:00"] , \
                  ["<", "timestamp","2015-12-18T15:00:00"] ]'

Show all the changes related to resources provided by a specific manifest file:

curl -X GET 'http://localhost:8080/pdb/query/v4/events' --data-urlencode \
'query=["=","file","/etc/puppet/modules/hosts/manifests/init.pp"]'

Show the count of resources of type Service, summarized per resource:

curl -X GET 'http://localhost:8080/pdb/query/v4/event-counts' \
  --data-urlencode 'query=["=", "resource-type", "Service" ]' \
  --data-urlencode 'summarize-by=resource'

Show the count of resources of type Package, summarized per node name:

curl -X GET 'http://localhost:8080/pdb/query/v4/event-counts' \
  --data-urlencode 'query=["=", "resource-type", "Package" ]' \
  --data-urlencode 'summarize-by=certname'

Show the aggregated count of events for a node:

curl -G 'http://localhost:8080/pdb/query/v4/aggregate-event-counts' \
  --data-urlencode 'query=["=", "certname", "db.example.com"]' \
  --data-urlencode 'summarize-by=containing-class' 

Show the aggregated count for all events on services on any node:

curl -G 'http://localhost:8080/pdb/query/v4/aggregate-event-counts' \
  --data-urlencode 'query=["=", "resource-type", "Service"]' \
  --data-urlencode 'summarize-by=certname'

Show PuppetDB server's time, in ISO-8601 format (the format we'll deal with when querying timestamps):

curl http://localhost:8080/pdb/query/v4/server-time

Show PuppetDB's version:

curl http://localhost:8080/pdb/query/v4/version

The puppetdbquery module uses a custom format for queries which is different (and easier to use) from the native one. All the queries we can do with this module are in the following format:

Type[Name]{attribute1=foo and attribute2=bar}

By default, they are made on normal resources, using the @@ prefix to query exported resources.

The comparison operators are: =, !=, >, < and ~ (regexp matching).

The expressions can be combined with and, not, and or.

puppet help query

puppet query nodes '(osfamily=RedHat and lsbmajdistrelease=6)'

puppet query facts '(osfamily=RedHat and lsbmajdistrelease=6)'

puppet query facts --facts ipaddress '(osfamily=RedHat and lsbmajdistrelease=6)'

$webservers = query_nodes('osfamily=Debian and Class[Apache]')
$webserver_ip = query_nodes('osfamily=Debian and Class[Apache]', ipaddress)

query_facts('Class[Apache]{port=443}', ['osfamily', 'ipaddress'])

---
:backends:
  - yaml
  - puppetdb

:hierarchy:
  - nodes/%{fqdn}
  - common

ntp::servers:
  - 'ntp1.example.com'
  - 'ntp2.example.com'

ntp::servers::_nodequery: 'Class[Ntp::Server]'

ntp::servers::_nodequery: ['Class[Ntp::Server]', 'ipaddress']

ntp::servers::_nodequery:
  query: 'Class[Ntp::Server]'
  fact:  'ipaddress'

This is typically done when, in Puppet, we talk about nodes' classification: the task that the Puppet Master accomplishes when it receives a client's request and determines the classes and parameters to use when compiling the relevant catalog.

Nodes' classification can be done in different ways:

This is another crucial part, as with parameters we can characterize our nodes and define the resources we want for them.

Generally, to identify and characterize a node in order to differentiate it from the others and provide the specific resources we want for it, we need very few key parameters, such as (names used here may be common, but are arbitrary and are not Puppet's internal ones):

With parameters like these, any node can be fully identified and be served with any specific configuration. It makes sense to provide them, where possible, as facts.

The parameters and the variables we use in our manifests may have different natures, such as:

Many times, the values of these variables and parameters have to change according to the values of other variables, and it's important to have a general idea, from the beginning, of what the variations involved and the possible exceptions are, as we will probably define our logic according to them. As a general rule we will most of the time use the identifying parameters (role/env/zone...) to define most of the other parameters, so we'll probably need to use them in our Hiera hierarchy or in Puppet selectors. This also means that we will probably need to set them as top scope variables (for example via an ENC) or facts.

As with classes to include, parameters may be set by various components; some of them are actually the same, since in Puppet, node classification involves both classes to include and parameters to apply:

It's almost certain that we will need to manage configuration files with Puppet and that we need to store them somewhere, either as plain static files to serve via Puppet's fileserver functionality using the source argument of the File type, or via .erb templates.

While it's possible to configure custom fileserver shares for static files and absolute paths for templates, it's definitely recommended to rely on the modules' auto-loading conventions and place such files inside custom or shared modules, unless we decide to use Hiera for them.

Configuration files, therefore, are typically placed in the following:

We'll probably need to provide custom resources to our nodes that are not declared in the shared modules because they are too specific, and we'll probably want to create some grouping classes, for example, to manage the common baseline of resources and classes we want applied to all our nodes.

This is typically a bunch of custom code and logic that we have to place somewhere. The usual locations are as follows:

By default, Puppet doesn't use an ENC and lets us classify nodes directly in /etc/puppet/manifests/site.pp (or in files imported from there) with the node statement. So a very basic setup would have site.pp with content like the following:

node www01 {
  # Place here resources to apply to this node in Puppet DSL:
  # file, package service, mount...
}
node lb01 {
  # Resources for this node: file, package service...
}

This is all we need: no modules with their classes, no Hiera, no ENC; just good old plain Puppet code as they teach us in schools, so to speak.

This basic approach, useful just for the first tests, obviously does not scale well and would quickly become a huge mess of duplicated code.

The next step is to use classes which group resources, and if these classes are placed inside modules, we can include them directly without the need to explicitly import the containing files:

node www01 {
  include ::apache
  include ::php
}

Also, this approach, even if definitely cleaner, will quickly be overwhelmed by redundant code, and so we will probably want to introduce grouping classes that group other classes and resources according to the desired logic.

One common example is a class that includes all the modules, classes and resources we want to apply to all our nodes: a general class.

Another example is role classes, which include all the extra resources needed by a particular node:

node www01 {
  include ::general
  include ::role::www
}

We can then have other grouping classes to better organize and reuse our resources, such as the profiles we have just discussed.

node www01 {
  include ::site
  include ::site::role::www
}

Up to now, we have just included classes and often the same classes are included by nodes that need different effects from them, for example, slightly different configuration files or specific resources, or in case of any kind of variation we have in the real world while configuring the same application on different systems.

Here is where we need to use variables and parameters to alter the behavior of a class according to custom needs.

And here is where the complexity begins, because there are various elements to consider, such as the following:

We also need classes that actually allow things to be changed, via parameters or variables, if we want to avoid placing our logic directly inside them.

Patterns on how to manage variables and their effect on the applied resources have changed a lot with the evolution of Puppet and the introduction of new tools and functionalities.

We won't indulge in how things were done in the good old days. In a modern, and currently recommended Puppet setup, we expect to have the following:

In this case, we can basically follow two different approaches:

We can keep on including classes and set on Hiera the values we want for the parameters we need to modify. So, in our example we could have in site/manifests/role/www.pp something like the following:

class site::role::www {
  include ::apache
  include ::php
}

The same is true on Hiera for a file like hieradata/env/devel.yaml, where we set parameters like the following:

---
  apache::port: 8080

Alternatively, we might use explicit class declarations such as:

class site::role::www {
  $apache_port = $env ? {
    devel   => '8080',
    default => '80',
  }
  class { '::apache':
    port => $apache_port,
  }
  include ::php
}

Data, and logic on how to determine it, is definitively inside code.

The ENC and Hiera can be alternative or complementary; this approach gets advantages from both and uses the site module for most of the logic for class inclusion, the configuration files, and the custom resources.

In Hiera, all the class parameters are placed.

In the ENC, when not possible via facts, we set the variables that identify our nodes, and can be used on our Hiera hierarchy.

In site.pp in the same ENC, we include just a single site class and here we manage our grouping logic. For example, with a general baseline and role classes:

class site {
  include ::site::general
  if $::role {
    include "::site::roles::${::role}"
  }
}

In our role classes, which are included if the $role variable is set on the ENC, we can manage all the role-specific resources, eventually dealing with differences according to environment or other identifying variables directly in the role class, or using profiles.

The Foreman can act as a Puppet ENC; it's probably the most common ENC around and we can use both Foreman and Hiera in our architecture.

In this case we should strictly separate responsibilities and scopes, even if they might be overlapping. Let's review our components and how they might fit in a scenario based on the Foreman, Hiera, and the usual site module(s):

A common scenario involves the usage of Hiera to manage both the classes to include in nodes and their parameters.

No ENC is used; site.pp just needs the following, together, eventually, with a few handy resource defaults:

hiera_include('classes')

Classes and parameters can be assigned to nodes enjoying the flexibility of our hierarchy, so in a common.yaml we can have the following:

---
# Common classes on all nodes
classes:
  - puppet
  - openssh
  - timezone
  - resolver
# Common Class Settings
timezone::timezone: 'Europe/Rome'
resolver::dns_servers:
  - 8.8.8.8
  - 8.8.4.4

In a specific data source file such as role/web.yaml, add the classes and the parameters we want to apply to that group of nodes:

---
classes:
  - stack_lamp
stack_lamp::apache_include: true
stack_lamp::php_include: true
stack_lamp::mysql_include: false

The modules used (here a sample stack_lamp, but it could be something such as profile::webserver or apache and php) should expose parameters that are needed to configure things as expected.

They should also allow creation of custom resources, such as apache::vhost, by providing hashes to feed a create_resources() function present in a used class.

Configuration files and templates can be placed in a site module with, eventually, additional custom classes.

We can also use the hiera-file plugin to deliver configuration files having a Hiera-only setup. This is a somewhat extreme approach. Everything is managed by Hiera: the classes to include in nodes, their parameters, and also the configuration files to serve to clients. Also, here we need modules and relevant classes that expose parameters to manage the content of files.

Secrets, credentials, and sensitive data may be encrypted via hiera-eyaml or hiera-gpg.

We may wonder if a site module is still needed, since most of its common functions (providing custom files, managing logic, defining and managing variables) can be moved to Hiera.

The answer is probably yes; even in a similar, strongly Hiera-oriented scenario, a site module is probably needed. We might for example use custom classes to manage edge cases or exceptions that could be difficult to replicate with Hiera without adding a specific entry in the hierarchy.

One important point to consider when we move most of our logic to Hiera is how much this costs in terms of hierarchy size. Sometimes a simple (even if not elegant) custom class that deals with a particular case may save us from adding a layer in the hierarchy.

This is the Foreman alternative approach to Hiera for the full management of the variables used by nodes.

Foreman can automatically detect the parameters exposed by classes and allows us to set values for them according to custom logic, providing them as parameters for parameterized classes via the ENC functionality (support for parameterized classes via ENC has been available since Puppet 2.6.5).

To each class we can map one or more smart variables, which may have different values according to different, customizable conditions and hierarchies.

The logic is somewhat similar to Hiera, with the notable difference that we can have a different hierarchy for each variable and have other ways to define its content via matchers.

User experience benefits from web interface and may turn out to be easier than editing Hiera files directly. Foreman auditing features allow us to track changes as would a SCM on plain files.

We don't have the multiple backend flexibility that we have on Hiera and we'll be completely tied to Foreman for the management on our nodes.

Personally, I have no idea how many people are extensively using smart variables in their setups; just be aware that there exists this alternative for data management.

A facts driven approach was theorized by Jordan Sissel, Logstash's author in a 2010 blog post (http://www.semicomplete.com/blog/geekery/puppet-nodeless-configuration). The most authoritative information we can have about a node comes from its own facts.

We may decide to use facts in various places: in our hierarchy, in our site code, in templates, and if our facts permit us to identify the node's role, environment, zone, or any identifying variable, we might not even need node classification and manage all our stuff in our site module or on Hiera.

It is now very easy to add custom facts placing a file in the node's /etc/facter/facts.d directory. This can be done, for instance, by a (cloud) provisioning script.

Alternatively, if our nodes' names are standardized and informative, we can easily define our identifying variables in facts that might be provided by our site module.

If all the variables that identify our node come from facts, we can have in our site.pp a single line as simple as the following:

include site

In our site/manifests/init.pp have something like:

class site {
  if $::role {
    include "site::roles::role_${::role}"
  }
}

The top scope $::role variable would be, obviously, a fact.

Logic for data and classes to include can be managed where we prefer: on site modules, Hiera, or the ENC.

The principle here is that as much data as possible, and especially the nodes' identifying variables, should come from facts.

Also, in this case common sense applies and extreme usage deviations should be avoided; in particular, a custom ruby fact should compute its output without any local data. If we start to place data inside the fact in order to return data, we are probably doing something wrong.

We have seen that site.pp does not necessarily need to have node definitions in its content in imported files. We don't need them when we drive everything via facts, where we manage class inclusion in Hiera, and we don't need them with an approach where conditionals based on host names are used to set the top scope variables that identify our nodes:

# nodeless site.pp

# Roles are based on hostnames
case $::hostname {
  /^web/: { $role = 'web' }
  /^puppet$/: { $role = 'puppet' }
  /^lb/: { $role = 'lb' }
  /^log/: { $role = 'log' }
  /^db/: { $role = 'db' }
  /^el/: { $role = 'el' }
  /^monitor/: { $role = 'monitor' }
  default: { $role = 'default' }
}

# Env is based on hostname or (sub) domain
if 'devel' in $::fqdn { $env = 'devel' }
elsif 'test' in $::fqdn { $env = 'test' }
elsif 'qa' in $::fqdn { $env = 'qa' }
else { $env = 'prod' }

include site
# hiera_include('classes')

Here, the $role and $env variables are set at top scope according to hostnames that would benefit from a naming standard we can parse with Puppet code.

At the end, we just include our site class or use hiera_include to manage the grouping logic for what classes to include in our nodes.

Such an approach makes sense only where we don't have to manage many different hostnames or roles, and where the names of our nodes follow a naming pattern that lets us derive identifying variables.

The introduction of parameterized classes, with Puppet 2.6, has been a crucial step in standardizing the interfaces of classes. On earlier versions there was no unique way to pass data to a class. Variables defined anywhere could be dynamically used inside Puppet code or in templates to manage the module's behavior; there was no standard API to access or set them. We used to define parameter less classes as follows:

class apache {
  # Variables used in DSL or in templates were dynamically scoped 
  # and referenced without using their fully qualified name.
  # IE: $port, not $apache::port or $::apache_port
}

To declare them always and only with Apache as follows:

include apache

The introduction of parameters in classes has been important because it allowed a single entry point for class data:

class apache (
  $port = 80  ) {
}

The default value of the parameter could be overridden with an explicit declaration, such as the following:

class { 'apache':
  port => 8080,
}

Such a solution, anyway, has not been completely decisive: usage of parameterized classes introduced new challenges, such as the need to declare them only once in our catalog for each node. This forced people to rethink some of their assumptions on how and where to make class inclusion in their code.

We still could include apache as many times as we wanted in a catalog but we didn't have any method to set specific parameters if the class didn't explicitly manage a way to lookup for external variables, for example with a syntax like the following:

class apache (
  $port = hiera('apache::port','80') {
}

This, obviously, would have required all the module's users to use Hiera.

I wouldn't dare to say that the circle has been closed in Puppet 3's data bindings: the automatic Hiera lookup of class parameters allows setting parameters via both explicit declaration and plain inclusion with parameter values set in Hiera.

After years of pain, alternative solutions, creative and unorthodox approaches and evolution of the tool, I'd say that now the mainstream and recommended way to use classes is to just include them and manage their parameters on Hiera, using Puppet 3's data bindings feature.

On the manifests, we can declare classes with the following:

include apache

Be sure that whatever parse order is followed by Puppet, its data can be defined in Hiera files, so with a YAML backend, we'll use a syntax as simple as the following:

---
apache::port: '8080'

When people had to cope with different OS in a module, they typically started using selectors or conditionals for assigning variables or parameters the correct values according to facts such as operatingsystem and operatingsystemrelease, and the more recent osfamily.

A typical case with a selector would be as follows:

class apache {
  $apache_name = $::operatingsystem ? {
    /(?i:Debian|Ubuntu|Mint)/       => 'apache2',
    /(?i:RedHat|CentOS|Scientific)/ => 'httpd',
    default                         => 'apache'
  }
  package { $apache_name:
    ensure => present,
  }
}

Having this mix of variable definitions and resource declarations was far from elegant and in some time people started to place in a dedicated class, usually called params, the management of the module's variables.

They can be set with selectors, as in the previous example, or, more commonly, inside case statements, always based on facts related to the underlying OS:

class apache::params {
  case $::osfamily {
    'RedHat': {
      $apache_name = 'httpd'
    }
    'Debian': {
      $apache_name = 'apache2' 
    }
    default: {
      fail("Operating system ${::operatingsystem} not supported")
    }
  }

The main class has to just include the params class and refer to internal variables using their fully qualified name:

class apache {
  include apache::params
  package { $apache::params::apache_name:
    ensure => present,
  }
}

This is a basic implementation of the so-called params pattern, and has the advantage of having a single place where we define all the internal variables of a module or the default values for its parameters.

In the next example, the package name is also exposed as a parameter (this can be considered a reusability feature, as it allows users to override the default package name for the application that is going to be installed), and since the default value is defined in params.pp, the main class has to inherit it:

class apache (
  $package_name = $apache::params::package_name,
) inherits apache::params {
  package { $apache::params::package_name:
    ensure => present,
  }
}

The params pattern has been widely used and it works well. Still, it embraces a code and data mixup that so many wanted to avoid.

The first proposals about the way to separate modules' data from their code date back to 2010, with a blog post from Dan Bode titled Proposal: Managing Puppet's configuration data.

At that time, Hiera was still not available (the post refers to its ancestor, extlookup) but most of the principles described there have been considered when a solution was implemented.

When Hiera was introduced, it seemed a good solution to manage OS related variations via its hierarchy. It soon became clear, anyway, that global site related data, as it can be the one we place on our data sources, is not an appropriate backend for modules' internal data if we want them to be reusable and distributable.

Possible solutions, inspired or derived from Dan's post, have been identified for some time, but only with the release of Puppet 3.3.0 was it converted into reality as an experimental feature that finally addressed what's generally summarized by the term Data in modules: have a dedicated hierarchy inside the module and its relevant Hiera data.

It seemed finally, the long sought after solution to have data separation also for modules internal data, but it failed to pass Puppet Labs' user acceptance tests.

It was not so easy for modules authors to manage and this implementation has been removed in the following Puppet versions, but the issue is too important to be ignored, so R.I.Pienaar proposed an implementation based on an independent module: https://github.com/ripienaar/puppet-module-data

This approach is much simpler to use, doesn't require big changes to existing code; being implemented as a module, it can be used on most Puppet installations (version 3.x is required), and does exactly what we expect.

Besides the init.pp file, with the main class, all the other classes defined in the manifests directory of a module can have any name. Still, some names are more common than others. We have seen that params.pp is a sort of standard de facto pattern, and is not the only one. It's common, for example, to have files like server.pp and client.pp with subclasses to manage the server/client components of an application.

R.I.Pienaar (definitively one of the most influential contributors to Puppet's evolution) suggested in a blog post a module layout that involves splitting the main modules' resources in three different classes and relevant files: install.pp to manage the installation of the application, config.pp to manage its configuration, and service.pp to manage its service(s). So a typical package-service-configuration module can have in its init.pp file something like the following:

class postgresql […] { […]
  class{'postgresql::install': } ->
  class{'postgresql::config': } ~>
  class{'postgresql::service': }
}

And in the referred sub classes the relevant resources to manage.

This pattern has pros and cons. Its advantages are as follows:

Some drawbacks can be as follows:

In any case, be aware that such an approach requires the contain function (available from Puppet 3.4.0) or the usage of the anchor pattern to work correctly.

Puppet has had a long-standing issue that affected and confused many users for years: https://tickets.puppetlabs.com/browse/PUP-99. One of its effects is that when we define a dependency on a class, Puppet extends that dependency to the resources declared in that class, as we may expect, but not to other classes eventually declared (included) there.

This may create problems and lead to unexpected behaviors (dependencies not managed in the order expected) when referring to a class like the postgresql we have seen, where other sub classes are declared.

A widely used work around is the anchor pattern, defined by Puppet Labs' Jeff McCune.

It is based on the anchor type, included in Puppet Labs' stdlib module, which can be declared as a normal resource:

anchor { 'postgresql::start': }
anchor { 'postgresql::end': }

This can then be used to contain the declared classes in a dependency chain:

anchor { 'postgresql::start': } -> 
class{'postgresql::install': } ->
class{'postgresql::config': } ~>
class{'postgresql::service': } ->
anchor { 'postgresql::end': }

In this way, when we create a relationship that involves a whole class, like the following, we are sure that all the resources provided in the postgresql class are applied before the wordpress class once, because they are explicitly contained in the anchor resource type:

class { 'postgresql': } -> class { 'wordpress': }

The modules ecosystem has grown erratically, with many people redoing modules for the same application, either forking existing ones or developing them from scratch.

In this case, quantity is not a plus: there's really no advantage for Puppet users to have dozens of different modules for a single application, written with different layouts, parameters names, entry points, OS coverage, and features sets.

Even worse, when we use modules that have dependencies on other modules (as described in Modulefile), we may quickly end up having conflicts that may prevent us from installing a module with the puppet module tool and force us to fork and fix, just for our case.

Module interoperability is a wider issue which has not yet been solved, but there is something that can be done if there is the common will: an attempt to standardize naming conventions for modules' class and parameters names.

The various benefits are as follows:

It's common sense that such an effort should be driven by Puppet Labs, but for the moment, it remains a community effort, under the label stdmod: https://github.com/stdmod.

The naming conventions defined there embrace some standard de facto naming and try to define general rules for parameters and class names. For the ones that affect the resource arguments, the pattern is [prefix_]resource_attribute, so these may be some parameters names:

config_file_path, config_file_content, package_name, init_config_file_path, client_package_name

Many other names are defined; the basic principle is that a stdmod compliant module is not expected to have them all, but if it exposes parameters that have the same functions, it should call them as defined by the standard.

Everything is a file in UNIX, and Puppet, most of the time, manages files.

A module can expose parameters that allow its users to manipulate configuration files and it can follow one or both the files/setting approaches, as they are not alternative and can coexist.

To manage the contents of a file, Puppet provides different alternative solutions:

For the first two cases, we can expose parameters which allow us to define the module's main configuration file, either directly via the source and content arguments, or by specifying the name of the template to be passed to the template() function:

class redis (
  $config_file           = $redis::params::file,
  $config_file_source    = undef,
  $config_file_template  = undef,
  $config_file_content   = undef,
  ) {

We can manage the configuration file arguments with the following:

  $managed_config_file_content = $config_file_content ? {
    undef   => $config_file_template ? {
      undef   => undef,
      default => template($config_file_template),
    },
    default => $config_file_content,
  }

The $managed_config_file_content variable computed here takes the value of the $config_file_content, if present; otherwise it uses the template defined with $config_file_template. If this parameter is unset, the value is undef.

  if $redis::config_file {
    file { 'redis.conf':
      path    => $redis::config_file,
      source  => $redis::config_file_source,
      content => $redis::managed_config_file_content,
    }
  }
}

In this way, users can populate redis.conf, either via a custom template (placed in the site module), as follows:

class { 'redis':
  config_file_template => 'site/redis/redis.conf.erb',
}

Or directly provide the content attribute, as shown here:

class { 'redis':
  config_file_content => template('site/redis/redis.conf.erb'),
}

Or, finally, provide a fileserver source path, such as the following:

class { 'redis':
  config_file_source => 'puppet:///modules/site/redis/redis.conf',
}

In case users prefer to manage the file in other ways (augeas, concat, and so on), they can just include the main class, which by default does not manage the configuration file contents, and use whichever method they prefer to alter them:

class { 'redis': }

A good module could also provide custom defines that allow easy and direct ways to alter configuration files' single lines, either using augeas or other in-file line management tools.

class openssh (
  $config_file_options_hash   = { },
}

# File Managed by Puppet
[...]
  Port <%= scope.function_hash_lookup(['Port','22']) %>
  PermitRootLogin <%= scope.function_hash_lookup(['PermitRootLogin','yes']) %>
  UsePAM <%= scope.function_hash_lookup(['UsePAM','yes']) %>
[...]

  Port <%= scope.lookupvar('openssh::config_file_options_hash')['Port'] ||= '22' %>
  PermitRootLogin <%= scope.lookupvar('openssh::config_file_options_hash')['PermitRootLogin'] ||= 'yes' %>
  UsePAM <%= scope.lookupvar('openssh::config_file_options_hash')['UsePAM'] ||= 'yes' %>
[...]

---
openssh::config_file_template: 'site/openssh/sshd_config.erb'
openssh::config_file_options_hash:
  Port: '22222'
  PermitRootLogin: 'no'

class { 'openssh':
  config_file_template     => 'site/openssh/sshd_config.erb'
  config_file_options_hash => {
    Port            => '22222',
    PermitRootLogin => 'no',
  }
}

class redis (
  $config_dir_path            = $redis::params::config_dir,
  $config_dir_source          = undef,
  $config_dir_purge           = false,
  $config_dir_recurse         = true,
  ) {
  $config_dir_ensure = $ensure ? {
    'absent'  => 'absent',
    'present' => 'directory',
  }
  if $redis::config_dir_source {
    file { 'redis.dir':
      ensure  => $redis::config_dir_ensure,
      path    => $redis::config_dir_path,
      source  => $redis::config_dir_source,
      recurse => $redis::config_dir_recurse,
      purge   => $redis::config_dir_purge,
      force   => $redis::config_dir_purge,
    }
  }
}

class { 'redis':
  config_dir_source => 'puppet:///modules/site/redis/conf/',
}

class { 'redis':
  config_dir_source => [
                  "puppet:///modules/site/redis/conf--${::fqdn}/",
"puppet:///modules/site/redis/conf-${::role}/",
                  'puppet:///modules/site/redis/conf/' ],
  config_dir_purge  => true,
}

class postgresql (
  $hba_file_path             = $postgresql::params::hba_file_path,
  $hba_file_template         = undef,
  $hba_file_content          = undef,
  $hba_file_options_hash     = { } , 
  ) { […] }

apache::vhost { 'vhost.example.com':port    => '80',docroot => '/var/www/vhost',
}

class elasticsearch (
  $user_class          = 'elasticsearch::user',
  ) { [...]
  if $elasticsearch::user_class {
    require $elasticsearch::user_class
  }

class { 'elasticsearch':
  user_class => 'site::users::elasticsearch',
}

class { 'elasticsearch':
  user_class => '',
}

class postgresql (
  $install_class             = 'postgresql::install',
  $config_class              = 'postgresql::config',
  $setup_class               = 'postgresql::setup',
  $service_class             = 'postgresql::service',
  […] ) { […] }

class elasticsearch (
  $install_class       = 'elasticsearch::install',
  $install             = 'package',
  $install_base_url    = $elasticsearch::params::install_base_url,
  $install_destination = '/opt',
  ) {

include $install_class

class elasticsearch::install {
  case $elasticsearch::install {
    package: {
      package { $elasticsearch::package:
        ensure   => $elasticsearch::managed_package_ensure,
        provider => $elasticsearch::package_provider,
      }
    }
    upstream: {
      puppi::netinstall { 'netinstall_elasticsearch':
        url             => $elasticsearch::base_url,
        destination_dir => $elasticsearch::install_destination,
        owner           => $elasticsearch::user,
        group           => $elasticsearch::user,
      }
    }
    default: { fail('No valid install method defined') }
  }
}

class elasticsearch (
  $my_class            = undef,
  ) { [...]
  if $elasticsearch::my_class {
    include $elasticsearch::my_class
    Class[$elasticsearch::my_class] ->  
    Anchor['elasticsearch::end']
  }
}

class network (
  $interfaces_hash           = undef,
  […] ) { […]
  if $interfaces_hash {
    create_resources('network::interface', $interfaces_hash)
  }
}

---
  network::interfaces_hash:
    eth0:
      method: manual
      bond_master: 'bond3'
      allow_hotplug: 'bond3 eth0 eth1 eth2 eth3'
    eth1:
      method: manual
      bond_master: 'bond3'
    bond3:
      ipaddress: '10.10.10.3'
      netmask: '255.255.255.0'
      gateway: '10.10.10.1'
      dns_nameservers: '8.8.8.8 8.8.4.4'
      bond_mode: 'balance-alb'
      bond_miimon: '100'
      bond_slaves: 'none'

There are different modules for each OpenStack component (such as Nova, Glance, Horizon, Cinder, Ceilometer, Keystone, Swift or Quantum/Neutron). They can be retrieved from https://github.com/openstack/puppet-<component>, so, for example, Nova's module can be found at https://github.com/openstack/puppet-nova.

These modules manage all the different configurations via a settings-based approach, with native types that set the single lines of each configuration file (which may be more than one for each component) and with different subclasses that expose all the parameters needed to manage different services or features of each component.

For example, in the Nova module, you have native types like nova_config or nova_paste_api_ini that are used to manage single lines respectively in the /etc/nova/nova.conf and /etc/nova/api-paste.ini configuration files.

These types are used in classes like nova::compute, nova::vncproxy (and many others) to set specific configuration settings according to the parameters provided by users.

There is also a nova::generic_service which manages the installation and the service of specific Nova services.

Finally, subclasses like nova::rabbitmq or nova::db::mysql manage the integration with third-party modules to create, for example, database users and credentials.

A similar structure is replicated on the modules of the other OpenStack components with the added benefit of having a coherent and predictable structure and usage pattern, which makes users' lives much more comfortable.

The general approach followed, therefore, for OpenStack's component modules is to have modules with multiple entry points for data, basically one for each subclass, with configuration files managed with a settings-based approach.

These single component modules can be composed in different ways; we will review a few of them:

The official and general openstack module (https://github.com/stackforge/puppet-openstack) can manage the specific roles in an OpenStack infrastructure as it is dedicated to computing, storage, networking or controllers and allow users to quickly and easily define specific (although rather limited) topologies.

For example, the openstack::all class manages an all-in-one installation of all the components on a single node, openstack::controller installs the OpenStack controller components on a node, and the openstack::compute class installs the OpenStack compute components.

All these classes expose parameters that allow users to set public and private addresses, users' credentials, network settings, and whatever is needed to configure, at high level, the whole OpenStack environment. In many cases the parameters and parts of the code are duplicated, for example, the openstack::all and openstack::controller classes have many parts in common and for each parameter added to the main OpenStack classes there's the need to replicate it on the declared component classes.

This openstack module can be definitively considered a module that operates at a higher abstraction layer and uses classes and defines application modules but has some severe limitations on flexibility and maintainability. Its development has been discontinued in favor of Puppet Labs' module that we'll see next.

Puppet Labs has published another higher abstraction module to manage OpenStack (probably the one used for their internal infrastructure) which is an alternative to Stack Forges puppet-openstack but uses the same mainstream component modules.

You can find it at https://github.com/puppetlabs/puppetlabs-openstack (there are also versions for earlier OpenStack releases, as puppetlabs-havana for Havana or puppetlabs-grizzly for grizzly) and it's definitely worth a look as it's also a good real-life example of the application of the roles and profiles pattern.

On your nodes you include the relevant role classes:

node /^controller/ {
  include ::openstack::role::controller
}
node /^network/ {
  include ::openstack::role::network
}
node /^compute/ {
  include ::openstack::role::compute
}

The role classes include the profile classes and manage a high level dependency order, for example in openstack::role::controller, which is the most complex:

class openstack::role::controller inherits ::openstack::role {
  class { '::openstack::profile::firewall': }
  class { '::openstack::profile::rabbitmq': } ->
  class { '::openstack::profile::memcache': } ->
  class { '::openstack::profile::mysql': } ->
  class { '::openstack::profile::mongodb': } ->
  class { '::openstack::profile::keystone': } ->
  class { '::openstack::profile::swift::proxy': } ->
  class { '::openstack::profile::ceilometer::api': } ->
  class { '::openstack::profile::glance::auth': } ->
  class { '::openstack::profile::cinder::api': } ->
  class { '::openstack::profile::nova::api': } ->
  class { '::openstack::profile::neutron::server': } ->
  class { '::openstack::profile::heat::api': } ->
  class { '::openstack::profile::horizon': }
  class { '::openstack::profile::auth_file': }
}

In the profile classes the required resources are finally declared using both the official OpenStack components modules and classes and defines from other modules. Some of these modules are not part of the OpenStack modules, but are needed for the setup, such as Puppet Labs' firewall module, to manage firewalling, or the MongoDB, MySQL, RabbitMQ, Memcache modules. Here you can also find spot resources declarations like SELinux settings or the usage of create_resources() to manage OpenStack's users.

These profile classes don't expose parameters but access configuration data through explicit hiera() function calls. Hiera's name spacing is quite clear and well-organized and users can configure the whole setup with YAML files with content like:

openstack::region: 'openstack'
######## Network
openstack::network::api: '192.168.11.0/24'
openstack::network::api::device: 'eth1'
openstack::network::external: '192.168.22.0/24'
openstack::network::external::device: 'eth2'
openstack::network::management: '172.16.33.0/24'
openstack::network::management::device: 'eth3'
openstack::network::data: '172.16.44.0/24'
openstack::network::data::device: 'eth4'

######## Fixed IPs (controller)
openstack::controller::address::api: '192.168.11.4'
openstack::controller::address::management: '172.16.33.4'
openstack::storage::address::api: '192.168.11.5'
openstack::storage::address::management: '172.16.33.5'

######## Database
   openstack::mysql::root_password: 'spam-gak'
openstack::mysql::allowed_hosts: ['localhost', '127.0.0.1', '172.16.33.%']

######## RabbitMQ
openstack::rabbitmq::user: 'openstack'
openstack::rabbitmq::password: 'pose-vix'

The module supports only RedHat-based distributions and, compared to Stack Forge's official OpenStack module, looks better organized, with less code duplication and some added flexibility.

Let me tell you a small personal story about Puppet, OpenStack, and choices. I had been asked by a customer to puppetize a multi-region fully HA OpenStack infrastructure.

The customer's internal crew had great OpenStack skills and no Puppet experience, they configured manually an internal testing setup and they wanted to be able to quickly reproduce it on different data centers.

At that times, I had no experience of OpenStack and started to study both it and its existing Puppet modules.

Soon I realized that it would have been quite a pain to reproduce the already existing customer's configuration files with the settings-based OpenStack's official modules and that it would have been quite hard for the crew to manage their configurations with a pure data driven approach.

The situation therefore was:

I opted for what can be definitely considered a bad idea: to write my own OpenStack modules, well aware that I was trying to do it alone in a few days, without even knowing the product, part of which many skilled and knowledgeable people had done in months of collaborative work.

Modules are published on GitHub and are based on the reusability and duplicability patterns I've refined over the years: they all have a standard structure that can be quickly cloned to create a new module with limited effort.

For example, the Nova module (https://github.com/example42/puppet-nova) has standard stdmod compliant parameters in the nova class that allow users to freely manage packages, services, and configuration files. OpenStack components like Nova, may have different services to manage, so I copied the idea of a general purpose nova::generic_service definition to manage them and added a useful definition to manage any configuration file: nova::conf.

Once defined the basic structure of all the other modules were cloned and adapted to manage the other OpenStack components (which most of the time was just a matter of a few retouches on the module's params.pp manifest).

Configurations are delivered via ERB templates, defined as all the classes grouping logic in a single site module. Data is stored on Hiera, and is mostly limited to endpoint addresses, credentials, networking, and other parameters that may change per region or role.

Most of the complexity of an OpenStack configuration is managed as static or dynamic data in templates. The structure is therefore not particularly flexible but reproduces the designed and requested layout.

Adaptation to different topologies is not supposed to be frequent and is a matter of changing templates and logic in the site class.

Tuning of configurations, which might be more frequent, is a matter of changing the ERB templates, so it is probably easier for the local crew to manage them directly.

Frankly, I'd hardly recommend the usage of these modules to setup an OpenStack infrastructure, but they can be useful to replicate an existing one or to manage it in a file-driven way.

The general lesson we can take from this is the usual golden one, especially true in the Puppet world: there's never a best way to do things, but there is the most appropriate way for a specific project.

If we work on a startup or a new project, we have the opportunity to create our infrastructure from scratch, which is often a preferred, easier, more comfortable, and safer option for various reasons:

When we need to automate existing systems things are quite different, as we have to cope with stratifications of years of many people's work with different skills under variable conditions. They may be the result of a more or less frantic and inorganic growth and evolution of an IT infrastructure, whose design patterns may have been changed or evolved with time. This might not be the case for everybody, but is quite common with infrastructures where a configuration management tool has not been steadily introduced.

We will probably have to manage:

We have two possible approaches to follow when dealing with existing systems. They are not mutually exclusive, we can adopt both of them in parallel, evaluating for each node which one is the most fitting:

What has been described in the previous paragraphs might have been a common situation some years ago, now times have definitely changed and people have been using configuration management tools and cloud computing for years.

Actually, it is likely that we might already have a Puppet setup and we are looking for ways to reorganize and refactor it.

This is becoming a quite common situation: we learn Puppet while we work on it and during the years of usage patterns evolve, the product offers new and better ways to do things, and we understand better what has worked and what hasn't.

So we might end up in scenarios where our Puppet architecture and code is in different shapes:

Different cases may need different solutions, they can follow some of the patterns we describe for the setup of new infrastructures, or the migration or update of existing ones, but here the work is mostly done on the Puppet Master rather than the clients, so we might follow different approaches:

A gradual approach involves a step-by-step process that needs informed decisions, reiterations and verification of what is done, evaluating each case according to its uniqueness.

The Puppet coverage of our infrastructure can be seen in two dimensions, on one side the number of nodes managed by Puppet, and on the other side the percentage of configurations managed by Puppet in these nodes. So we can extend our Puppet coverage following two axes:

The delivery process should be iterative and can be divided in to five phases:

The first iterations will probably concentrate on developing the common baseline of resources to apply to all the nodes. Then these configurations can be propagated both horizontally and vertically, looping on the above phases with more or less emphasis on some of them according to the specific step we are making. Each iteration might last days or a few minutes.

Let's explore more deeply what happens during the phases of each reiteration.

Before starting to run Puppet on an existing server's infrastructure, we need to know it.

It sounds obvious, it is obvious.

So obvious that sometimes we forget to do it properly.

We have to know what we want to automate and we must gather as much info as possible about:

We may be able to find much of this information from an existing inventory system or internal documentation, so it should not be hard to gather it and have a general idea of the numbers involved.

For each application or system resource that we plan to manage with Puppet we also need to know further details like:

We said that automation of an existing infrastructure is a gradual process, where we start from 0% of systems and configuration coverage and step by step we move them to a state where they are Puppet managed. We might not reach a full 100% coverage, and for each step we might consider what's worth managing with Puppet and what's not.

Ideally, every server of our infrastructure should be fully managed by Puppet and we should be able to provision a system from scratch and have it up and running after a Puppet run: the setup procedure is fully automated, controlled, replicable, and scalable.

There are cases anyway where this rule is not necessarily recommended. When the setup procedure of an application requires execution of few specific commands, with varying arguments that might not be easily predictable or which needs users' interactivity, automation becomes difficult and expensive, in terms of implementation times.

If such setups have to be done only once on a system (during installation time), and are supposed to be done on very few systems that have a medium to long life, we may question if it makes sense to try to do it with Puppet, spending several hours on what can be quickly done manually in few minutes.

This becomes even more evident when we have to manage with Puppet an existing system where such a setup is already done, or when the version of our application may change in the future and the setup procedure will be different, frustrating all our efforts to fully manage it with Puppet.

So, when we face the big task of automating an infrastructure, we definitely have to set priorities and make choices, decide what makes sense to automate first and what can be postponed or simply ignored.

We have to consider different tasks such as:

When we know our territory and have set priorities on what to do, we have more opportunities to make decisions. There are many choices to ponder but we don't need to make them all at once. Let's start from the priorities we've set and define how to behave case by case.

Among the decisions to make, we have to evaluate:

We have gathered info about our systems, set priorities, and evaluated solutions. Now it's finally time to write some code. The previous steps may have involved a variable amount of time, from a few minutes to days. If we have spent too much time on them, we have probably done something wrong, or too early, or we might have made too much upfront planning instead of a step-by-step approach where the single phases reiterate relatively quickly.

In any case, when we need to write Puppet code in line with our priorities we should:

Develop the common baseline for all our systems: This is probably the first and most useful thing to do. We can start to cover the most important and used OS in our infrastructure, we don't need to manage at the beginning the general baseline of all our servers. The common classes and resources can be placed in a grouping class (something like ::site::general) which will likely include other classes which might be both from shared modules (::openssh, ::ntp) or from custom ones (::site::users, ::site::hardening). It makes sense also to include in ::site::general, a class that includes a subset of common resources, something we can call ::site::minimal. This can be useful to let us install Puppet with minimal configurations (eventually just what's needed to manage the same Puppet client and nothing more) on as many servers as possible and then gradually extend coverage of new systems configurations just by moving resources and classes from ::site::general to ::site::minimal. These are just samples on how practically a common baseline might be introduced and mileages may vary.

Besides the macro activities that involve high level planning, we will face at every step some micro decisions on how to manage the configurations for a given application on our servers. There are some common rules to consider and they replicate the phases we have described that involve information gathering and decisions on the approaches to follow.

We must know how the configuration file(s) of an application changes on our systems and this is generally affected by two factors: operating system and infrastructure logic.

Operating systems differences are influenced by the version of the installed application (that might support different configuration options), how it has been packaged, and where files are placed. Even a configuration file as common as /etc/ssh/sshd_config can change in a relevant way on a different OS.

Infrastructure logic may affect in various ways the content of a file. Various factors strictly related to our infrastructure can determine the values of single parameters, for example we can have servers with the very same role that need to use different servers or domains according to the region or the datacenter where they are located. Also, two nodes running the same software may need a completely different configuration depending on their role, as with Puppet configuration in the case of servers and clients.

According to how the configuration files of an application change on our servers we can use different approaches to how we manage them with Puppet:

So we are finally ready to run our Puppet code and apply it to our servers.

Some of the precautions to take here can be applied whenever we work with Puppet. Some are more specific to cases where Puppet is introduced on existing systems and it changes running configurations.

The procedure to follow when we have to apply new Puppet code can definitely vary according to the amount of the managed node, their importance for our business, and the maintenance rules we follow in our company.

Still, a few common points can definitely be recommended:

We should have realized by now that there are things that Puppet does well and things that are not properly in its chords. Puppet is great at managing files, packages and services, and is less effective in executing plain commands especially if they happen occasionally.

Once we start to introduce Puppet in our infrastructures we should start, where possible, to make things in a way that favors its implementation.

Some time ago in a conference open space I remember a discussion with a developer about how to make a Java application more manageable with Puppet. From his point of view a good API could be a solid approach to make it more configurable. All the systems administrators, and Puppet users, replied that actually that was far from being a nice thing to have: it is much better to have good old plain configuration files.

If we can express it with a file, we can easily manage it with Puppet.

Whatever requires the execution of commands, even if doable, is always considered with skepticism. Installation of software should definitely be done via the OS native packages. Even if most of us have done our definitions that fetch a tar ball, unpack it and compile it, nobody really likes it: that involves a procedural and not a declarative approach, it makes idempotence more difficult and is definitely worse to express with Puppet DSL.

Services should be managed following the OS native approach, be that init, systemd, upstart, or whatever. Managing them with the execution of a command in a custom startup script is definitely not Puppet friendly.

Configuration files from common stacks of applications should be leveraged. For example when we have to manage the Java settings on an application server it makes sense to manage them always in the same place; standardizing them is a matter of overall coherence and easier management.

Once we gain affinity and expertise with Puppet we quickly understand what are the things that can be done quickly and safely and what are the things that make our life harder, requiring more resources or quick and dirty patches.

It is our responsibility to discuss this with the developers and the stakeholders of an application to find a collaborative approach to its setup that makes it easier to be managed with Puppet while preserving the requested implementation needs.

During this chapter we have seen some characteristics that make working with Puppet a much better experience. Let's remark on some of these characteristics with insights about how its deficiencies can make our code more complex or less likely to behave in a deterministic way. Some of these characteristics depend on how the software is packaged, but others may depend on the software itself. Let's look at them:

In most of the cases we cannot take the decision to use software just because it's more Puppet friendly. But if we are managing our infrastructure with Puppet it's worth considering these characteristics. If we can choose between different options, we can add these arguments to the balance of the software that implements them. If we cannot choose, check if the software to be used can be used in a Puppet-friendly way even if it's not its default behavior.

Geppetto (http://puppetlabs.github.io/geppetto/) is one of the tools of reference for writing Puppet code. It is an open source software by Puppet Labs, which is a result of the acquisition of the startup Cloud smith that developed it.

We can install Geppetto as a standalone application or as an Eclipse plugin, if we are already using it.

It has very useful features such as:

When we launch it, we are prompted for a directory from which to create our Geppetto workplace. In a workplace, we can create projects that most of the time are modules we can directly import from the Forge or a SCM repository.

If we have to think about an evergreen tool that has accompanied many sys admins' keyboards for many years, I think that Vim is one of the first names that comes to mind.

Its power and flexibility is also well expressed in how it can be made to manage Puppet code effectively.

There are two Vim bundles that are particularly useful for us:

To install them easily we need to install Pathogen (a Vim extension that allows easy management of plugins) by copying the ~/.vim/autoload/pathogen.vim file from https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim.

Then, be sure to have Pathogen installed in the ~/.vimrc of our home directory:

call pathogen#infect()
syntax on
filetype indent plugin on

Once we have Pathogen loaded, we can easily add bundles in the .vim/bundle directory:

git clone git://github.com/rodjek/vim-puppet.git .vim/bundle/puppet
git clone git://github.com/scrooloose/syntastic.git .vim/bundle/syntastic

Now, we can use Vim with these powerful plugins that make coding with Puppet DSL definitively more comfortable.

Git is generally available as the native package in every modern OS. Once we have installed it, we can configure our name and e-mail (that will appear in all our commits) with:

git config --global user.name "Alessandro Franceschi"
git config --global user.email al@lab42.it

These commands simply create the relevant entries in the ~/.gitconfig file. We can add more configurations either by directly editing this file or with the git config command, and check their current values with git config -l.

To create a Git repository, we just have to move into the directory that we want to track and simply type git init. This command initializes a repository and creates a local .git directory where all Git data is stored.

Now we can type git status to see the status of the files in our repository. If we have files in this directory, we will see them as untracked. This means that they are not managed under Git and, before being able to commit changes on our files, we have to stage them; this is done by the git add command, which adds all the files from the current working directory to the index (we can add only specific and selected files or changes). If we type git status again, we will notice that the files we added are ready to be committed, we can use git commit to create a commit with all the changes in the files we've previously staged. Our default editor is open and we can type the title (first line) and the description of our commit.

For later readability and better management, it is important to make single and atomic commits that involve only the changes for a specific fix, feature, or ticket.

Now we can type git log to see the history of our commits on this repository.

Git is a distributed SCM; we can work locally on a repository, which is a clone of a remote one. To clone an existing repository, we can use the git clone command (we have seen earlier, with Vim bundles, some usage samples).

Once we have cloned a repository, we can use git push to update it with our local commits and git pull to retrieve all the changes (commits) made there since we cloned it (or made our latest pull).

We may want to work on separated branches where we can manage different versions of our code. To create a new branch, we use git branch <branchname>, to work inside a branch we type git checkout <branchname>. When we are inside a branch, all the changes we make and commit are limited and confined to that branch; if we switch branches, we will see our changes to the local files magically disappearing. To incorporate the commits made in a branch into another branch, we can use the git merge or git rebase commands.

These are just the very basics of Git, which just shows the kind of commands we might end up using when working with it, if we want to learn more, there are some great resources online:

Git hooks are scripts that are executed when specific Git actions are done. We can add a hook just by placing an executable file under our local .git/hooks directory.

The name of the file reflects the phase when the hook is executed.

When we deal with Puppet, we have generally three different hooks that we can use:

Usage possibilities for hooks are endless; you can find some useful ones at https://github.com/drwahl/puppet-Git-hooks.

When we want to manage our Puppet code with Git, we can follow different approaches. Our main objectives are:

The first two points are implicit with Git usage; the third one can be achieved using Puppet's environments. Since we can set different local directories for the modulepath and the manifest that are mapped to different Puppet environments on the Puppet Master, we can couple different Git branches or working repositories to them.

Here, we outline two possible approaches:

[main]
  server = puppet.example.com
  environment = production
  confdir = /etc/puppet
[master]
  environment = production
  manifest = $confdir/environments/$environment/manifests/site.pp
  modulepath = $confdir/environments/$environment/modules

When we work with Git, the natural companion to manage peer review and workflow authorization schemes is Gerrit, a web interface made by Google for the development of Android, that integrates perfectly with Git and allows commenting on any commit, allows you to vote for them, and has users that may authorize their acceptance.

Different user roles with different permission schemes and authorization workflows can be defined. Once new repositories are added, they can be cloned via Git:

git clone ssh://al@gerrit.example.com:29418/puppet-modules

When we work on repositories managed under Gerrit, we have to configure our Git to push to the special for ref, instead of the default heads. This is the key step that allows us to push our commits to a special intermediary place where they can be accepted or rejected:

git config remote.origin.push refs/heads/*:refs/for/*

We also need a precommit hook that automatically places a (required) Change-ID in our commits:

gitdir=$(git rev-parse --git-dir) ; scp -p -P 29418 \al@gerrit.example.com:hooks/commit-msg ${gitdir}/hooks/

Once we have made our setup, we can normally work on our local repo with Git, and when we push our changes they are submitted to Gerrit, to the special refs/for for code peer review. When a commit is accepted, Gerrit moves it to refs/heads from where it can be pulled and distributed.

Gerrit can be introduced at any time in our workflow to manage the acceptance of commits in a centrally managed way and, besides the client's configurations, we have seen it doesn't require further changes in our architecture.

There are various online services that can be used for peer review and provide an alternative to an on-premise solution based on Gerrit.

GitEnterprise (https://gitent-scm.com) offers a Gerrit and code repository online service.

RBCommons (https://rbcommons.com) is the SaaS version of the review board's (http://www.reviewboard.org/) open source code review software.

Repository management sites such as GitHub (https://github.com) or BitBucket (https://bitbucket.com) offer, via the web interface, an easy way to fork a repository, make local changes and push them back to the upstream repository with pull requests that can be discussed and approved by the upstream repository owner.

Rspec-puppet (http://rspec-puppet.com) makes it possible to test Puppet manifests using rspec, a widely used Ruby behavior-driven development framework.

It can be installed as gem:

gem install rspec-puppet

Once installed, we can move into a directory of a module and execute:

rspec-puppet-init

This command creates a basic rspec environment composed by:

Test files have names that reflect what they are testing, for example, if our module is called apache, the tests for the apache class could be placed in spec/classes/apache_spec.rb and the tests of a define called apache::vhost are in spec/defines/apache_vhost_spec.rb.

In these files, we have the normal Ruby spec code, where the test condition and the expected result is defined, in terms of Puppet resources and their properties.

This is a sample test file for an apache class (taken from Puppet Labs' apache module).

The first line requires the spec_helper.rb file previously described as:

require 'spec_helper'

Then, a Debian context is defined for which the relevant facts are provided:

describe 'apache', :type => :class do
  context "on a Debian OS" do
    let :facts do
      let :facts do
      {
        :id                     => 'root',
        :kernel                 => 'Linux',
        :lsbdistcodename        => 'squeeze',
        :osfamily               => 'Debian',
        :operatingsystem        => 'Debian',
        :operatingsystemrelease => '8',
        :path                   => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
        :concat_basedir         => '/dne',
        :is_pe                  => false,
      }
    end

Under this context (defined according to the specified facts or parameters) are then listed the resources that we expect in a catalog; note that it is possible to use the is_expected.tocontain_* matcher for any Puppet resource type, and each argument of that resource can be tested:

    it { is_expected.tocontain_package("httpd").with(
      'notify' => 'Class[Apache::Service]',
      'ensure' => 'installed'
      )
    }
    it { should contain_user("www-data") }
  }

We can also set specific parameters and test the behavior we expect when they are provided:

    describe "Don't create user resource" do
      context "when parameter manage_user is false" do
        let :params do
          { :manage_user => false }
        end

        it { is_expected.not_to contain_user('www-data') }
        it { is_expected.to contain_file("/etc/apache2/apache2.conf").with_content %r{^User www-data\n} }
      end
    end

Note the with_content matcher that permits to verify the same contents of a file.

Rspec-puppet's intended usage is to verify that the logic of our module is correctly applied under different conditions, for example, testing its behavior when different parameters are provided or when different OSes are simulated. It is useful to prevent regressions during code refactoring and to quickly validate pull requests from contributors.

It is not intended to validate the effect of our code when applied on a system.

This is a key point to understand: rspec-puppet works on the catalog and checks whether Puppet resources are correctly provided. It does not test what happens when those resources are actually applied for real: in software testing terms, rspec-puppet is a unit-testing tool for modules.

Vagrant (http://www.vagrantup.com) is a very popular tool created by Mitchell Hashimoto to quickly create Virtual Machines (VM) mostly for testing and developing purposes.