Let's install Vagrant. Firstly, we will need our virtualization backend, so let's download and install VirtualBox. At the time of writing, we use VirtualBox 5.0.10 r104061. If that's outdated by the time you read this book, just grab the latest version.

You can download VirtualBox from https://www.virtualbox.org/wiki/Downloads. Choose the version for your OS, as shown in the following screenshot:

Once the package is downloaded, follow the given installation process for your OS.

VirtualBox

Follow these steps to install Vagrant on Mac OSX:

1. Go to your Downloads folder and double-click on VirtualBox.xxx.xxx.dmg. The following installation box will pop up:

   

2. Then, click on VirtualBox.pkg. Move on to the next step, as shown in the following screenshot:

   

   The installer will then check whether the software is compatible with the Mac OSX version.

3. After this, click on Continue. Once the check is successful, we can move on to the next step:

   

4. We then choose the default location for the installation and click on Install.

5. Then, enter your admin password and click on Install Software:

   

The installation is now complete. The following screenshot shows what the screen looks like after completing the installation:

Now that we have the virtualization backend, we can install Vagrant:

Note

At the time of writing this book, we are going to use Vagrant 1.7.4; if that is no longer the latest version, please grab the latest one. You can find this version of Vagrant at https://www.vagrantup.com/downloads.html. Again, download the installation package for your OS.

Now that we have a fully working Vagrant environment, we can start with and look at how Vagrant works and how we are going to provision our machines. As this book is not about Vagrant, we won't be writing a Vagrantfile from scratch. Instead, I have created a Vagrantfile that we will be using throughout the book:

Let's look at the Vagrantfile construct:

As you can see from the preceding screenshot, the Vagrantfile is actually a Ruby file. As it is Ruby, it opens up a world of opportunities for us to make our code elegant and efficient. So, in this Vagrantfile, we have extracted all the low-level configurations and replaced them with a few parameters. Why are we doing this? The reason is to split up our logic from our configuration and also iterate our configuration in order to stop replication of our code. So, where is all the configuration stored? The answer is in the servers.yaml file. This is where we set the vagrant box that we want to deploy, the number of CPUs for the box, the internal network's IP, the hostname, the forwarded ports between the guest and host, and the RAM and shell provider for bash commands that we need to get the environment ready for Puppet to run, for example, downloading modules and their dependencies from the Puppet Forge:

The benefit of this approach is also that any developer using a Vagrantfile does not need to actually modify the logic in the Vagrantfile. They only need to update the configuration in servers.yaml. As we go through the book, we will work with the other files in the repository, such as Puppetfile, hieradata, and manifests. Now that we have set up our Vagrant environment, let's look at how to get our Puppet modules from the Puppet Forge.

One of the great things about puppetlabs and their products is the community. If you ever get a chance to attend PuppetConf or a Puppet Camp, depending on where you live, I would really recommend you to attend it. There will be a wealth of knowledge there and you will meet some really great people.

The Puppet Forge is a website that puppetlabs runs. It is a place where other Puppet developers publish modules that are ready to use. You might be asking, what about GitHub? Can't you get modules from there? Yes, you can. The difference between the Puppet Forge and GitHub is that the Puppet Forge is the stable, releasable version of the module, whereas GitHub is the place to contribute to the module, that is, a place to create pull requests.

The following screenshot shows the home page of Puppet Forge:

Now that we have been introduced to the Puppet Forge, let's use it to find our Docker module that we will be using to build our environment.

Now that we have selected our module, we can move on to setting up our puppetfile:

In the previous topic, we cloned our Vagrant template using Git. In that repo, there is also a puppetfile. A puppetfile is used as a control file for our modules. It will list all the modules that we need (in this instance, just to install Docker). r10k will then reference the puppetfile and pull the modules from the Puppet Forge into our environment's directory.

As modules have dependencies, we need to make sure that we capture them in our puppetfile. For the Docker module, we have three dependencies: puppetlabs/stdlib (>= 4.1.0), puppetlabs/apt (>= 1.8.0 <= 3.0.0), and stahnma/epel (>= 0.0.6), as shown in the following screenshot.

Now, we know all the modules that we need to build a Docker environment. We just need to add them to our puppetfile.

The following screenshot is an example of what the puppetfile should look like:

Now, when we run vagrant up, r10k will pull the modules from the Puppet Forge. We invoke r10k on line 13 of servers.yaml with the r10k puppetfile install—verbose command. The following screenshot shows the output of this command:

If we are successful, the terminal will provide the following output:

Now that we have our puppetfile set up, we can install Docker.

The first thing that we need to do to install Docker is set our manifest to include the Docker class on our node. To do this, let's go to our Vagrant repo. In the repo, there is a file in the manifests directory called default.pp. We need to edit the file to include the Docker class node 'node-01' { include docker}. We can now save the file, and we are ready to run our environment.

The first step is to open our terminal and change to root of the Vagrant repo. Then, we need to enter the vagrant up command:

This will now provide us with our CentOS 7 box. Install r10k and then run Puppet and apply the Docker class. This will take about 4 minutes depending on your laptop and network connection. If the box was provisioned successfully, you will see the following output:

We can also verify that the Docker installation was successful by logging in to the box via SSH. We will do that with the vagrant ssh command. Once we are in, we will sudo up to root (sudo -i). Now, let's just check whether Docker is installed with the docker command.

You will see the following output on the terminal:

In the last chapter, we looked at the Puppet's repo service, The Forge as it is called by the community (https://forge.puppetlabs.com/). Now, let's look at Docker's repo service, Docker Hub. We can find Docker Hub at https://hub.docker.com/.

The following screenshot shows what the screen looks like:

In Docker Hub, there are two type of images:

First, we will talk about official images. On Docker Hub, you can get official images for just about any major operating system or application. So, the benefit for you as a developer is that the work to install the application is done for you, saving you the time and effort. This allows you to focus your time on developing. Let's look at an example—we will use golang.

First, we will search for golang in the search box at the top right-hand side of the front page, as shown in the following screenshot:

Our search will return the following results:

We will click on the official release of golang, as shown in the following screenshot:

As we can see in the preceding screenshot, this repository gives us a lot of options. So, we can use multiple, different versions of golang, even on multiple different operating systems. So, to build a golang app, all we need to do is choose an image. We will use the following image in our Dockerfile:

We will then use the COPY method in our Dockerfile to get our code into the container on build. Lastly, we will run the command shown in the following screenshot to build our container:

So, as you can see, it was very easy to build our app, where almost all of our development time would be spent on the actual application. This will increase our productivity and bring the applications to production a lot faster. In this day and age, where agility is everything, you would have to be mad to not see the benefit.

The second type of image on Docker Hub is developed and open sourced by developers and is maintained by them individually. The easiest way to tell whether an image is official or has been developed by an individual is through the image's name. In our last example, we looked at the golang image. The name of that image is golang. Now, let's look at a container that I have open sourced. For this example, we will look at my consul image. If you want to use my image, you would call it scottyc/consul. As you can see, the name is different, as it calls the author name scottyc and then the image name, consul. Now, you can see the difference in the naming convention between an official image and an authored image.

Now that we have covered the different images hosted at Docker Hub, we can move on to how images get to Docker Hub. There are two different ways to get images to Docker Hub. Both ways, we need a Docker Hub account, which we will cover in the next section.

The first way is to build the image locally and simply use the docker push command. The second way is using automated builds, which is an awesome functionality that Docker has built into Docker Hub. We will cover this later in much more detail. At a high level, it is a CD (continuous delivery) process to build the image based on a Dockerfile that is stored in a GitHub or Bitbucket public repository.

In this topic, we will create a Docker Hub account and look at how to log in to the Docker daemon manually (we will cover how to do this with Puppet in the next chapter). So, let's start. First, we will need to go to Docker Hub (https://hub.docker.com/) and fill out the form on the right-hand side of the page. Just replace yourusername with your desired username, you@youremail.com, with your e-mail ID, and, of course, enter a secure password:

After that, go to your e-mail ID and confirm your account. This will then redirect you to the Docker login page. Log in and you should see the following web page:

Now that we have an account, let's log in to our daemon. So, let's use vagrant ssh back into our Docker vagrant box. We will change to root (sudo –i) and then enter the docker login command:

Enter the username that we just created:

Then, enter your password:

After this, enter your e-mail ID. Once this is done, you should see the following output:

You have now successfully logged in the Docker daemon.

In this topic, we are going to provide a quick overview of how to search for images on Docker Hub. There are two ways to do this:

Let's look at the website first. If you remember, in our golang example we already used the web interface to search for an image. Let's look at another example. In this example, we will look for bitbucket, Atlassian's git server. So, we will go back to Docker Hub (https://hub.docker.com/) and enter bitbucket in the search field:

Our search will return the following output:

As you can see from the preceding screenshot, we got 43 results. So what should we look for to choose the right image? We always look for three things, which are as follows:

Using the three metrics, let's pick an image. Looking at the results, atlassian/bitbucket-server looks good, with 21 stars and 7.3k pulls. So, let's click on the repo and look for a Dockerfile:

If we click on the Dockerfile tab under the main image title, it takes us to the Dockerfile page. Not every repository has a Dockerfile; however, this does not mean that it's a bad image. This just means that it will take more testing before you would be able use it in production. Some authors, such as Jess (Jessie Frazelle) from Docker, have their Dockerfiles on their GitHub page. She has awesome images on Docker Hub and the Dockerfiles can be found at https://github.com/jfrazelle/dockerfiles. Alright, back to our example. As you can see in the following screenshot, there is a Dockerfile:

So, I think this is the winner!!!!

Now, let's do the same search from the command line. In the command line, type docker search bitbucket, and the search will return the following output:

As you can see, it has returned the same information, and the only thing missing is the number of pulls. Again, it looks like we will use atlassian/bitbucket-server.

In Docker Hub, we have two ways to publish images: via a simple push method or via an automated build. In this topic, we will cover automated builds. First, we will look at the flow of an automated build. In this example, we will be using GitHub, but you can also use Bitbucket. So, the first thing that we need to do is link our Docker Hub account to our GitHub account. This is done by navigating to Settings | Linked Accounts & Services:

Just follow the prompts to link the accounts.

Once this is completed, let's go to our GitHub account and create a repo. I am going to use the one that I have already set up:

As you can see in the preceding screenshot, the repo contains a Dockerfile. Now, let's looks at the same repo except Docker Hub:

After this, we will look at the Build Details tab:

So, how does that build automate ? Well, it is quite simple. Every time we check in a change to the GitHub repo, it will trigger web hooks at Docker Hub. When Docker Hub receives the trigger, it will grab the Dockerfile and build the image. Docker Hub will take care of things such as version numbers for us with every build. So, at a high level, this is how automated builds work.

This is a quite simple way to get an image to Docker Hub, but the downside is that there is no automated build process and the Dockerfile does not get placed in the Docker Hub repo automatically. So, in this example, we will assume that we have created an image called scottyc/super_app. To push this to Docker Hub, we simply type docker push scottyc/super_app in the terminal. Note that the Docker daemon needs to be logged in at the time of the push.

In this topic, we will look at using nginx in a basic Dockerfile. In a Dockerfile, we need to add a few things. The first is the image that we are basing our application on; for us it will be nginx. The second is a maintainer. It should look like as shown in the following screenshot:

As the base nginx image has already got port 80 and 443 exposed, we will not need that configuration for our Dockerfile. The next thing we will add is a simple run command to update the packages in the container. As its base OS is Debian, we will add the command shown on line 5 in the following screenshot:

As we are building a simple application, this is all that we are going to add to our Dockerfile. There are heaps of configurations that can be done with a Dockerfile.

Now, let's build our image. You will note that server.yaml in our Vagrant repo already has port 80 forwarding to port 8080, so we won't need to make any changes there. Copy the Dockerfile that we created into the root of your Vagrant repo. Then, let's start our vagrant box with vagrant up from our terminal. Then, use vagrant ssh once the box is up. Let's change to root (sudo -i). Then, if we change directories to /vagrant, we should see our Dockerfile. Now, let's build our image with the command, docker build -t YOUR AUTHOR NAME/nginx . (note that . is part of the command). You will get the following output on your terminal:

Next, let's test our image and spin up a container with the following command:

docker run -d -p 80:80 --name=nginx YOUR AUTHOR NAME/nginx

If it was successful, we should get the nginx default page in your browser at 127.0.0.1:8080, as follows:

Now, we are going to deploy the same nginx image with Docker Compose. We are going to run Docker Compose at a high level in this topic just to get an understanding of the technology. We will look at it in depth in another chapter of this book. The first thing we need to do is install Docker Compose.

To do this, let's modify our puppetfile in the Vagrant repo with the commands shown in the following screenshot:

So, in the Puppetfile we added a new module dependency, stankevich/python, as Docker Compose is written in Python. We also updated our epel module to use the latest. Just to get a fresh working environment, we will run the command, vagrant destroy && vagrant up, in our terminal. Once the box is up, we will use vagrant ssh and then change to root (sudo -i). We will then change the directory to /vagrant and type docker-compose.

If the build was successful, we will see the following screen:

Now, let's create docker-compose.yaml:

As you can see, we used the official image, gave the container a name nginx, and exposed ports 80:80 again to be able to hit the nginx page. So, if we copy our docker-compose.yml file to the root of the Vagrant directory, log in to our vagrant box, and change the directory to root (vagrant ssh, then sudo -i), we will be able to change the directory to /vagrant again. Now, run docker-compose up -d. We will get the following output after running it:

We can then go to our web browser and visit the nginx page at 127.0.0.1:8080:

In this section, we are going to build the same ngnix container with a simple Puppet manifest. This is just a proof of concept. In the next chapter, we will write a full module. This is just to give us a foundation and understanding of how Puppet interacts with Docker. So, in our Vagrant repo, let's modify manifest/default.pp. The file should contain the following code:

We will then open our terminal at the root of our Vagrant repo and run vagrant provision. Note that you should have no other containers running at this time. You will see the following output, which shows that Puppet provisioned a Docker container called nginx:

We can then check our browser again at 127.0.0.1:8080. We will get the nginx page again:

One of the best things about working with Puppet is the number of tools out there, both from the community and from puppetlabs itself. The Puppet module generator is a tool that is developed by puppetlabs and follows the best practices to create a module skeleton. The best thing about this tool is that it is bundled with every Puppet agent install. So, we don't need to install any extra software. Let's log in to our Vagrant box that we built in the last chapter. Let's change directory to the root of our Vagrant repo and then use the vagrant up && vagrant ssh command. Now that we are logged in to the box, let's sudo to root (sudo -i) and change the directory to /vagrant. The reason for this is that this folder will be mapped to our local box. Then, we can use our favorite text editor later in the chapter. Once we're in /vagrant, we can run the command to build our Puppet module skeleton. The puppet module generate <AUTHOR>-consul command for me will look like this: puppet module generate scottyc-consul.

The script will then ask a few questions such as the version, author name, description, where the source lives, and so on. These are very important questions that are to be considered when you want to publish a module to the Puppet Forge (https://forge.puppetlabs.com/), but for now, let's just answer the questions as per the following example:

Now that we have our Puppet module skeleton, we should look at what the structure looks like:

Now, we are going to add a few files to help us with unit tests. The first file is .fixtures.yml. This file is used by spec-puppet to pull down any module dependencies into the spec/fixtures directory when we run our unit tests. For this module, the .fixtures.yml file should look like the one shown in the following screenshot:

The next file that we are going to add is a .rspec file. This is the file that rspec-puppet uses when it requires spec_helper, and it sets the pattern for our unit test folder structure. The file contents should look as shown in this screenshot:

Now that we have our folder structure, let's install the gems that we need to run our unit tests. My personal preference is to install the gems on the vagrant box; if you want to use your local machine, that's fine as well. So, let's log in to our vagrant box (cd into the root of our Vagrant repo, use the vagrant ssh command, and then change the directory to root using sudo -i). First, we will install Ruby with yum install -y ruby. Once that is complete, let's cd into /vagrant/<your modules folder> and then run gem install bundler && bundle install. You should get the following output:

As you can see from the preceding screenshot, we got some warnings. This is because we ran gem install as the root. We would not do that on a production system, but as this is our development box, it won't pose an issue. Now that we have all the gems that we need for our unit tests, let's add some basic facts to /spec/classes/init_spec.rb. The facts we are going to add are osfamily and operatingsystemrelease. So, the file will look as shown in this screenshot:

The last file that we will edit is the metadata.json file in the root of the repo. This file defines our module dependencies. For this module, we have one dependency, docker, so we need to add that at the bottom of the metadata.json file, as shown in the following screenshot:

The last thing we need to do is put everything in its place inside our Vagrant repo. We do that by creating a folder called modules in the root of our Vagrant repo. Then, we issue the mv <AUTHOR>-consul/ modules/consul command. Note that we removed the author name because we need the module to replicate what it would look like on a Puppet master. Now that we have our basic module skeleton ready, we can start with some coding.

Let's create two new files, install.pp and params.pp, in the manifests folder. The structure should look as shown in the following screenshot:

Let's start writing our module. We will start with init.pp; this module is not going to be very complex as we are only going to add a few lines of code and some parameters. As you can see in the preceding screenshot, we created three files in the manifests directory. When I write a module, I always like to start at params.pp, as it gives me a good starting structure to work for the code that provides the module logic. So, let's look at params.pp for this module, which is shown in the following screenshot:

Now, let's look at the parameters that we have set:

Now that we have set up our parameters, let's get started on install.pp. As the name implies, this class will contain the logic that installs Docker, pulls the image, and runs the container. So, let's take a look at the code shown in the following screenshot:

To look at the code in more depth, we will break the class into two, the Docker installation and the container configuration. In the Docker installation, the first piece of code is a simple package type for device-mapper-libs. The reason we make sure that this package and its dependencies are installed is that it will be the storage drive that Docker will use to mount the container's filesystem.

Now, we move on to the Docker install. We start by declaring the docker class. For this class, we will set the Docker version, calling the parameters we set in params.pp, and the version of Docker that we are using is 1.9.1 (which is the latest at the time of writing this book). The next piece of configurations we will declare are the Docker API's TCP bind. Again, we will call our params.pp class and set the value to tcp://127.0.0.1:4242. This binds the API to listen to the localhost address on the TCP port 4242.

The last value we will set to our Docker install is the Unix socket, which Docker will use. We will declare this without calling a parameter. The last piece of code makes sure that device-mapper-libs is installed before Docker, as it is a prerequisite to the Docker install:

Now that we have Docker installed, let's look at the code to build our Consul container. The first class that we call is docker::image. This will pull the image from Docker Hub before we call the docker::run class. In the docker::run class, we set navmar as the same value as the container's hostname. We will get that value from params.pp and it will be set to consul.

The next configuration we will set is the image. Now, this is different from calling docker::image. When you call docker::image, it pulls the image from Docker Hub to the local filesystem. When we set the image value in the docker::run class, it sets the value for the base image where the container will be deployed from. The value is set to scottyc/consul, and again we will get that value from params.pp. The hostname parameter is going to set the hostname inside the container.

Now we get to the the resource attribute that passes running configurations' parameters to the container. The command attribute is an arbitrary attribute that allows you to pass configurations to the container at boot. In this case, we are going to pass the boot configuration for the Consul setting as the server role, the IP address that the Consul application will bind to, and the number of servers that are there in the Consul's cluster. In the first case, all the values in the arguments that we are passing to the command attribute come from params.pp:

Now, last but definitely not least, let's look at what our init.pp file contains. The first thing that you will note at the top after the main class declaration is the mapping of all our parameters to params.pp. The reason we do this is to set any sensible configurations or defaults in params.pp and any sensitive data we can overwrite the defaults with Hiera lookups. We will look at Hiera lookups in the next chapter. The last line of code includes our consul::install class, which we covered in the preceding section:

Now, let's run our module.

Now that we have written our module, I am sure we are all keen to run it; however, before we can do that, there is one more piece of configuration that we need to add to servers.yml and default.pp. First, we need to make sure that our module consul is located in modules/consul. The next step is to open our servers.yml file and add the following line at the bottom of the shell commands:

- { shell: cp /home/vagrant/node-01/modules/* -R /tmp/modules }

This will copy our module into the correct module path in the vagrant box. We also need to forward the Consul port so that we can hit the GUI. This is done by adding - { guest: 8500, host: 8500 } to the forwarded port's attribute. It should look as shown in the following screenshot:

Now, let's open our manifests directory and edit default.pp. We just need to add our module to the node definition. You can do this by adding the include consul configuration, as shown in the following screenshot, and saving both files:

Let's head to our terminal, change the directory to the root of our Vagrant repo, and type the vagrant up command. Now, if this box is already running, issue a vagrant destroy -f && vagrant up command. The output should look as shown in this screenshot:

Even though we a have successful run Puppet, it can take a couple of minutes for the container to come up the first time, as it downloads the image from Docker Hub, so just be patient. You can easily check when the container is up by going to your browser and navigating to 127.0.0.1:8500. You should get the consul GUI, as shown in the following screenshot:

As you can see, we have one node that is running, named consul, which is the hostname that we gave to our container.

In the chapter, we are going to look at how to use docker-compose as an .erb template file. In this example, we are only deploying a single container, but when an application contains five or six containers with links, this way is much more efficient than using the standard manifest declarations.

So, we will take our consul module from the last topic and modify it now to use docker-compose. If you want to keep that module, just make a copy. First, we are not going to touch init.pp and params.pp—they will remain the same. Now, let's look at install.pp:

As you can see in the preceding screenshot, the top half of the class is exactly the same. However, we still install device-mapper-libs and declare the docker class in exactly the same way. The next attribute is different though; here, we call the file resource type. The reason is that this is the configuration that is used to place our docker-compose file on the local filesystem. You can see that we are declaring the contents with a template file located in the templates directory of the module. We will come back to this in a minute.

Now, let's look at the last resource type in install.pp. We are calling the docker_compose type because this is the resource type that will run the docker-compose command to bring up our container. Let's look at the attributes that we have configured. The first is navmar; this will set the name tag in Docker for the container. We are calling this value from params.pp, and it will be set to consul. ensure is a puppet meta parameter container which ensures that the container is always there.

If we want to delete the container, we would have to set this value to absent. The next attribute is source; this sets the folder where the docker-compose command can find the docker-compose file. We have set this to root. You could change the value to any folder on your system. The last attribute is scale. This tells docker-compose how many containers we want. The value is set to 1 in this instance. If we were deploying an nginx web farm, we might set the value to a figure such as 5. Now, let's get back to that template file. The first thing we need to do is create a folder called templates in the root of our consul module:

The next step after that is to create our .erb template file. In install.pp, we declared the filename as docker-compose.yml.erb, so in our templates directory, let's create a file with that name. The contents of the file should look as shown in the following screenshot:

So, the first thing that you should note in the preceding screenshot are the variables that are set, such as <%= @container_hostname %>. This maps back to init.pp as $container_hostname. As you can see, attributes such as image, hostname, ports, and command look very familiar. This is because they are the same attributes that we declared in the preceding section. In this example, we only configured our docker-compose file for a single container; in the next topic, we will look at a much more complex configuration. Before we get to that, let's make sure that this module runs.

To run our module, let's make sure that our module is located in the modules/consul directory in the root of your Vagrant repo. We have the configuration to forward port 8500 (forwarded_ports: - { guest: 8500, host: 8500 }) and copy our module to our module's path directory using - { shell: cp /home/vagrant/node-01/modules/* -R /tmp/modules }.

Once this is in place, let's run vagrant up in our terminal in the root of our Vagrant repo. Again, if you have a box that is running, issue the vagrant destroy -f && vagrant up command. The terminal should give you the following output:

Again, we can go to 127.0.0.1:8500 and get the Consul GUI:

Now, let's log in to our vagrant box; we will do that by issuing the vagrant ssh command in our terminal from the root of our Vagrant repo. Once we have logged in, we can su to root (sudo -i). Then, we can issue the docker ps command to look at all the running containers. The terminal should give you the following output:

As you can see, the container is up and running.

So, we briefly touched on the topic of state versus stateless in the preceding section, but now, let's get into the nuts and bolts of it. So, let's look at two example applications: one with state and one without. First, we will start with a redis container. Its job is to be the backend store for activemq. The application that uses the queue has logic to check whether the message has been received and expects a response. If that fails, it will retry to send the message. Thus, the queue itself is stateless. We are using redis as a cache. There is no need to give the redis container state, as the information it holds is ephemeral. If the container fails, we just respawn a new one and it will cache the queue again.

Now, let's look at a container that needs state and the options we have to keep state. In the application, we are going to build two containers. One is Bitbucket itself, and then Postgres will be used as the backend database. So, both of these need state. Let's look at Bitbucket first. If we don't give states to the Bitbucket server, every time we restart the container, we would lose all the projects that we checked in via Git. That doesn't sound like it would be a viable solution. Now, let's look at the options that we have to give to the container's state. First, we could add volume to our Dockerfile:

This will give the container state; we can reboot the application and all the data will be there, which is good. That is what we want. There is a downside to this method. Volume resides inside the container and is not decoupled from the application. So when we run into issues with this method, the main operational issue is when we need to update the version of the application in the container. As all our data lives in the container, we can't just pull the latest. So we are stuck with the available options. Now, let's look at how to map a folder from localhost to the container. We do this using /data:/var/atlassian/application-data/bitbucket. The left side of the colon is the localhost running the Docker daemon and the right side is the container.

Docker uses fully qualified paths, so it will create the /data directory. Now, we have decoupled our application from our data. If we want to update the version of Bitbucket, all we would need to do is change the image tag in our Puppet code to the new version. Then, we would run Puppet. Once that is complete, the new version of Bitbucket will boot with our existing data. Now, there is a cost to using this method as well. We have now tied this container to the one host. If you are using schedulers such as Kubernetes or Docker Swarm, this probably isn't the best idea. This problem has been solved by the new volume driver added to the Docker engine 1.8 and above. This allows us to create storage objects that are external of the host on which the engine is running.

This is just going to be a quick refresher, as we covered this in the last chapter. If you are still not feeling comfortable with this step, I would recommend you to go over the last chapter again until you have a good grasp of the process.

So, we are going to open up our terminal and change directory or cd into the root of our Vagrant repo. Then, we will type vagrant up, and once the box is up, we will use SSH into it with vagrant ssh. The next step is to change to root (sudo -i). Now that we are root, let's change the directory to /vagrant, which maps back to our local machine. We are then going to issue the puppet module generate <AUTHOR>-docker_bitbucket command. Again, there are a few more tweaks that we need, but they are in the last chapter, so let's not repeat ourselves here. Once you have completed the outstanding task, you can move on to the next chapter.

Now, we have our module skeleton and we have moved it into the modules directory in the root of our Vagrant environment. We need to add two new files: install.pp and params.pp. Our module should look like as shown in the following screenshot:

In this example, we have a few new things going on, so I have not used params.pp in this example. This gives you a perfect opportunity to use the knowledge that you gained in the last chapter and apply it. So, for now, we will leave params.pp empty. Seeing as we are not putting parameters in init.pp, let's look at that first:

As you can see in the preceding screenshot, we are only calling the docker_bitbucket::install class. We can now move on to the chunky install.pp class. Again, we are going to brake this down into three parts. This will make it easier to explain the logic of the class. Let's look at part one, which is as follows:

In this top section of the class, we are installing the device-mapper-libs package. This is a prerequisite for the RHEL family and Docker. The next thing that we are declaring is the Docker class. In this resource, we are defining the version of Docker that we want to be installed, the TCP bind that Docker will use, and lastly, the Unix socket that Docker will bind to. This is the same configuration that we used to define our Docker daemon in the last chapter. This will stay fairly static until we move into Docker schedulers. Let's now move to Postgres:

First, we will define the image of Postgres that we would like to use. For this example, we are using Postgres 9.2. So, the correct tag from Docker Hub is postgres:9.2. Now, let's look at the docker::run class; this is where all the configurations for Postgres will be defined. So, you can see that we are calling the image that we set in the preceding resource postgres:9.2. We will then set the hostname as bitbucket-db. This setting is important, so let's store it into our memory for later use.

Let's look at the env resource declaration as we have a bit going on there. In this one line, we are declaring the Postgres user, the database password, the name of the database that we will connect with Bitbucket, and lastly the path where Postgres with store the database. Lastly, we are declaring our volumes as /root/db:/var/lib/postgresql/data/pgdata.

As mentioned earlier, the left-hand side of the colon is mapping the the local machine and the right is mapping the container. There are two major call outs with our configuration. First, the /root/db folder is arbitrary and not what you would use in production. The second is that you will note that the left side of the colon, /var/lib/postgresql/data/pgdata, and the value in env, PGDATA, are the same. This is no coincidence; that folder holds the only state that we care about: the actual database. That is the only state that we will keep. Nothing more and nothing less. You will note that we have not exposed any ports from this container. This is by design. We are going to link our Bitbucket container to Postgres. What does link mean? It means that by default, the Postgres image exposes port 5432. This is the port that we will use to connect to our database. By linking the containers, only the Bitbucket container has access to 5432; if we exposed the port (5432:5432), any other application that has access to the host instance could hit the port. Thus, linking is much more secure. So, we need to remember a few things from this section of code for later use: the hostname and the entire env line. For now, let's move on to the Bitbucket container:

As you can see in the preceding screenshot, the image resources are the same, but instead of calling Postgres, we are going to call atlassian/bitbucket-server. The next resource we will declare is the ports resource. You will note that we are declaring two ports 7990:7990, which will be the port that we hit the web UI on, and 7999:7999, which is the port that Bitbucket uses for SSH. We will set the username to root. This is recommended in Atlassian's documentation (https://hub.docker.com/r/atlassian/bitbucket-server/).

Next, we are going to map our volume drive. In this case, we are only going to map Bitbucket's data directory. This is where all our Git repo, user information, and so on is kept. Again, /data is an arbitrary location; you could use any location you like. The important location to note is on the left-hand side of the colon, /var/atlassian/application-data/bitbucket.

Lastly, we link our two containers. Another benefit of linking containers is that the Docker daemon will write to both the containers' /etc/hosts files with their hostname and IP address. So, the containers have no issue talking to each other. There is no need to worry about the IP address, as it is arbitrary and is looked after by the Docker daemon. Now that we have written our module, we can build our application.

The first thing that we need to do is forward the correct ports on our servers.yml file that allow us to hit the ports we forwarded to Bitbucket. To do this, we need to modify the file so that it looks as shown in the following screenshot:

So, let's open our terminal and change the directory to the root of our Vagrant repo and run vagrant up. You should get the following output after this:

Now that our application is built, we can go to http://127.0.0.1:7990. We should get the following page:

Earlier in the topic, we had remembered some details about our Postgres install. Now is the time to use them, so let's begin. The first thing that we need to do is use an external database. The next piece of configuration we need to choose is the database type. Of course, we are going to choose Postgres.

The hostname will be set to the hostname of the bitbucket-db container, the port is 5432, and the database name is bitbucket, as we set in our code. We will use PostgreSQL as the username and the password will be Gr33nTe@. Refer to the following screenshot to know more:

Next, click on the Test button, and we should get the Successfully established database connection. message, as shown in the following screenshot:

I will let you finish the rest of the setup. But what we just set up was not simple, and now, we have a very solid base to move on to more complex applications.

We covered a lot of what happens under the hood in the last topic. We will not be repeating ourselves, so this topic will be just about the code. We are going to keep both init.pp and params.pp the same as we did in the last topic. So, let's jump straight to install.pp. It will look very similar to install.pp from the last chapter:

All the magic happens in our template file. So, let's jump to our .erb file that lives in the templates folder in the root of our module:

As you can see in our .erb file in the preceding screenshot, all the configurations are familiar. There are absolutely no changes to what we covered in our last topic.

Let's open our terminal and change the directory to the root of our Vagrant repo and run vagrant up. You should get the following output:

Now, let's just go to http://127.0.0.1:7990, and we should get the following page:

Just follow the same setup as in the preceding topic to configure Bitbucket. You can use a trail license to try the application, or as I mentioned earlier, there is a development/startup license at https://bitbucket.org/product/pricing?tab=server-pricing with the proceeds of the $10 license going to charity.

Service discovery is essential when we start to work with multinode applications, as it allows our applications to talk to each other as they move from node to node. So, as you can see in the world of containers, this is fairly important. We have a few choices when we choose a service discovery backend. The two big names in this space are etcd (https://coreos.com/etcd/), which again is from CoreOS, and Consul from HashiCorp (https://www.consul.io/).

You might remember that we have already written a consul module. So for this chapter, we are going to choose the same, as we already have the written code. First, let's look at the architecture of Consul so we can understand how the backend works, how it handles failure, and what option do we get with our configuration of Consul.

So, let's talk about how Consul works. In Consul, we have two types of configuration that we can give to a server. First is a server role and the second is an agent role. Although the two interact, they serve different purposes. Lets dive into the server role first. The server's role is to participate in the RAFT quorum; this is to maintain the state of the cluster. In Consul, we have the idea of data centers. What is a data centre, you may ask? It is a group of logical servers and agents. For example, if you are an AWS, a data center could be an AZ or even a VPC. Consul allows connectivity between data centers; it is the role of the sever to look after the communications between data centers. Consul uses the gossip protocol to achieve this. The server also holds the key/value store and replicates it between the servers using the serf protocol. Let's look at a diagram of what we discussed:

The agent's role is to report to the server about the state of the machine and any health checks that may be assigned to it. Again, Consul will use the serf protocol to pass the communication.

Now that we have an understanding of what Consul is doing behind the scenes, let's look at the features that it has that we can take advantage of in our Puppet modules. The first feature we will take advantage of is DNS service discovery. In the container world, this is pretty important. As our containers move from node to node, we need to know how to connect to them. DNS service discovery solves this very neatly. So, let's look at an example to understand this.

In this example, we have a mario service and we have Docker swarm cluster of three nodes. When we hit the Docker API and swarm schedules the container, we don't know which of the three machines mario will end up on. But we have other services that will need to find mario as soon as he is up. If we tell the other services that mario is actually at mario.service.consul, no matter what node the container comes up on, it will resolve mario.service.consul to the right address. Refer to the following diagram to understand this in detail:

In this case, if we were to ping mario.service.consul, we would get 192.168.100.11. Pending our scheduling configuration in swarm, if Server b fails, mario.service.consul could end up on Server d. So, the response to mario.service.consul would now come from 192.168.100.13. This would take no human intervention and would be seamless to the applications. That is all the theory we will see for service discovery in this chapter; there is more that we will cover in the later chapters. Now, let's get to writing some code.

In this module, we are going to write a module that uses consul as our DNS service discovery backend. As we already have a consul module, we won't start from scratch but add new features to the existing module. We will again write the module with manifests and Docker Compose. So, let's start with the manifests.

Our folder structure should look like this:

Let's jump straight to install.pp. Without making any changes, it should look as shown in the following screenshot:

Now, we are going to add one extra container that is going to be part of the plumbing for our DNS service discovery solution. We will need something to register our containers with Consul as they spawn. For this, we will use a golang application called registrator (https://github.com/gliderlabs/registrator). This is a fantastic app. I have been using it for over a year, and it has been faultless. So, let's make changes to our params.pp file to allow the new container. At the moment, params.pp looks like the one shown in the following screenshot:

The first thing that we will do is make changes to the docker_image and container_hostname parameters. As we already have the convention of consul_xxx, we can carry on with that:

Now, let's add the parameters for registrator:

As you can see, we have added the parameter for the image as $reg_docker_image = 'gliderlabs/registrator' and the parameter for the hostname as $reg_container_hostname = 'registrator'. We have told the container to listen to the host's $reg_net = 'host' network. The next parameter will need some explaining. The registrator maps the Unix socket that the Docker daemon is bound to into its Unix socket. It does this to listen to any new services that get spawned and need to be registered in consul for discovery. As you can see, we do this with $reg_volume = ['/var/run/docker.sock:/tmp/docker.sock']. The last parameter tells registrator where to find consul. We are going to set that with $reg_command = "consul://$::ipaddress_enp0s8:8500". Now, let's move over to our init.pp file.

Our init.pp file should look as shown in the following screenshot:

Let's add our new parameters, as shown in the following screenshot:

Now that we have all our parameters set up, we can go to our install.pp file to add our code in order to install registrator:

As you can see in the preceding screenshot, we have added a new block of code at the bottom of our file. It's similar to our code that configures Consul; however, there are a few different parameters. We covered those earlier, so let's not repeat ourselves. Now that we've made a fair chunk of changes to our module, we should run it in Vagrant to check whether we have any issues. Before we can run Vagrant, we need to change our servers.yaml file in the root of our Vagrant repo so that it allows us to hit the Consul URL on port 8500. We do this with the following change to the code:

Now, let's open our terminal and change the directory to the root of our Vagrant repo. From there, we will just issue the vagrant up command. The output from our terminal should look as shown in the following screenshot:

After this, let's open our browser and go to 127.0.0.1:8500:

You will notice now that there are a lot more services listed in the Consul web UI than when we ran the module in the last chapter. This is because now, registrator is listening on the Unix socket, and any container with a port mapped to the host will be registered. So the good news is that our module is working. Now, let's add an application to the module.

The easiest way to do this is to add another container module to our node. So, let's add our bitbucket module. We do this by adding the class to our default.pp file that lives in our manifests directory:

We will also need to make some quick modifications to the bitbucket module so that we don't get duplicate declaration errors. Note that this is not something you would do in production. But it is good enough for our test lab. We need to comment out the top block of code as shown in the following screenshot:

We can even comment out the code as shown in the following screenshot:

This depends on whether you used the manifest module or the compose module. I used the compose module.

So, let's go back to our terminal in the root of our Vagrant repo and issue the vagrant provision command. The output of the terminal should look as shown in the following screenshot:

Now, let's look at our browser again. We can see that our bitbucket services have been registered, as shown in this screenshot:

We have the service discovery working; however, we still need to add another class to our module for DNS service discovery. So, let's go back to our consul module. We will add a new file called package.pp. In this file, we will install the bind package and add two templates, one to configure named.conf and the other to configure consul.conf in the directory named /etc/. Let's start coding. The first thing we will need to do is create our package.pp file in the manifests directory of our module:

We will add the following code to the file:

Now, let's create a templates folder. In this example, we are not parameterizing the files, and in a production instance, you would. That's why we are using the templates folder and not files:

Now, let's create a file called named.conf.erb and add the following code to it:

The code is just setting our DNS resolver to listen to 127.0.0.1. Remember that we have set port forwarding on our Consul container to forward port 53. That is how the host will connect to the container. Lastly, it will call our next template file /etc/named/consul.conf. Let's create that now:

The code that we will add is as follows:

You will note that we are forwarding port 8600, which is the port that Consul uses for its DNS traffic, and removing port 53. As TCP bind will use port 53, we will forward the request to 8600, as shown in the following piece of code:

We need to make one more change before we can run Puppet. We need to add the new code of package.pp to our init.pp file. We can do so like this:

Now, we can run our module. Let's go to the terminal and change to root of our Vagrant repo. We will issue the vagrant up command and if you already have a box running, just issue the vagrant destroy -f && vagrant up command. Now, let's check the web UI (127.0.0.1:8500):

As you can see in the preceding screenshot, we have a new service registered on port 8600 (consul-8600). Now, we need to make sure that our machines are listening to the right DNS servers on their interfaces. We are going to do this in servers.yaml, as I would usually add this configuration to my user data in AWS. You could very well control this with Puppet. So, in future, you can decide the right place for the configuration of your environment. The line we are going to add is - { shell: 'echo -e "PEERDNS=no\nDNS1=127.0.0.1\nDNS2=8.8.8.8">>/etc/sysconfig/network-scripts/ifcfg-enp0s3 && systemctl restart network'}. We will add it as shown in the following screenshot:

Now, let's go to our terminal and issue the vagrant up command. If you have a box already running then issue the vagrant destroy -f && vagrant up command. The terminal output should look like the one shown in the following screenshot:

We can then log in to our vagrant box using vagrant ssh and test whether our DNS setup works. We can do this by selecting a service and trying to ping it. We are going to choose our ping bitbucket-server-7990 service by entering the ping bitbucket-server-7990.service.consul command, and we should get the following results:

As you can see in the preceding screenshot, it returns the echo response as the loopback, as the service is running locally on this host. If we were external of the host, it would return the IP of the host that is running the service. Now, we run our container schedulers, such as Docker swarm, that have multiple host. We now know how service discovery works.

Now, let's have a look at what this would look like using Docker Compose.

In order to not repeat ourselves, let's make our init.pp file the same as the module that uses the manifests method. We have to make one small change to the params.pp file; docker-compose expects that you pass it strings. So, we need to remove the brackets around $reg_volume as shown in the following screenshot:

Then, we will add our package.pp file as we did earlier and also create the two templates for our bind config. Then, we need to update our docker-compose.yml.erb file in our templates directory. We need to add our second container, regisrator. We are going to use the same parameters as we did in the manifest module earlier in this chapter. The code for this should look as shown in the following screenshot:

You will also note that we changed the ports on our Consul container as we did earlier in the chapter (we removed port 53 and added 8600 tcp/udp). Now, we can go to our terminal, change to root of our Vagrant repo, and issue the vagrant up command. Our terminal should look like the one shown in the following screenshot:

Again, we can also check our browser at 127.0.0.1:8500:

As you can see in the preceding screenshot, it looks the same as it did earlier in the chapter.

Let's log in to our box and test our DNS service discovery. For this, enter the vagrant ssh command and then ping a service. This time, we will choose something different. We will use the ping consul-8500.service.consul command. We should get the following response after this:

So that's all for service discovery in this chapter. We will be picking it up again in the container scheduler chapter.

Before we can even start to code for our network, there are a few things we need. First, we need a key/value store. Docker will use this to map all the containers, IP addresses, and vxlans that are created. Seeing as there usually would be more than one host attached to a network, the key/value store is usually distributed to give it resiliency against failure. Luckily enough, we have already built a key/value store that we can take advantage of, it's Consul of course. The other configuration that you will need are extra args when we start our Docker Engine. This is to let Docker Engine know how to access the key/value store. These are the basic prerequisites that we need to get coding.

Let's create our first Docker network. To do this, we are going to add to our consul module. I am not going to do this twice for both manifests and docker-compose, as the configuration can be ported between the two. I am going to use the docker-compose module for my example. If this is the first time that you are creating a Docker network, it would be a worth while exercise to port the configuration to both. So, lets' start. We are only going to make changes to our install.pp file. The first change that we are going to make is to our extra arguments for our docker-engine daemon. We do this by adding the code shown in the following screenshot:

The code sets our key/value store's address and port. Then, it also tells other machines what interface and port we are advertising our networks on.

The next code we are going to add will create the network. We will create a new file called network.pp. Then, we will add the code shown in the following screenshot to it:

The next thing we will have to do is make sure that our classes get installed in the correct order, as the Docker network is dependent on Consul being there. If Consul is not there, our catalogue will fail. So, we need to use the contain functionality built into Puppet. We do this by adding the code shown in the following screenshot:

As you can see, we are just setting up a basic network. We could set things such as IP address range, gateway, and so on. If we do that, it would look like this:

Now that we have our code, let's go to our terminal and issue the vagrant up command from root of our Vagrant repo. Our terminal output should look like the one shown in the following screenshot:

Now, we can check to make sure that our network is there by logging in to our vagrant box (vagrant ssh from the root of our Vagrant repo). Once we log in to our box, we need to change to root (sudo -i) and then issue the docker network ls command. This will list the available networks on the box. The one we are looking for is docker-internal with the overlay driver:

As you can see from the output of our terminal, we were successful and our network has been configured. That is all we are going to do with networking in this chapter. In the next chapter, we will be attaching containers and spanning our Docker network across multiple hosts.

In the design, we are going to use four servers: node-01, node-02, node-03, and node-04. We will use node-01 to bootstrap our Consul cluster. We will add the other three nodes to the cluster as servers. They will be able to join the conciseness, vote, and replicate the key/value store. We will set up an IP network that is on the 172.17.8.0/24 network. We will map our container ports to the host ports that sit on the 172.17.8.0/24 network. The following image will show the network flow:

Now that we have our Consul cluster, we can look at what our ELK stack is going to look like. First, we will walk through the design of the network. The stack will be connected to the native Docker network. Note that I have not listed the IP addresses of our Docker network. The reason for this is that we will let the Docker daemon select the networking address range. For this solution, we are not going to route any traffic out of this network, so letting the daemon choose the IP range is fine. So you will also note that Elasticsearch is only connected to our Docker network. This is because we only want Logstash and Kibana. This is to keep other applications from being able to send requests or queries to Elasticsearch. You will note that both Logstash and Kibana are connected to both the Docker network and the host network. The reason for this is that we want applications to send their logs to Logstash, and we will want to access the Kibana web application:

To get the full picture of the architecture, we just need to overlay both the diagrams. So, let's start coding!!!

We will now look at the changes we are going to make to our new Vagrant repo. The first file that we are going to look at is our servers.yaml file. The first thing we need to do is change our base box. As we are going to be connecting containers to the native Docker network, our host machines must run a kernel version above 3.19. I have created a prebuilt vagrant box with just this configuration. It is the Puppetlabs box that we have been using in all the other chapters with the kernel updated to version 4.4:

As you can note in the preceding screenshot, the other change that we have made to the servers.yaml file is that we have added entries to the /etc/hosts directory. We have done this to simulate a traditional DNS infrastructure. If this had been in your production environment, we wouldn't have needed to add that configuration.

Now, we have to add the other three servers. The following screenshot will show exactly how it should look:

So, the ports that we will hit once all our servers are built are 8500 on node-01 (the Consul web UI 127.0.0.1:8500) and 8081 (the Kibana web UI 127.0.0.1:8081).

We are well aquatinted with the consul module now, but we are going to take it to the next level. For this chapter, we are just going to use the compose version. The reason why is because when you start getting into more complex applications or need to use an if statement to add logic, an .erb file gives us that freedom. There are a fair few changes to this module. So, let's start again at our params.pp file:

As you can see, we have added two new parameters. The first one is $consul_master_ip and the other parameter is $consul_is_master. We will use this to define which server will bootstrap our Consul cluster and which server will join the cluster. We have hardcoded the hostname of node-01. If this was a production module, I would not hardcode a hostname that should be a parameter that is looked up in Hiera (https://docs.puppetlabs.com/hiera/3.0/). We will pick up on this again when we look at our docker-compose.yml.erb file. The other parameters should look very familiar to you.

Next, let's look at our init.pp file:

As you can see here, we have not changed this file much, as we have added a Boolean ($consul_is_master). However, we will want to validate the input. We do this by calling the stdlib function, validate_bool.

Let's quickly browse through the install.pp file:

Now, let's look at the network.pp file:

Finally, we will look at the package.pp file:

As you can see, we have not made any changes to these files. Now, we can look at the file that will really have the logic for deploying our container. We will then move to our templates folder and look at our docker-compose.yml.erb file. This is where most of the changes in the module have been made.

So, let's look at the contents of the file, which are shown in the following screenshot:

So as you can see, the code in this file has doubled. Let's break it down into three pieces, as shown in the following screenshot:

In the first block of code, the first change that you will note is the if statement. This is a choice to determine whether the node will be a master for bootstrapping Consul or a server in the cluster. If you remember from our params.pp file, we set node-01 as our master. When we apply this class to our node, if its node-01, it will bootstrap the cluster. The next line that we want to pay attention to is as follows:

command: -server --client 0.0.0.0 --advertise <%= @consul_advertise %>  -bootstrap-expect <%= @consul_bootstrap_expect %>

We should just take note to compare the same line in the next block of code:

First, we can see that this is elsif, the second half of our if statement. So, this will be the block of code that will install Consul on the other three nodes. They will still be servers in the cluster. They will just not have the job of bootstrapping the cluster. We can tell this from the following line:

command: -server -bind 0.0.0.0 --client 0.0.0.0  --advertise <%= @consul_advertise %> -join <%= @consul_master_ip %>

Remember earlier that we looked at the same line from the first line of code. You see the difference? In block one, we declare -bootstrap-expect <%= @consul_bootstrap_expect %>, and in the second block, we declare -join <%= @consul_master_ip %>. By looking at the code, this is how we can tell the bootstrap order. Lastly, we can see that we are declaring <% end -%> to close the if statement.

Now, let's look at the last block of code:

As you can see, it's going to deploy the registrator container. As this sits outside the if statement, this container will be deployed on any node that the consul class is applied to. We have made a lot of progress till now. We should check the module changes before moving on to creating our new elastic modules. The one last thing we need to change is our default.pp manifest file, which is as follows:

As you can see, we have a node declaration for each node and have applied the consul class. Now, let's open our terminal change directory to the root of our Vagrant repo and issue the vagrant up command. This time, it will download a new base box from Hashicloud. So, depending on your Internet connection, this could take some time. Remember that the reason we need this new box is that it has an updated kernel to take advantage of the native Docker networking. In the last chapter, we were able to create a network, but we weren't able to connect containers to it. In this chapter, we will. Also, we are going to build four servers, so running Vagrant should take about 5 minutes. As soon as our first machine is up, we can log on to the consul web UI. There we can watch the progress as each node is joined.

As you can see in the following screenshot, our cluster is bootstrapped:

We can also check whether all our services are up and stable on the SERVICES tabs, as shown in this screenshot:

As you can see in the following screenshot, our second node has checked in:

The following screenshot shows what the screen looks like when we go to our SERVICES tab:

As you can see, our services have doubled. So things are looking good.

Now, the vagrant up command is complete, and our terminal output should look like the following screenshot:

Let's log back into our web browser to our Consul UI (127.0.0.1:8500). Under our NODES tab, we should now see all four nodes:

We can see that our cluster is in a good state as all four nodes have the same amount of services, which is 10, and that all services are green. The last thing that we need to check is our DNS service discovery. So lets login to one of our boxes. We will choose node-03. So in our terminal, let's issue the vagrant ssh node-03 command. We need to specify the node now as we have more than one vagrant box. We are going to ping the Consul service 8500. So, we just issue the ping consul-8500.service.consul command. The terminal output should look like the following screenshot:

This is now working perfectly. So, let's check one more thing now. We need to make sure that our Docker network is configured. For that, we will need to change the directory to root (sudo -i) and then issue the docker network ls command, as follows:

Now that everything is up and running, let's move on to our ELK stack.

One of the focuses I had when planning this book was to use examples that could be ported to the real world so that readers get some real value. The ELK stack is no exception. The ELK stack is a very powerful stack of applications that lets you collate all your application logs to see the health of your application. For more reading on the ELK stack, visit https://www.elastic.co/. It has great documentation on all the products. Now, let's start our first new module.

As per our design, there is an order in which we need to install the ELK stack. Seeing as both Logstash and Kibana depend on Elasticsearch, we will build it first. All the images that we are going to use in our modules are built, maintained, and released by Elasticsearch. So we can be sure that the quality is good. The first thing that we need to do is create a new module called <AUTHOR>-elasticsearch. We covered how to create a module in the previous chapter, so if you are unsure how to do this, go back and read that chapter again. Now that we have our module, let's move it into the modules directory in the root of our Vagrant repo.

Seeing as the containers are already built by elastic, these modules are going to be short and sweet. We are just going to add code to the init.pp file:

As you can see, we are calling the docker::image class to download ealsticsearch. In the docker::run class, we are calling our elasticearch container, and we are going to bind our container only to the docker-internal Docker network. You will note that we are not binding any ports. This is because, by default, this container will expose port 9200. We only want to expose port 9200 on the Docker network. Docker is smart enough to allow the exposed ports automatically on a Docker native network. In the next resource we are declaring just the host network for elasticsearch. We are specifying 0.0.0.0 as we don't know the IP that the container is going to get from the Docker network. As this service will be hidden from the outside world, this configuration will be fine. We will then map a persistent drive to keep our data.

The next thing that we need to do now is to add elasticsearch to a node. As per our design, we will add elasticsearch to node-02. We do this in our default.pp file in our manifests directory, as shown in the following screenshot:

You will note that I have used contain instead of include. This is because I want to make sure that the consul class is applied before the elasticsearch class, as we need our Docker network to be there before elasticsearch comes up. If the network is not there, our catalogue will fail as the container will not build.

The next module we are going to write is logstash. Our logstash module is going to be a little more complicated, as it will be on both the Docker network and the host network. The reason we want it on both is because we want applications to forward their logs to logstash. We also need logstash to talk to elasticsearch. Thus, we add logstash to the Docker network as well. We will create the module in the same way we did for elasticsearch. We will call our <AUTHOR>-logstash module. So, let's look at our code in our init.pp file, which is as follows:

Here, the first thing you will note is that we are creating a directory. This is to map to the container and will contain our logstash.conf file. The next declaration is a file type. This is our logstah.conf file, and as we can see, it's a template from the code. So, let's come back to it after looking at the rest of the code in our init.pp file. The next line of code will pull our logstash image from Docker Hub. In the docker::run class we will call our logstash container, use the logstash image, and attach the container to our docker-internal Docker network.

The next line of code will tell logstash to start using our logstash.conf file; we will then mount the directory we created earlier in our init.pp file to the container. Now, you can see in this module that we've exposed ports to the host network. In the last line, we tell logstash about our Elasticsearch host and Elasticsearch port. How does Logstash know where Elasticsearch is? We are not linking the containers like we did in previous chapters. This works in the same way to when we named our Elasticsearch container elasticsearch, and our Docker network has an inbuilt DNS server that lives at the address 127.0.0.11. Any container that joins that network will register itself as its container name. This is how services on the docker-internal network find each other.

The last thing we need to look at is our template file for our logstash.conf file that we declared in our init.pp file. So, create a new folder called templates in the root of our module and then a file called logstash.conf.erb. We will add the following configuration to accept logs from syslog and Docker.

Lastly, at the bottom, we put our Elasticsearch configuration, as shown in this screenshot:

Now, let's add our logstash module to node-03 in the same way that we did with our elastcsearch module.

Again, we will use contain instead of include. Now it's time to move on to our last module. We will create this in the same way as we have done for the last two modules. We will call this module <AUTHOR>-kibana.

In Kibana, we will only be adding code to the init.pp file, as shown in the following screenshot:

As you can see, we are downloading the kibana image. In the docker::run class, we are calling our kibana container using the kibana image, attaching the container to our local Docker network. In the next line, we are mapping the container port 5601 (Kibana's default port) to port 80 on the host. This is just for ease of use for our lab. In the last line, we are telling kibana how to connect to elasticsearch.

Let's add kibana to node-04 again using contain instead of include:

We are now ready to run our Vagrant environment. Let's open our terminal and change the directory to the root of our Vagrant repo. We will build this completely from scratch, so let's issue the vagrant destroy -f && vagrant up command.

This will take about 5 minutes or so to build, depending on your Internet connection, so please be patient. Once the build is complete, our terminal should have no errors and look like the following screenshot:

The next thing we will check is our Consul web UI (127.0.0.1:8500):

In the preceding screenshot, you can see that our Logstash and Kibana services are there, but where is Elasticsearch ? Don't worry, Elasticsearch is there, but we can't see it in Consul as we have not forwarded any ports to the host network. Registrator will only register services with exposed ports. We can make sure that our ELK stack is configured by logging in to our Kibana web UI (127.0.0.1:8080):

The next thing we need to do is click on the Create button. Then, if we go to the Discover tab, we can see the logs from Logstash:

Logstash's logs

In this example, we are going to build five servers, where two will be replicated masters and the other three nodes will be in the swarm cluster. As Docker Swarm needs a key/value store backend, we will use Consul. In this instance, we are not going to use our Consul modules; instead, we are going to use https://forge.puppetlabs.com/KyleAnderson/consul. The reason for this is that in all three examples, we are going to use different design choices. So, when you are trying to build a solution, you are exposed to more than one way to skin the cat. In this example, we are going to install Swarm and Consul onto the OS using Puppet and then run containers on top of it.

In this example, we are going create a new Vagrant repo. So, we will Git clone https://github.com/scotty-c/vagrant-template.git into the directory of our choice. The first thing that we will edit is the Puppetfile. This can be found in the root of our Vagrant repo. We will add the following changes to the file:

The next file we will edit is servers.yaml. Again, this is located in the root of our Vagrant repo. We are going to add five servers to it. So, I will break down this file into five parts, one for each server.

First, let's look at the code for server 1:

Now, let's look at the code for server 2:

The following screenshot shows the code for server 3:

The following screenshot shows the code for server 4:

Finally, the code for server 5 is as follows:

All this should look fairly familiar to you. The one call out is that we have used servers one through three as our cluster nodes. Four and five will be our masters.

The next thing we are going to do is add some values to Hiera. This is the first time we have used Hiera, so let's look at our hiera.yaml file to see our configuration in the root of our Vagrant repo, which is as follows:

As you can see, we have a basic hierarchy. We just have one file to look at, which is our global.yaml file. We can see that the file lives in our hieradata folder as it is declared as our data directory. So, if we open the global.yaml file, we are going to add the following values to it:

The first value will tell Swarm what version we want to use. The last value sets the version of Consul that we will be using, which is 0.6.3.

The next thing we need to do is write our module that will deploy both our Consul and Swarm clusters. We have created quite a few modules so far in the book, so we won't cover it again. In this instance, we will create a module called <AUTHOR>-config. We will then move the module into our modules folder at the root of our Vagrant repo. Now, let's look at what we are going to add to our init.pp file:

As you can see, this sets up our module. We will need to create a .pp file for each of our entries in our init.pp file, for example, consul_config.pp, swarm.pp, and so on. We have also declared one variable called consul_ip, whose value will actually come from Hiera, as that is where we set the value earlier. We will look at our files in alphabetical order. So, we will start with config::compose.pp, which is as follows:

In this class, we are setting up a registrator. We are going to use Docker Compose for this, as it is a familiar configuration and something that we have already covered. You will note that there is an if statement in the code. This is a logic to run the container only on a cluster member. We don't want to run container applications on our masters. The next file we will look at is consul_config.pp, which is as follows:

In this class, we will configure Consul, something that we have already covered in this book. I always like to look at multiple ways to do the same job, as you never know the solution that you might need to produce tomorrow and you always want the right tool for the right job. So, in this instance, we will not configure Consul in a container but natively on the host OS.

You can see that the code is broken into three blocks. The first block bootstraps our Consul cluster. You will note that the configurations are familiar, as they are the same as those we used to set up our container in an earlier chapter. The second block of code sets the members of the cluster after the cluster is bootstrapped. We have not seen this third block before. This sets up a service for Consul to monitor. This is just a taste of what we can manage, but we will look at this in anger in the next chapter. Let's look at our next file, that is, dns.pp:

You will note that this file is exactly the same as we used in our consul module. So, we can move on to the next file, which is run_containers.pp:

This is the class that will run our containers on our cluster. At the top, we are declaring that we want master number two to initiate the call to the cluster. We are going to deploy three container applications. The first is jenkins, and as you can see, we will expose Jenkins web port 8080 to the host that the container runs on. The next container is nginx, and we are going to forward both 80 and 443 to the host, but also connect nginx to our private network across our swarm-private cluster. To get our logs from nginx, we will tell our container to use syslog as the logging driver. The last application that we will run is redis. This will be the backend of nginx. You will note that we are not forwarding any ports, as we are hiding Redis on our internal network. Now that we have our containers sorted, we have one file left, swarm.pp. This will configure our Swarm cluster and internal Swarm network as follows:

The first resource declaration will install the Swarm binaries. The next resource will configure our Docker network on each host. The next if state will define if the Swarm node is a master or part of the cluster via hostname. As you can see, we are declaring a few defaults that are the first values that tell Swarm what backend to use Consul with. The second value tells Swarm the IP address of the Consul backend which is 172.17.8.101. The third value tells Swarm about the port it can access Swarm on, 8500. The fourth value tells Swarm which interface to advertise the cluster on, which in our case is enp0s8. The last value sets the root of the key value store Swarm will use. Now, let's create our templates folder in the root of our module.

We will create three files there. The first file will be consul.conf.erb and the contents will be as follows:

The next file will be named.conf.erb and its contents will be as follows:

The last file will be registrator.yml.erb, and the file's content will be as follows:

The next thing that we need to add is our config class to our default.pp manifest file in the manifest folder in the root of our Vagrant repo:

Now, our module is complete and we are ready to run our cluster. So, let's open our terminal and change the directory to the root of our Vagrant repo and issue the vagrant up command. Now, we are building five servers, so be patient. Once our last server is built, our terminal output should look like that shown in this screenshot:

Terminal output

We can now look at our Consul cluster's web UI at 127.0.0.1:9501 (remember that we changed our ports on our servers.yaml file) as follows:

Now, let's see what host our Jenkins service is running on:

In this example, my service has come up on cluster node 101. You could get a different host for your cluster. So, we need to check what port 8080 forwards to our servers.yaml file guest. In my example, it's 8081. So, if I go to my web browser and open a new tab and browse to 127.0.0.1:8081, we will get the Jenkins page, which is as follows:

You can do the same with nginx, and I will leave that up to you as a challenge.

So, in this example, we are going to build three nodes. The first will be our UCP controller and the other two nodes will be UCP HA replicas that give the design some fault tolerance. As UCP is a wrapped product, I am not going to go into all the moving parts.

Refer to the following diagram to get a visualization of the main components:

We are going to do something different in this module, just to show you that our modules are portable to any OS. We will build this module with the Ubuntu 14.04 server. Again, for this environment, we are going to create a new Vagrant repo. So, let's Git clone our Vagrant repo (https://github.com/scotty-c/vagrant-template.git). As in the last topic, we will look at the plumbing first, before we write our module. The first thing we are going to do is create a file called config.json. This will have your Docker Hub auth in the file:

The next will be docker_subscription.lic. This will contain your trial license.

Now, let's look at our servers.yaml file in the root of our Vagrant repo, which is as follows:

The main call out here is that now we are using the puppetlabs/ubuntu-14.04-64-puppet-enterprise vagrant box. We have changed yum to apt-get. Then, we are copying both our config.json and docker_subscription.lic files to their correct place on our vagrant box.

Now, we will look at the changes we need to make to our Puppetfile:

You will see that we need a few new modules from the Forge. The Docker module is familiar, as is stdlib. We will need Puppetlab's apt module to control our repos that Ubuntu will use to pull Docker. The last module is the Puppetlabs module for UCP itself. To find out more information about this module, you can read all about it at https://forge.puppetlabs.com/puppetlabs/docker_ucp. We will write a module that wraps this class and configures it for our environment.

Now, let's look at our Hiera file in hieradata/global.yaml:

As you can see, we have added two values. The first is ucpconfig::ucp_url:, which we will set to our first vagrant box. The next is the value for ucpconfig::ucp_fingerprint:, which we will leave blank for the moment. But remember it, as we will come back to this later in the topic.

Now, we will create a module called <AUTHOR>-ucpconfig. We have done this a few times now, so once you have created a module, create a folder in the root of our Vagrant repo called modules and move ucpconfig into that folder.

We will then create three manifests files in the module's manifest directory. The first will be master.pp, the second will be node.pp, and the last file will be params.pp.

Now, let's add our code to the params.pp file, as follows:

As you can see, we have four values: Ucp_url, which comes from Hiera; ucp_username, whose default value is set to admin; we then have ucp_password whose default value is set to orca. The last value is ucp_fingerprint, which again comes from Hiera. Now, in a production environment, I would set both the username and password in Hiera and overwrite the defaults, which we have set in params.pp. In this case, as it is a test lab, we will just use the defaults.

The next file we will look at is our init.pp file, which is as follows:

You can see that at the top of the class, we are mapping our params.pp file. The next declaration installs the docker class and sets the socket_bind parameter for the daemon. Now, the next bit of logic defines whether the node is a master or a node depending on the hostname. As you can see, we are only setting ucp-01 as our master.

Now, let's look at master.pp:

In this class, we have the logic to install the UCP controller or master. At the top of the class, we are mapping our parameters to our init.pp file. The next block of code calls the docker_ucp class. As you can see, we are setting the value of controller to true, the host address to our second interface, the alternate name of the cluster to our first interface, and the version to 1.0.1 (which is the latest at the time of writing this book). We will then set the ports for both the controller and Swarm. Then, we will tell UCP about the Docker socket location and also the location of the license file.

Now, let's look at our last file, node.pp:

As you can see, most of the settings might look familiar. The call out for a node is that we need to point it to the controller URL (which we set in Hiera). We will get to know about the admin username and password and the cluster fingerprint just a little bit later. So, that completes our module. We now need to add our class to our nodes which we will do by adding the default.pp manifest file in the manifests/default.pp location from the root of our Vagrant repo, as follows:

Let's go to our terminal and change the directory to the root of our Vagrant repo. This time, we are going to do something different. We are going to issue the vagrant up ucp-01 command. This will bring up only the first node. We do this as we need to get the fingerprint that is generated as UCP comes up.

Our terminal output should look like that shown in the following screenshot:

Terminal output

You will note that the fingerprint has been displayed on your terminal output. For my example, the fingerprint is INFO[0031] UCP Server SSL: SHA1 Fingerprint=C2:7C:BB:C8:CF:26:59:0F:DB:BB:11:BC:02:18:C4:A4:18:C4:05:4E. So, we will add this to our Hiera file, which is global.yaml:

Now that we have our first node up, we should be able to log in to the web UI. We do this in our browser. We will go to https:127.0.0.1:8443 and get the login page as follows:

We will then add the username and password that we set in our params.pp file:

Then, after we log in, you can see that we have a health cluster, as follows:

Health cluster after logging in

Now, let's return to our terminal and issue the vagrant up ucp-02 && vagrant up ucp-03 command.

Once that is complete, if we look at our web UI, we can see that we have three nodes in our cluster, which are as follows:

In this book, we are not going to go into how to administer the cluster through the web UI. I would definitely recommend that you explore this product, as it has some really cool features. All the documentation is available at https://docs.docker.com/ucp/overview/.

We will be building Kubernetes on a single node. The reason for this is that it will cut out some of the complexity on how to use flannel over the Docker bridge. This module will give you a good understanding of how Kubernetes works using more advanced Puppet techniques. If you want to take your knowledge further after you have mastered this chapter, I would recommend that you check out the module on the forge at https://forge.puppetlabs.com/garethr/kubernetes. This module really takes Puppet and Kubernetes to the next level.

So what we are going to code looks like the following diagram:

As you can see, we have a container that runs etcd (to read more about etcd, go to https://coreos.com/etcd/docs/latest/). etcd is similar to Consul, which we are familiar with. In the next few containers, we are going to use hyperkube (https://godoc.org/k8s.io/kubernetes/pkg/hyperkube). This will load balance the required Kubernetes components across multiple containers for us. It seems pretty easy, right? Let's get into the code so that we get a better perspective on all the moving parts.

We are going to create a new Vagrant repo again. We won't cover how to do that as we have covered it twice in this chapter. If you are unsure, just take a look at the initial part of this chapter.

Once we have created our Vagrant repo, let's open our servers.yaml file, as follows:

As you can see, there is nothing special there that we have not covered in this book. There's just a single node that we mentioned earlier, kubernetes. The next file we will look at is our Puppetfile. We will, of course, need our Docker module, stdlib, and lastly, wget. We need wget to get kubectl:

That is all the plumbing that we need to set up for our repo. Let's create a new module called <AUTHOR>-kubernetes_docker. Once it is created, we will move our module to the modules directory in the root of our Vagrant repo.

We are going to create two new folders in our module. The first will be the templates folder, and the other folder will be the lib directory. We will get to the lib directory toward the end of our coding. The first file we will create and edit is docker-compose.yml.erb. The reason for this is that it is the foundation of our module. We will add the following code to it:

Let's break this file down into three chunks, as there is a lot going on there. The first block of code is going to set up our etcd cluster. You can see from the screenshot name that we are using Google's official images, and we are using etcd version 2.2.1. We are giving the container access to the host network. Then, in the command resource, we pass some arguments to etcd as it starts.

The next container we create is hyperkube. Again, it is an official Google image. Now, we are giving this container access to a lot of host volumes, the host network, and the host process, making the container privileged. This is because the first container will bootstrap Kubernetes and it will spawn more containers running the various Kubernertes components. Now, in the command resource, we are again passing some arguments for hyperkube. The two major ones we need to worry about are the API server address and config manifests. You will note that we have a mapped folder from /kubeconfig:/etc/kubernetes/manifests:ro. We are going to modify our manifest file to make our Kubernetes environment available to the outside world. We will get to that next. But, we will finish looking at the code in this file first.

The last container and the third block of code is going to set up our service proxy. We are going to give this container access to the host network and process. In the command resource, we are going to specify that this container is a proxy. The next thing to take notice of is that we specify where the proxy can find the API. Now, let's create the next file, master.json.erb. This is the file that hyperkube will use to schedule all the Kubernetes components, which are as follows:

As you can see, we have defined three more containers. This is the first time we will define a Kubernetes pod (http://kubernetes.io/docs/user-guide/pods/). A pod is a group of containers that creates an application. It is similar to what we have done with Docker Compose. As you can see, we have changed all the IP addresses to the <%= @master_ip %> parameter. We will create four new files: apps.pp, config.pp, install.pp, and params.pp.

We will now move on to our files in the modules manifest directory. Now, strap yourselves in, as this is where the magic happens. Well, that's not true. The magic happens here and in our lib directory. We will need to write some custom types and providers for Puppet to be able to control Kubernertes as kubectl is the user interface (for types, visit https://docs.puppetlabs.com/guides/custom_types.html, and for providers, visit https://docs.puppetlabs.com/guides/provider_development.html).

Let's start with our init,pp file, which is as follows:

As you can see, there is not much in this file. We are going to use our init.pp file to control the order in which classes are executed. We are also declaring param <%= @master_ip %>. We will now move on to our install.pp file, as follows:

In this file, we install Docker as we did before. We will place our two templates that we created earlier. Then, we will run Docker Compose to bring up our cluster. Now, we will move on to config.pp, as follows:

The first thing that we declare is that we want wget to be our kubectl client, and we place it at /usr/bin/ (http://kubernetes.io/docs/user-guide/kubectl/kubectl/). You really need to understand what this interface does; otherwise, you might get a bit lost from here. So, I suggest that you have a fairly good idea of what kubectl is and what it is capable of. Next, we will make it executable and available for all our users. Now, this last piece of code does not make sense, as we have not called the kubectl_config class yet:

We now need to jump to our lib directory. This first thing we will do is create all our folders that we need. The first folder that we will create is puppet under the lib directory. We will look at our custom types first. We will create a folder called type under puppet. The following screenshot will give you a visualization of the structure:

Under the type folder, we will create a file called kubectl_config.rb. In that file, we will add the new parameters of type as follows:

Let me explain what is happening here. In the first line, we are going to declare our new type, kubectl_config. We are then going to set the default value of the new type when it is declared as present. We are going to declare three values to our type, name, cluster, and kube_context. These are all settings that we will add to our config file that kubectl will use when we interface with it. Now, we will create a folder under the lib directory called provider. Then, under that, we will create a folder with the same name as our custom type, kubectl_config. Inside that folder, we will create a file called ruby.rb. In this file, we will put the Ruby code that provides logic to our type as follows:

A provider needs to have three methods for Puppet to be able to run the code. They are exsists?, create, and destroy. These are pretty easy to understand. The exists? method checks whether the type has already been executed by Puppet, create runs the type, and destroy is invoked when the type is set to absent.

We are now going to work through the file, from top to bottom. We need to first load a few Ruby libraries for some of our methods. We will then tie this provider to our type. The next thing that we need to declare is the kubectl executable.

Now we will write our first method, interface. This will get the IP address from the hostname of the box. We will then create three more methods. We will also create an array and add all our configuration to them. You will note that we are mapping our parameters from our type in the arrays.

In our exists? method, we will check for our kubectl config file.

In our create method, we are calling our kubectl executable and then passing our arrays as arguments. We will then link our config file to roots' home directory (this is fine for our lab. In a production environment, I would use a named user account).

Lastly, we will remove the config file if the type is set to absent. We will now go back to our manifests directory and look at our last file, which is apps.pp:

In this file, we are going to run a container application on our Kubernetes cluster. Again, we will write another custom type and provider. Before we get to that, we should look at the code in this class. As you can see, our type is called kubernetes_run. We can see that our service is named nginx, the Docker image we will pull is nginx, and we will then expose port 80.

Let's go back to our lib directory. We will then create a file in our type folder called kubernetes_run.rb. In this file, we will set up our custom type as we did earlier:

As you can see, we are mapping the same parameters that we had in our apps.pp file. We will then create a folder under the provider folder with the same name as our kubernetes_run type. Again, in the newly created directory, we will create a file called ruby.rb. It will have the code shown in the following screenshot:

In this file, we are going to add two commands this time: the first is kubectl and the second is docker. We will create two methods, again with arrays that map the values from our type.

Now, let's look at our exists? method. We are going to pass an array as an argument to kubectl to check whether the service exists. We are then going to catch the error if kubectl throws an error with the request and returns false. This is used if there are no services deployed on the cluster.

In our create method, we will first pass an array to kubectl to get the nodes in the cluster. We are using this as an arbitrary command to make sure that the cluster is up. Under that, we will capture the error and retry the command until it is successful. Once it is successful, we will deploy our container with the ensure resource.

In the destroy method, we will use docker to remove our container.

Now we have all our coding done. We just need to add our class to our node by editing our default.pp file in the manifests folder in the root of our Vagrant repo as follows:

Now, let's open our terminal and change the directory to the root of our Vagrant repo and issue the vagrant up command. Once Puppet has completed its run, our terminal should look like the following screenshot:

We will now log in to our vagrant box by issuing the vagrant ssh command and then sudo -i and change to root. Once we are root, we will look at our service on our cluster. We will do this by issuing the kubectl get svc command as follows:

As you can see, we have two services running our cluster: Kubernetes and nginx. If we go to our web browser and go to the address we gave to the second network interface, http://172.17.9.101, we will get the following nginx default page:

Now, our cluster is running successfully with our nginx service.

As I mentioned in the introduction of this chapter, we will be using Docker Swarm for this solution. The reason for this choice is that I want to highlight some of the features of Swarm as it has come on leaps and bounds in the last few releases. For the logging portion of this chapter, we are going to deploy our three containers and let Swarm schedule them. We will use a combination of the Docker networking DNS and our service discovery with Consul to tie everything together. In Swarm, we will use the same servers as we did in the last chapter: three member nodes and two replicated masters. Each node will be a member of our Consul cluster. We will again use Puppet to install Consul on the host system natively.

In this chapter, we will build on the code that we used in the last chapter for Docker Swarm. So, we will go through the plumbing of the Vagrant repo fairly swiftly, only calling out the differences from the last chapter. We are going to create a new Vagrant repo again for this chapter. You should be a master at this by now. Once the new repo is set up, open the servers.yaml file. We will add the following code to it:

Code for the severs.yaml file

As you can see, it's not that different from the last chapter. There is one call out. We have added a new line to each server - { shell: 'echo -e "PEERDNS=no\nDNS1=127.0.0.1\nDNS2=8.8.8.8">>/etc/sysconfig/network-scripts/ifcfg-enp0s8 && systemctl restart network'}. We will add this as the server is multihomed. We will want to make sure that we are resolving DNS correctly on each interface.

The next thing we will look at is the puppetfile, which is as follows:

As you can see, there are no new changes compared with the last chapter. So, let's move to our Hiera file located at heiradata/global.yml in the root of our module:

Again, we are setting the Swarm version to v1.1.3, and the backend is set to consul. We set the IP address of the first node in the Consul cluster, and we set the Consul port to 8500. We will set the swarm interface that we will advertise from, and last but not least, we will set our Consul version to 0.6.3.

Now, we will create our module. We will again call the config module. Once you have created your <AUTHOR>-config module, move it to the modules folder in the root of your Vagrant repo.

Now that we have our module, let's add our code to it. We will need to create the following files in the manifests directory: compose.pp, consul_config, dns.pp, run_containers.pp, and swarm.pp. We have no params.pp as we are using Hiera in this example.

So, let's go through the files in an alphabetical order. In our compose.pp file, we will add the following code:

As you can see from the code, we are adding our docker-compose.yml file to any node that is not a swarm master. We will come back to the docker compose.yml file when we look at the templates directory. The next file is consul_config.pp, which is as follows:

In this file, we are declaring our Consul cluster and defining the bootstrap server. This should look familiar as it is the same code that we used in the last chapter. The next file is dns.pp, which is given as follows:

Again, this code should look familiar to you, as we have used it before in the last chapter. Just to recap, this is setting and configuring our bind package to use Consul as the DNS server. The next file we will look at is init.pp:

In the init.pp file, we are just ordering our classes within our module. We will now move on to run_containers.pp. This is where we will schedule our ELK containers across the swarm cluster:

Let's have a look at this in detail, as there is a lot of new code here. The first declaration that we will use is to schedule the containers from the second Swarm master.

The next block of code will configure our logstash container. We will need to first have these containers here in this example, as we are using them as the syslog server. If at the time of spawning the containers they can't connect to logstash on TCP port 5000, the build of the container will fail. So, let's move on to the configuration of logstash. We will use the container that I put in, as it is the official container with the logstash.conf file that we already added. We will then add logstash to our internal swarm-private Docker network and expose all the ports for logstash on all networks. So, we can pipe logs from anywhere to it. After this, we will set the location of elasticsearch and then we will give the command to start.

Logstash

In the second block of code, we will install and configure elasticsearch. We will use the official elasticsearch container (version 2.1.0). We will only add elasticsearch to our private Docker network, swarm-private. We will make the data persistent by declaring a volume mapping. We will set the command with arguments to start elasticsearch with the command value. Next, we will set the log drive to syslog and the syslog server to tcp://logstash-5000.service.consul:5000. Note that we are using our Consul service discovery address as we are exposing logstash on the external network. Lastly, we set the dependency on logstash. As I mentioned earlier, the syslog server needs to be available at the time this container spawns, so we need logstash to be there prior to either this container or kibana. Talking of Kibana, let's move on to our last block of code.

In our kibana container, we will add the following configuration. First, we will use the official kibana image (Version 4.3.0). We will add kibana to our swarm-private network so it can access our elasticsearch container. We will map and expose ports 5601 to 80 on the host network. In the last few lines, we will set the syslog configuration in the same way as we did with elasticsearch.

Now, it's time for our last file, swarm.pp, which is as follows:

In this code, we are configuring our Swarm cluster and Docker network.

We will now move to our templates folder in the root of our module. We need to create three files. The two files Consul.conf.erb and named.conf.erb are for our bind config. The last file is our registrator.yml.erb Docker compose file. We will add the code to the following files.

Let's first see the code for consul.conf.erb, which is as follows:

Now, let's see the code for named.conf.erb, which is as follows:

Finally, let's see the code for registrator.yml.erb, which is as follows:

All the code in these files should look fairly familiar, as we have used it in previous chapters.

Now, we have just one more configuration before we can run our cluster. So, let's go to our default.pp manifest file in the manifests folder located in the root of our Vagrant repo.

Now, we will add the relevant node definitions to our manifest file:

We are ready to go to our terminal, where we will change the directory to the root of our Vagrant repo. As so many times that we did before, we will issue the vagrant up command. If you have the boxes still configured from the last chapter, issue the vagrant destroy -f && vagrant up command.

Once Vagrant is run and Puppet has built our five nodes, we can now open a browser and enter the http://127.0.0.1:9501/. We should get the following page after this:

As you can see, all our services are shown with green color that displays the state of health. We will now need to find what node our kibana container is running on. We will do that by clicking on the kibana service.

In my example, kibana has come up on swarm-101. If this is not the same for you, don't worry as the Swarm cluster could have scheduled the container on any of the three nodes. Now, I will open a browser tab and enter the 127.0.0.1:8001/, as shown in the following screenshot:

If your host is different, consult the servers.yaml file to get the right port.

We will then create our index and click on the Discovery tab, and as you can see in this screenshot, we have our logs coming in:

Logs after creating index

One really good thing about using Consul is that Hashicorp did an awesome job at documenting their applications. Consul is no exception. If you would like to read more about the options you have with Consul monitoring, refer to the documentation at https://www.consul.io/docs/agent/services.html and https://www.consul.io/docs/agent/checks.html. We are going to set up both checks and a service. In the last chapter, we wrote a service with Consul to monitor our Docker service on every node:

On the Consul web UI, we get the following reading of the Docker service on the node:

We are going to roll out all our new checks on both the Swarm masters. The reason for this is that both the nodes are external from the nodes cluster. The abstraction gives us the benefit of not having to worry about the nodes that the containers are running on. You also have the monitoring polling from multiple locations. For example, in AWS, your Swarm masters could be split across multiple AZs. So, if you lose an AZ, your monitoring will still be available.

As we are going to use the logging solution from the example that we covered in the previous section, we will check and make sure that both Logstash and Kibana are available; Logstash on port 5000 and Kibana on port 80.

We are going to add two new service checks to our config module in the consul_config.pp file, as follows:

As you can see, we have set a TCP check for both kibana and logstash, and we will use the service discovery address to test the connections. We will open our terminal and change the directory to the root of our Vagrant repo.

Now, we are going to assume that your five boxes are running. We will issue the command to Vagrant to provision only the two master nodes. This command is vagrant provision swarm-master-01 && vagrant provision swarm-master-02. We will then open our web browser and enter 127.0.0.1:9501. We can then click on swarm-master-01 or swarm-master-02, the choice is up to you. After this, we will get the following result:

As you can see, our monitoring was successful. We will move back to our code and add a check for our swarm master to determine its health. We will do that with the following code:

We will then issue the Vagrant provision command, vagrant provision swarm-master-01 && vagrant provision swarm-master-02. Again, we will open our web browser and click on swarm-master-01 or swarm-master-02. You should get the following result after this:

As you can see from the information in the check, we can easily see which Swarm master is primary, the strategy for scheduling. This will come in really handy when you have any issues.

So as you can see, Consul is a really handy tool, and if you want to take away the things we covered in this chapter, you can really do some cool stuff.

Docker Swarm has two kind of nodes: master nodes and member nodes. Each one of these has different built-in protection for failure. The first node type we will look at is master nodes.

In the last topic, we set up a health check to get the information regarding our Swarm cluster. There, we saw that we had a master or primary Swarm master and a replica. Swarm replicates all its cluster information over TCP port 4000. So, to simulate failure, we are going to turn off the master. My master is swarm-master-01, but yours could be different. We will use the health check that we already created to test out the failure and watch how Swarm handles itself. We will issue the vagrant halt swarm-master-01 command. We will open up our browser again to our Consul web UI, 127.0.0.1:9501. As we can see in the following screenshot, swarm-master-02 is now a master:

Now, we will move on to container resecluding with our Swarm node HA. As of version 1.1.3, Swarm is shipped with a feature where a container will respawn on a healthy node if the original node fails. There are some rules to this such as when you have filtering rules or linked containers. To know more on this topic, you can read the Docker docs about Swarm located at https://github.com/docker/swarm/tree/master/experimental.

To test this out, I will halt the node that hosts Kibana. We will need to add some code to our kibana container so that it will restart on failure. This is added to the env resource, as shown in the following screenshot:

We will first need to kill our old container to add the restart policy. We can do that by setting the ensure resource to absent and running the Vagrant provision swarm-master-02.

Once the Vagrant run is complete, we will change it back to present and run vagrant provision swarm-master-02.

For me, my kibana container is on swarm-102 (this could be different for you). Once that node fails, kibana will restart on a healthy node. So, let's issue vagrant halt swarm-102. If we go to our Consul URL, 127.0.0.1:9501, we should see some failures on our nodes and checks, as shown in the following screenshot:

If you wait a minute or so, you will see that kibana alerts came back and the container spawned on another server. For me, kibana came back on swarm-101, as you can see in the following screenshot:

We can then go to our browser and look at kibana. For me, it will be at 127.0.0.1:8001:

Kibana after connecting to Elasticsearch

As you can see, all our logs are there; our service discovery worked perfectly as once the container changed nodes, our health checks turned green.

When we first sit down and start development on a new module, some of the things that we should consider are: whether we can make our module OS agnostic, how we can run the module on multiple machines without logic changes or extra development, and how we can protect our sensitive data?

The answer to all these questions is Hiera.

We are able to leverage Hiera by parameterizing our classes. This will allow Puppet to automatically look up Hiera at the beginning of the catalogue compilation. So, let's explore some examples of the data that you will put in Hiera.

Note that in our chapter about container schedulers, we briefly used Hiera. We set the following values:

As you can see, we are setting parameters such as versions. Why would we want to set versions in Hiera and not straight in the module? If we set the version of Consul, for example, we might be running version 5.0 in production. Hashicorp just released version 6.0 by changing the Hiera value for the environment (for more information about Puppet environments, go to https://docs.puppetlabs.com/puppet/latest/reference/environments.html). In Hiera versions from dev to 6.0, we can run multiple versions of the application with no module development.

The same can be done with IP addresses or URLs. So in the hieradata for dev, your Swarm cluster URL could be dev.swarm.local and production could be just swarm.local.

Another type of data that you will want to separate is passwords/keys. You wouldn't want the same password/key in your dev environment as there are in production. Again, Hiera will let you obfuscate this data.

We can then take the protection of this data further using eyaml, which Puppet supports. This allows you to use keys to encrypt your Hiera .yaml files. So, when the files are checked into source control, they are encrypted. This helps prevent data leakage. For more information on eyaml, visit https://puppetlabs.com/blog/encrypt-your-data-using-hiera-eyaml.

As you see, Hiera gives you the flexibility to move your data from the module to Hiera to externalize configurations, making the module stateless.

There are some really handy tips that you can refer to when using Hiera. Puppet allows you to call functions, lookups, and query facts from Hiera. Mastering these features will come in very handy. If this is new to you, read the document available at https://docs.puppetlabs.com/hiera/3.1/variables.html before moving on in the chapter.

So, let's look at an example of looking up facts from Hiera. First, why would you want to do this? One really good reason is IP address lookups. If you have a class that you are applying to three nodes and you have to advertise an IP address like we did in our consul module, setting the IP address in Hiera will not work, as the IP address will be different for each machine. We create a file called node.yaml and add the IP address there. The issue is that we will now have multiple Hiera files. Every time Puppet loads the catalogue, it looks up all the Hiera files to check whether any values have changed. The more files we have, the more load it puts on the master and the slower our Puppet runs will become. So we can tell Hiera to look up the fact and the interface we want to advertise. Here is an example of what the code would look like:

ucpconfig::ucp_host_address: "%{::ipaddress_eth1}"

The one call out is that if we called the fact from a module, we would call it with the fully qualified name, $::ipaddress_eth1. Unfortunately, Hiera does not support the use of this. So we can use the short name for the ::ipaddress_eth1 fact.

The first thing that we need to do is create a new Vagrant repo for this chapter. By now, we should be masters at creating a new Vagrant repo. Once we have created that, we will create a new module called <AUTHOR>-ucpconfig and move it to our modules directory in the root of our Vagrant repo. We will first set up our servers.yml file by adding the code shown in the following screenshot:

As you can see, we are setting up three servers, where ucp-01 will be the master of the cluster and the other two nodes will join the cluster. We will add two more files: config.json and docker_subscription.lic. Here, config.json will contain the authorization key to Docker Hub, and docker_subscription.lic will contain our trial license for UCP. Note that we covered both of these files in the container scheduler chapter. If you are having issues setting up these files, refer to that chapter.

The file we will now look at is the puppetfile. We need to add the code shown in the following screenshot to it:

Now that we have our Vagrant repo set up, we can move on to our module. We will need to create four files: config.pp, master.pp, node.pp, and params.pp.

The first file we will look at is params.pp. Again, I'd like to code this file first as it sets a good foundation for the rest of the module. We do this as follows:

As you can see, we are setting all our parameters. We will look at each one in depth as we apply it to the class. This way, we will have context on the value we are setting. You might have noted that we have set a lot of variables to empty strings. This is because we will use Hiera to look up the values. I have hardcoded some values in as well, such as the UCP version as 1.0.0. This is just a default value if there is no value in Hiera. However, we will be setting the value for UCP to 1.0.3. The expected behavior is that the Hiera value will take precedence. You will note that there is a fact that we are referencing, which is $::ucp_fingerprint. This is a custom fact. This will automate the passing of the UCP fingerprint. If you remember, in the container scheduler chapter, we had to build ucp-01 to get the fingerprint and add it to Hiera for the other nodes' benefit. With the custom fact, we will automate the process.

To create a custom fact, we will first need to create a lib folder in the root of the module. Under that folder, we will create a folder called facter. In that folder, we will create a file called ucp_fingerprint.rb. When writing a custom fact, the filename needs to be the same as the fact's name. The code that we will add to our custom fact is as follows:

In this code, you can see that we are adding a bash command to query our UCP master in order to get the fingerprint. I have hardcoded the IP address of the master in this fact. In our environment, we would write logic for that to be more fluid so that we are allowed to have multiple environments, hostnames, and so on. The main part to take away from the custom fact is the command itself.

We will now move back to our init.pp file, which is as follows:

The first thing you can see it that we are declaring all our variables at the top of our class. This is where Puppet will look up Hiera and match any variable that we have declared. The first block of code in the module is going to set up our Docker daemon. We are going to add the extra configuration to tell the daemon where it can find the backend for the Docker native network. Then, we will declare a case statement on the $::hostname fact. You will note that we have set a parameter for the hostname. This is to make our module more portable. Inside the case statement, if the host is a master, you can see that we will use our Consul container as the backend for our Docker network. Then, we will order the execution of our classes that are applied to the node.

In the next block of code in the case statement, we have declared the $ucp_deploy_node variable for the $::hostname fact. We will use this node to deploy Kubernetes from. We will get back to this later in the topic. The final block of code in our case statement is the catch all or default. If Puppet cannot find the fact of $::hostname in either of our declared variables, it will apply these classes.

We will now move on to our master.pp file, which is as follows:

The first thing you will note is that we are declaring our variables again at the top of this class. We are doing this as we are declaring many parameters, and this makes our module a lot more readable. Doing this in complex modules is a must, and I would really recommend you to follow this practice. It will make your life a lot easier when it comes to debugging. As you can see in the following screenshot, we are tying the parameters back to our init.pp file:

As you can see from the preceding code, there are very few types which we are values that we are setting in the module.

We will now move on to our node.pp file, as follows:

As you can see, we are declaring the parameters at the top of the class, again tying them back to our init.pp file. We have declared most of our values as we will use Hiera.

We will now move on to our config.pp file, as follows:

In this class, we are going to make a few vital configurations for our cluster. So, we will walk through each block of code individually. Let's take a look at the first one:

In this block, we will declare our variables as we have in all the classes in this module. One call out that I have not mentioned yet is that we are only declaring the variables that are applied to its class. We are not declaring all the parameters. Let's look at the next set of code now:

In this block of code, we will pass an array of packages that we need to curl our UCP master in order to get the SSL bundle that we will need for TLS coms between nodes. Now, let's see the third block:

In this code, we are going to create a shell script called get_ca.sh.erb to get the bundle from the master. Let's create the file, and the first thing that we will need to do is create a templates folder in the root of the module. Then, we can create our get_ca.sh.erb file in the templates folder. We will add the following code to the file:

As you can see in the script, we need to create an auth token and pass it to the API. Puppet does not natively handle tasks like these well, as we are using variables inside the curl command. Creating a template file and running an exec function is fine as long as we make it idempotent. In the next block of code, we will do this:

In this block of code, we will run the preceding script. You can see that we have set the parameters, command path, and cwd (current working directory). The next resource is creates that tells Puppet that there should be a file called ca.pem in the current working directory. If Puppet finds that the file does not exist, it will execute the exec, and if the file does exist, Puppet will do nothing. This will give our exec idempotency.

In the next block of code, we will create a file in /etc/profile.d, which will then point the Docker daemon on each node to the master's IP address, allowing us to schedule containers across the cluster:

Now, let's create a docker.sh file in our templates directory. In the file, we will put the following code:

In this file, we are telling the Docker daemon to use TLS, setting the location of the key files that we got from the bundle earlier in the class. The last thing we are setting is the Docker host that will point to the UCP master.

This last block of code in this class should look very familiar, as we are setting up a Docker network:

We can now move on to the the file where all our data is, our Hiera file. That file is located in the hieradata folder, which is present in the root of our Vagrant repo. The following screenshot shows the various data present in the Hiera file:

So, let's list out all the data we have defined here. We are defining the UCP master as ucp-01, and our deploy node is ucp-03 (this is the node that we are deploying Kubernetes from). The UCP URL of the master is https://172.17.10.101. This is used when we connect nodes to the master and also when we get our ca bundle and UCP fingerprint. We keep the username and password as admin and orca. We will use UCP version 1.0.3. We will then use the Hiera fact lookup that we discussed earlier in the book to set the host address and alternate names for UCP.

The next parameter will tell UCP that we will use an internal CA. We will then set our scheduler to use spread, define the ports for both Swarm and the controller, tell UCP to persevere the certs, and send the location of the license file for UCP. Next, we will set some date for Consul, such as the master IP, the interface to advertise, the image to use, and how many nodes to expect at the time of booting the Consul cluster. Lastly, we will set the variables we want our Docker daemon to use, such as the name of our Docker network, swarm-private, the network driver, overlay, the path to our certs, and the Docker host that we set in our docker.sh file placed at /etc/profile.d/.

So, as you can see, we have a whole lot of data in Hiera. However, as we have already discussed, there is data that could be changed for different environments. So as you can see, there is a massive benefit in making your module stateless and abstracting your data to Hiera, especially if you want to write modules that can scale easily.

We will now add the following code to our manifest file, default.pp located in the manifests folder in the root of our Vagrant repo. The following code defines our node definition:

We can then open our terminal and change the directory to the root of our Vagrant repo. We will then issue the vagrant up command to run Vagrant. Once the three boxes are built, you should get the following terminal output:

We can then log in to the web URL at https://127.0.0.1:8443:

We will log in with the admin username and orca password. In the following screenshot, we can see that our cluster is up and healthy:

Since I thought to finish off with a bang, we will do something pretty cool now. I wanted to deploy an application that had multiple containers in which we could use interlock to showcase application routing/load balancing. What better application than Kubernetes. As I mentioned earlier, there are limitations to running Kubernetes like this, and it is only for lab purposes. The skills we can take away and apply to our Puppet modules is application routing/load balancing. In our last topic, we set the parameter for $ucp_deploy_node in the case statement in our init.pp file. In that particular block of code, we had a class called compose.pp. This is the class that will deploy Kubernetes across our UCP cluster. Let's look at the file:

The first resource just creates a directory called kubernetes. This is where we will place our Docker Compose file. Now, you will notice something different about how we are running Docker Compose. An exec? Why would we use an exec when we have a perfectly good provider that is tried and trusted. The reason we are using the exec in this case is because we are changing $PATH during the last Puppet run. What do I mean by that? Remember the file we added to the /etc/profile.d/ directory? It changed the shell settings for where DOCKER_HOST is pointing. This will only come into effect in the next Puppet run, so the Docker daemon will not be pointing to the cluster. This will mean that all the kubernetes containers will come up on one host. This will cause a failure in the catalogue, as we will get a port collision from two containers using 8080. Now, this will only come into effect when we run the module all at once, as a single catalogue.

Now, let's have a look at our Docker Compose file:

In this book, I have been stressing on how I prefer using the Docker Compose method to deploy my container apps. This Docker Compose file is a perfect example why. We have seven containers in this compose file. I find it easy as we have all the logic right there in front of us and we have to write minimal code. Now, let's look at the code. The first thing that is different and that we wouldn't have seen so far is that we are declaring version 2. This is because version 1.6.2 of Docker Compose has been released (https://github.com/docker/compose/releases/tag/1.6.2). So, to take advantage of the new features, we need to declare that we want to use version 2.

The first container we are declaring is interlock. We are going to use interlock as our application router that will make server requests to the Kubernetes API. For this container, we are going to forward ports, 443, 8080, and 8443, to the host. We will then map /etc/docker from the host machine to the container. The reason for this is we need the keys to connect to the Swarm API. So, we will take advantage of the bundle we installed earlier in the chapter.

In the command resource, we will tell interlock where to find the certs, the Swarm URL, and lastly, that we want to use haproxy. We will then add this container to our overlay network, swarm-private. The next thing is in the environment resource, we will set a constraint and we will tell Compose that we can only run interlock on ucp-03. We will do this to avoid port collision with the Kubernetes API service. The next container is etcd. Not much has changed regarding this since we configured Kubernetes in the scheduler chapter, so we will move on. The next container is the kubernetes API service. The one thing we need to call out with this is that we are declaring in the environment resource - INTERLOCK_DATA={"hostname":"kubernetes","domain":"ucp-demo.local"}. This is the URL that interlock will look for when it sends the request to the API.

This is the main reason we are running kubernetes to gain the skills of application routing. So, I won't go through the rest of the containers. Kubernetes is very well documented at http://kubernetes.io/. I would recommend that you read up on the features and explore Kubernetes—its a beast, there is so much to learn.

So, now we have all our code. Let's run it!

Just to see the end-to-end build process, we will open our terminal and change the directory to the root of our Vagrant repo. If you have the servers built from the earlier topics, issue vagrant destroy -f && vagrant up; if not, just a simple vagrant up command will do. Once the Puppet run is complete, our terminal should have the following output:

We can then log in to our web UI at https://127.0.0.1:8443:

Then, we will log in with the admin username and the orca password. We should see the following screenshot once we login successfully:

You will note that we have one application now. If we click on the application twice, we can see that Kubernetes is up and running:

You will note that we have our container split across ucp-02 and ucp-03 due to the environment settings in our Docker Compose file. One thing to take note of is that interlock in on ucp-03 and the Kubernetes API service is on ucp-02.

Now that we have built everything successfully, we need to log in to ucp-03 and download the kubectl client. We can achieve that by issuing the following command:

$'wget https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin/linux/amd64/kubectl'

So let's log in to ucp-03 and issue the vagrant ssh ucp-03 command from the root of our Vagrant repo. We will then change to root (sudo -i). Then, we will issue the wget command, as shown in the following screenshot:

We will then make the file executable by issuing the following command:

$'chmod +x kubectl'

Now remember that we set a URL in the environment settings for the - INTERLOCK_DATA={"hostname":"kubernetes","domain":"ucp-demo.local"} API container. We will need to set that value in our host file. So, use your favorite way to edit files on the local machine, such as vim, nano sed, and so on, and add 172.17.10.103 kubernetes.ucp-demo.local. The IP address points to ucp-03 as that is where interlock is running.

Now we are ready to test our cluster. We will do that by issuing the ./kubectl -s kubernetes.ucp-demo.local get nodes command. We should get the following output after this:

As you can see, everything is up and running. If we go back to our UCP console, you can see that the API server is running on ucp-02 (with the IP address 172.17.10.102). So, how are we doing this? Interlock is processing the HTTP calls on 8080 and routing them to our API server. This tells us that our application routing is in place. This is a very basic example, but something you should play with as you can really design some slick solutions using interlock.