From an architectural perspective, Keystone presents the simplest service in the OpenStack composition. It is the core component that provides identity service and it integrates functions for authentication, catalog services, and policies to register and manage different tenants and users in the OpenStack projects. The API requests between OpenStack services are being processed by Keystone to ensure that the right user or service is able to utilize the requested OpenStack service. Keystone performs numerous authentication mechanisms such as username/password as well as a token-authentication-based system. Additionally, it is possible to integrate it with an existing backend directory such as Lightweight Directory Access Protocol (LDAP) and the Pluggable Authentication Module (PAM).

A similar real-life example is a city game. You can purchase a gaming day card and profit by playing a certain number of games during a certain period of time. Before you start gaming, you have to ask for the card to get an access to the city at the main entrance of the city game. Every time you would like to try a new game, you must check in at the game stage machine. This will generate a request, which is mapped to a central authentication system to check the validity of the card and its warranty, to profit the requested game. By analogy, the token in Keystone can be compared to the gaming day card except that it does not diminish anything from your request. The identity service is being considered as a central and common authentication system that provides access to the users.

Although it was briefly claimed that Swift would be made available to the users along with the OpenStack components, it is interesting to see how Swift has empowered what is referred to as cloud storage. Most of the enterprises in the last decade did not hide their fears about a critical part of the IT infrastructure—the storage where the data is held. Thus, the purchasing of expensive hardware to be in the safe zone had become a habit. There are always certain challenges that are faced by storage engineers and no doubt, one of these challenges include the task of minimizing downtime while increasing the data availability. Despite the rise of many smart solutions for storage systems during the last few years, we still need to make changes to the traditional way. Make it cloudy! Swift was introduced to fulfill this mission.

We will leave the details pertaining to the Swift architecture for later, but you should keep in mind that Swift is an object storage software, which has a number of benefits:

When I had my first presentation on the core components and architecture of OpenStack with my first cloud software company, I was surprised by a question raised by the CTO: What is the difference between Glance and Swift? Both handle storage. Well, despite my deployment of OpenStack (Cacti and Diablo were released at the time) and familiarity with the majority of the component's services, I found the question quite tough to answer! As a system architect or technical designer, you may come across the following questions: What is the difference between them? Why do I need to integrate such a solution? On one hand, it is important to distinguish the system interaction components so that it will be easier to troubleshoot and operate within the production environments. On the other hand, it is important to satisfy the needs and conditions that go beyond your IT infrastructure limits.

To alleviate any confusion, we keep it simple. Swift and Glance are storage systems. However, the difference between the two is in what they store. Swift is designed to be an object storage where you can keep data such as virtual disks, images, backup archiving, and so forth, while Glance stores metadata of images. Metadata can be information such as kernel, disk images, disk format, and so forth. Do not be surprised that Glance was originally designed to store images. Since the first release of OpenStack included only Nova and Swift (Austin code name October 21, 2010), Glance was integrated with the second release (Bexar code name February 23, 2011).

The mission of Glance is to be an image registry. From this point, we can conclude how OpenStack has paved the way to being more modular and loosely coupled core component model. Using Glance to store virtual disk images is a possibility. From an architectural level, including more advanced ways to query image information via the Image Service API provided by Glance through an independent image storage backend such as Swift brings more valuable performance and well-organized system core services. In this way, a client (can be a user or an external service) will be able to register a new virtual disk image, for example, to stream it from a highly scalable and redundant store. At this level, as a technical operator, you may face another challenge—performance. This will be discussed at the end of the book.

You may wonder whether there is another way to have storage in OpenStack. Indeed, the management of the persistent block storage is being integrated into OpenStack by using Cinder. Its main capability to present block-level storage provides raw volumes that can be built into logical volumes in the filesystem and mounted in the virtual machine.

Some of the features that Cinder offers are as follows:

Several storage options have been proposed in the OpenStack core. Without a doubt, you may be asked this question: What kind of storage will be the best for you? With a decision-making process, a list of pros and cons should be made. The following is a very simplistic table that describes the difference between the storage types in OpenStack to avoid any confusion when choosing the storage management option for your future architecture design:

It is very important to keep in mind that unlike Glance and Keystone services, Cinder features are delivered by orchestrating volume providers through the configurable setup driver's architectures such as IBM, NetApp, Nexenta, and VMware.

Whatever choice you have made, it is always considered good advice since nothing is perfect. If Cinder is proven as an ideal solution or a replacement of the old nova-volume service that existed before the Folsom release on an architectural level, it is important to know that Cinder has organized and created a catalog of block-based storage devices with several differing characteristics. However, it is obvious if we consider the limitation of commodity storage redundancy and autoscaling. Eventually, the block storage service as the main core of OpenStack can be improved if a few gaps are filled, such as the addition of values:

The aforementioned Cinder specification reveals its Non-vendor-lock-in characteristic, where it is possible to change the backend easily or perform data migration between two different storage backends. Therefore, a better storage design architecture in a Cinder use case will bring a third party into the scalability game. More details will be covered in Chapter 4, Learning OpenStack Storage – Deploying the Hybrid Storage Model. For instance, you can keep in mind that Cinder is essential for our private cloud design, but it misses some capacity scaling features.

As you may already know, Nova is the most original core component of OpenStack. From an architectural level, it is considered one of the most complicated components of OpenStack.

In a nutshell, Nova runs a large number of requests, which are collaborated to respond to a user request into running VM. Let's break down the blob image of nova by assuming that its architecture as a distributed application needs orchestration to carry out tasks between different components.

Queue provides a central hub to pass messages between daemons. This is where information is shared between different Nova daemons by facilitating the communication between discrete processes in an asynchronous way.

Any service can easily communicate with any other service via the APIs and queue a service. One major advantage of the queuing system is that it can buffer a large buffer workload. Rather than using an RPC service, a queue system can queue a large workload and give an eventual consistency.

A database stores most of the build-time and runtime state for the cloud infrastructure, including instance types that are available for use, instances in use, available networks, and projects. It is the second essential piece of sharing information in all OpenStack components.

Neutron provides a real Network as a Service (NaaS) between interface devices that are managed by OpenStack services such as Nova. There are various characteristics that should be considered for Neutron:

Neutron has many core network features that are constantly growing and maturing. Some of these features are useful for routers, virtual switches, and the SDN networking controllers.

Neutron introduces new concepts, which includes the following:

In Neutron's low-level orchestration of Layer 1 through Layer 3, components such as IP addressing, subnetting, and routing can also manage high-level services. For example, Neutron provides Load Balancing as a Service (LBaaS) utilizing HAProxy to distribute the traffic among multiple compute node instances.

Horizon is the web dashboard that pools all the different pieces together from your OpenStack ecosystem.

Horizon provides a web frontend for OpenStack services. Currently, it includes all the OpenStack services as well as some incubated projects. It was designed as a stateless and data-less web application—it does nothing more than initiating actions in the OpenStack services via API calls and displaying information that OpenStack returns to the Horizon. It does not keep any data except the session information in its own data store. It is designed to be a reference implementation that can be customized and extended by operators for a particular cloud. It forms the basis for several public clouds—most notably the HP Public Cloud and at its heart, is its extensible modular approach to construction.

Horizon is based on a series of modules called panels that define the interaction of each service. Its modules can be enabled or disabled, depending on the service availability of the particular cloud. In addition to this functional flexibility, Horizon is easy to style with Cascading Style Sheets (CSS).

Most cloud provider distributions provide a company's specific theme for their dashboard implementation.

It is important to understand how different services in OpenStack work together, leading to a running virtual machine. We have already seen how a request is processed in OpenStack via APIs. Now, we can go further and closely check how such services and subsystems, which includes authentication, computing, images, networks, queuing, and databases, work in tandem with performing a complete workflow to provide an instance in OpenStack. The next series of steps describes how service components work together once a submission of an instance provisioning request has been done:

Let's figure out how things can be seen by referring to the following simple architecture diagram:

You may have certain limitations that are typically associated with network switches. Network switches create a lot of virtual LANs and virtual networks that specify whether there is a lot of input to data centers.

Let's imagine that we have 250 compute hosts scenario. You can conclude that a mesh of rack servers will be placed in the data center.

Now, you take the step to grow our data center, and to be geographically data-redundant in Europe and Africa: a data center in London, Amsterdam and Tunis.

We have a data center on each of these new locations and each of these locations are able to communicate with each other. At this point, a new terminology is introduced—cell concept.

To scale this out even further, we will take into consideration the entire system. We will take just the worker nodes and put them in other cells.

Another special scheduler works as a top-level cell and enforces the request into the child cell. Now, the child cells can do the work, and they can worry about VLAN and network issues.

The cells can share certain pieces of infrastructure, such as the database, authentication service Keystone, and some of the Glance image services. This is depicted in the following diagram:

Deployment is a huge step in distinguishing all the OpenStack components that were covered previously. It confirms your understanding of how to start designing a complete OpenStack environment. Of course, assuming the versatility and flexibility of such a cloud management platform, OpenStack offers several possibilities that might be considered an advantage. However, on the other hand, you may face a challenge of taking the right design decision that suits your needs.

Basically, what you should consider in the first place is the responsibility in the Cloud. Depending on your cloud computing mission, it is essential to know what a coordinating IaaS is. The following are the use cases:

Apart from the knowledge of the aforementioned cloud service model providers, there are a few master keys that you should take into account in order to bring a well-defined architecture to a good basis that is ready to be deployed.

Though the system architecture design has evolved and is accompanied by the adoption of several methodology frameworks, many enterprises have successfully deployed OpenStack environments by going through a 3D process—a conceptual model design, logical model design, and physical model design.

It might be obvious that complexity increases from the conceptual to the logical design and from the logical to the physical design.

The conceptual model design

As the first conceptual phase, we will have a high-level reflection on what we will need from certain generic classes from the OpenStack architecture:

Class	Role
Compute	Stores virtual machine images Provides a user interface
Image	Stores disk files Provides a user interface
Object storage	Provides a user interface
Block storage	Provides volumes Provides a user interface
Network	Provides network connectivity Provides a user interface
Identity	Provides authentication
Dashboard	Graphical user interface

Let's map the generic basic classes in the following simplified diagram:

Keep in mind that the illustrated diagram will be refined over and over again since we will aim to integrate more services within our first basic design. In other words, we are following an incremental design approach within which we should exploit the flexibility of the OpenStack architecture.

At this level, we can have a vision and direction of the main goal without worrying about the details.

The logical model design

Based on the conceptual reflection phase, we are ready to construct the logical design. Most probably, you have a good idea about different OpenStack core components, which will be the basis of the formulation of the logical design that is done by laying down their logical representations.

Even though we have already taken the core of the OpenStack services component by component, you may need to map each of them to a logical solution within the complete design.

To do so, we will start by outlining the relations and dependencies between the services core of OpenStack. Most importantly, we aim to delve into the architectural level by keeping the details for the end. Thus, we will take into account the repartition of the OpenStack services between the new package services—the cloud controller and the compute node. You may wonder why such a consideration goes through a physical design classification. However, seeing the cloud controller and compute nodes as simple packages that encapsulate a bunch of OpenStack services, will help you refine your design at an early stage. Furthermore, this approach will plan in advance further high availability and scalability features, which allow you to introduce them later in more detail.

Note

Chapter 3, Learning OpenStack Clustering – Cloud Controllers and Compute Nodes, describes in depth how to distribute the OpenStack services between cloud controllers and compute nodes.

Thus, the physical model design will be elaborated based on the previous theoretical phases by assigning parameters and values to our design. Let's start with our first logical iteration:

Obviously, in a highly available setup, we should achieve a degree of redundancy in each service within OpenStack. You may wonder about the critical OpenStack services claimed in the first part of this chapter—the database and message queue. Why can't they be separately clustered or packaged on their own? This is a pertinent question. Remember that we are still in the second logical phase where we try to dive slowly and softly into the infrastructure without getting into the details. Besides, we keep on going from general to specific models, where we focus more on the generic details. Decoupling RabbitMQ or MySQL from now on may lead to your design being overlooked. Alternatively, you may risk skipping other generic design topics. On the other hand, preparing a generic logical design will help you to not stick to just one possible combination, since the future physical designs will rely on it.

Note

What about storage

The previous logical figure includes several essentials solutions for a high-scalable and redundant OpenStack environment such as virtual IP (VIP), HAProxy, and Pacemaker. The aforementioned technologies will be discussed in more detail in Chapter 6, Openstack HA and Failover.

Compute nodes are relatively simple as they are intended just to run the virtual machine's workload. In order to manage the VMs, the nova-compute service can be assigned for each compute node. Besides, we should not forget that the compute nodes will not be isolated; a Neutron agent and an optional Ceilometer compute agent may run this node.

What about storage?

You should now have a deeper understanding of the storage types within OpenStack—Swift and Cinder.

However, we did not cover a third-party software-defined storage called Ceph, which may combine or replace either or both of Swift and Cinder.

More details will be covered in Chapter 4, Learning OpenStack Storage – Deploying the Hybrid Storage Model. For now, we will design from a basic point where we have to decide how Cinder and/or Swift will be a part of our logical design.

Ultimately, a storage system becomes more complicated when it faces an exponential growth in the amount of data. Thus, the designing of your storage system is one of the critical steps that is required for a robust architecture.

Depending on your OpenStack business size environment, how much data do you need to store? Will your future PaaS construct a wide range of applications that run heavy-analysis data? What about the planned Environment as a Service (EaaS) model? Developers will need to incrementally back up their virtual machine's snapshots. We need persistent storage.

Don't put all your eggs in one basket. This is why we will include Cinder and Swift in the mission. Many thinkers will ask the following question: If one can be satisfied by ephemeral storage, why offer block storage? To answer this question, you may think about ephemeral storage as the place where the end user will not be able to access the virtual disk associated with its VM when it is terminated. Ephemeral storage should mainly be used in production that takes place in a high-scale environment, where users are actively concerned about their data, VM, or application. If you plan that your storage design should be 100 percent persistent, backing up everything wisely will make you feel better. This helps you figure out the best way to store data that grows exponentially by using specific techniques that allow them to be made available at any time. Remember that the current design applies for medium to large infrastructures. Ephemeral storage can also be a choice for certain users, for example, when they consider building a test environment. Considering the same case for Swift, we have claimed previously that the object storage might be used to store machine images, but when is this the case?

Simply, when you provide the extra hardware that fulfils certain Swift requirements: replication and redundancy.

Running a wide production environment while storing machine images on the local file system is not really good practice. First, the image can be accessed by different services and requested by thousands of users at a time. No wonder the controller is already exhausted by the forwarding and routing of the requests between the different APIs in addition to the computation of each resources through disk I/O, memory, and CPU. Each request will cause performance degradation, but it will not fail! Keeping an image in a filesystem under a heavy load will certainly bring the controller to a high latency and it may fail.

Henceforth, we might consider loosely coupled models, where the storage with a specific performance is considered a best fit for the production environment.

Thus, Swift will be used to store images, while Cinder will be used for persistent volumes for virtual machines (check the Swift controller node):

Obviously, Cinder LVM does not provide any redundancy capability between the Cinder LVM nodes. Losing the data in a Cinder LVM node is a disaster. You may want to perform a backup for each node. This can be helpful, but it will be a very tedious task! Let's design for resiliency. We have put what's necessary on the table. Now, what we need is a glue!

Networking

One of the most complicated system designing steps is the part concerning the network! Now, let's look under the hood to see how all the different services that were defined previously should be connected.

The logical networking design

OpenStack shows a wide range of networking configurations that vary between the basic and complicated. Terms such as Open vSwitch, Neutron coupled with the VLAN segmentation, and VMware NSX with Neutron are not intuitively obvious from their appearance to be able to be implemented without fetching their use case in our design. Thus, this important step implies that you may differ between different network topologies because of the reasons behind why every choice was made and why it may work for a given use case.

OpenStack has moved from simplistic network features to complicated features, but of course there is a reason—more flexibility! This is why OpenStack is here. It brings as much flexibility as it can! Without taking any random network-related decisions, let's see which network modes are available. We will keep on filtering until we hit the first correct target topology:

Network mode	Network specification	Implementation
Nova-network	Flat network design without VMs grouping or isolation	Nova-network FlatDHCP
	Multiple tenants and VMs Predefined fixed private IP space size	Nova-network VLANManager
Neutron	Multiple tenants and VMs Predefined switches and routers configuration	Neutron VLAN
	Increased tenants and VM groups Lower performance	Neutron GRE

The preceding table shows a simple differentiation between two different logical network designs for OpenStack. Every mode shows its own requirements, which is very important and should be taken into consideration before the deployment.

Arguing about our example choice, since we aim to deploy a very flexible large-scale environment, we will toggle the Neutron choice for networking management instead of the nova-network.

Note that it is also possible to keep on going with nova-network, but you have to worry about SPOF. Since the nova-network service can run on a single node (cloud controller) next to other network services such as DHCP and DNS, it is required in this case to implement your nova-network service in a multihost networking model, where cloning such a service in every compute node will save you from a bottleneck scenario. In addition, the choice was made for Neutron, since we started from a basic network deployment. We will cover more advanced features in the subsequent chapters of this book.

We would like to exploit a major advantage of Neutron compared to the nova-network, which is the virtualization of layers 2 and 3 of the OSI network model.

Remember that Neutron will enable us to support more subnets per private network segment. Based on Open vSwitch, you will discover that Neutron is becoming a vast network technology.

Let's see how we can expose our logical network design. For performance reasons, it is highly recommended to implement a topology that can handle different types of traffic by using separated logical networks.

In this way, as your network grows, it will still be manageable in case a sudden bottleneck or an unexpected failure affects a segment.

Network layout

Let us look at the different networks that are needed to operate the OpenStack environment.

The external network

The features of an external or a public network are as follows:

• Global connectivity

• It performs SNAT from the VM instances that run on the compute node to the Internet for floating IPs

  Note

  SNAT refers to Source Network Address Translation. It allows traffic from a private network to go out to the Internet. OpenStack supports SNAT through its Neutron APIs for routers. More information can be found at http://en.wikipedia.org/wiki/Network_address_translation.

• It provides connection to the controller nodes in order to access the OpenStack interfaces

• It provides virtual IPs (VIPs) for public endpoints that are used to connect the OpenStack services APIs

  Note

  A VIP is an IP address that is shared among several servers. It involves a one-to-many mapping of the IP addresses. Its main purpose is to provide a redundancy for the attached servers and VIPs.

• It provides a connection to the external services that need to be public, such as an access to the Zabbix monitoring system

  Note

  While using VLANs, by tagging networks and combining multiple networks into one Network Interface Card (NIC), you can optionally leave the public network untagged for that NIC to make the access to the OpenStack dashboard and the public OpenStack API endpoints simple.

The storage network

The main feature of a storage network is that it separates the storage traffic by means of a VLAN isolation.

The management network

An orchestrator node was not described previously since it is not a native OpenStack service. Different nodes need to get IP addresses, the DNS, and the DHCP service where the Orchestrator node comes into play. You should also keep in mind that in a large environment, you will need a node provisioning technique which your nodes will be configured to boot, by using PXE and TFTP.

Thus, the management network will act as an Orchestrator data network that provides the following:

• Administrative networking tasks

• OpenStack services communication

• Separate HA traffic

Note

For a large-scale OpenStack environment, you can use a dedicated network for most of the critical internal OpenStack communication, such as the RabbitMQ messaging and the DB queries, by separating the messaging and database into separate cluster nodes.

The internal VM traffic

The features of the internal virtual machine network are as follows:

• Private network between virtual machines

• Nonroutable IPs

• Closed network between the virtual machines and the network L3 nodes, routing to the Internet, and the floating IPs backwards to the VMs

For the sake of simplicity, we will not go into the details of, for instance, the Neutron VLAN segmentation.

The next step is to validate our network design in a simple diagram:

Let's bring down the cloud architecture's layers under the scope and see what we have. Basically, we have Software as a Service (SaaS), which operates at all layers of the IT stack. Then comes Platform as a Service (PaaS), where databases and application containers are delivered on demand to reach the bottom, where we find Infrastructure as a Service (IaaS) delivering on-demand resources, such as virtual machines, networks, and storage. All these layers form complete, basic stacks of the cloud. You should think about how each layer of the cloud should be developed and implemented.

Obviously, layered dependency relies on the ability to create full stacks and deliver them under a request of simple steps. Remember that we are talking about a large scalable environment! The amazing switch to bigger environments nowadays is to simplify everything as much as possible. System architecture and software design are becoming more and more complicated. Every new release of software affords new functions and new configurations. Then, you are asked to integrate the new software in a particular platform where somehow, sufficient documentation about requirements or troubleshooting is missing! You may ask yourself questions such as: Did something change? What kind of change? To which person should we assign a ticket to fix it? What if it just does not work? According to your operational team, the software needs to be updated often in order to apply the new functions. The update might happen every time you get that e-mail asking you to update the software. You may start to wonder whether your operational team will be happy about this announcement, contrary to the software provider who sent the e-mail with the header; "we are happy to announce the release of the new version of our software; please push the button."

Let's take a serious example that crystallizes this situation. A company's operational team was extremely happy about purchasing a new storage appliance that worked well on redundancy. During the first few months, everyone was happy; nothing was broken! It worked like a charm!

When the day came to change the charm to a true headache, the storage system failed to fail over. Both nodes stopped working. No one could access any data! In spite of the existence of a backup somewhere else, the operational team did not like the "was that highly available?" part. After a long night of investigation, the error of causing the HA to fail was concluded from the log files: there was an appliance system update! The previous version was somehow automatically updated and broke the HA function in the active node. The update was propagated to the passive one. Unfortunately, the active version decided to fail over and tackle the cluster that was passive. However, that did not work. It was as if there was a bug somewhere in the code of the previous release!

What about if you are running similar solutions for other systems? Everything is running software to keep it running! In this case, it is wise to stop for a while and ask yourself questions such as this: What is missing? Shall I hire more people for system maintenance and troubleshooting? Obviously, if you take a look at the previous example, you will probably notice that the owner of the hardware does not really own it!

The simple reason is that being dependent on external parties will affect your infrastructure efficiency. Well, you may ask a pertinent question: Shall I rewrite the software appliance by myself? Let's reformulate the question: Shall I write the code? The answer, almost always, is yes! It is an ambiguous answer, right? Let's keep using examples in order to clear out this fogginess. We talked about DevOps, the synergetic point between developers and operational guys. Everything is communicated between them thanks to the magic of DevOps. Remember that it is our goal to simplify things as much as possible! Administrating and deploying a large infrastructure would not be possible without adopting a new philosophy: Infrastructure as code. At this point, we bring in another aspect of the DevOps style: we see our machines as pieces of code! In fact, we have now assigned the main tasks of DevOps.

Where everything will be seen as code, it might be possible to model a given infrastructure as modules of code. What you need to do is just abstract, design, implement, and deploy the infrastructure.

Furthermore, in such a paradigm, it will be essential to adhere to the same discipline as an infrastructure developer as compared to a software developer.

Without doubt, these terms are quite misty at the first glance. For this reason, you should ask this question related to our main topic about OpenStack: if infrastructure as code is so important for a well-organized infrastructure deployment, what is the case with OpenStack? The answer to this question is relatively simple: developers, network, and compute engineers and operators are working alongside each other to develop OpenStack Cloud that will run our next generation data center. This is the DevOps spirit.

OpenStack is an open source project, and its code is extended, modified, and fixed in every release. Of course, it is not your primary mission to check the code and dive into its different modules and functions. This is not our goal! What can we do with DevOps, then? Eventually, we will "DevOps" the code that makes the code run! As you might have noticed, a key measure of the success of a DevOps story is automation. Everything in a given infrastructure must be automated!

Ultimately, the code that abstracts, models, and builds the OpenStack infrastructure is committed to source code management. Most likely, we reach a point where we shift our OpenStack infrastructure from a code base to a redeployable one by following the latest software development best practices.

At this stage, you should be aware of the quality of your OpenStack infrastructure deployment, which roughly depends on the quality of the code that describes it.

Maintaining the code needs more attention in order to have a bug-free environment when it is delivered as a final release. We will consider the "bug" term in an infrastructure development context as harmful and functional to the system.

It is important to highlight a critical point that you should keep in mind during all deployment stages: automated systems are not able to understand human error when it is propagated to all pieces of your infrastructure. This is essential, and there is no way to ignore it. The same way is applicable to traditional software development discipline. You'll have to go through an ensemble of phases and cycles using agile methodologies to end up with a final release that is a normally bug-free software version in production.

Remember the example given previously? Surprises do happen! However, if an error occurs in a small corner of a specific system and needs to be fixed in that specific independent system, it might not be the same when considering the automation of a large infrastructure.

On the other hand, if mistakes cannot be totally eradicated at the first stage, you should think about introducing more flexibility into your systems by allowing wise changes without exaggeration. The code's life management is shown in the following figure:

Changes can be scary—very scary indeed! To handle changes, it is recommended that you:

Keep checking every point that has been described previously till you start to get more confident that your OpenStack infrastructure is being conducted by code that won't break.

To keep the OpenStack environment working with a minimum rate of surprises, ensure that its infrastructure code delivers the functionalities that are required.

Beyond these considerations, we will put the OpenStack deployment in a toolchain, where it will inform you about how we will conduct the infrastructure development from the test stage to the production stage. Underpinning every tool selection must be the purpose of your testing endeavors, and it will also help you ensure that you build the right thing.

Continuous integration and delivery

Let's see how continuous integration can be applied to OpenStack. Whatever we use for system management tools or automation code will be kept as a standard and basic topology, as shown in the next model, where the following requirements are met:

• SMTA can be any System Management Tool Artifact, such as Chef cookbook, Puppet manifest, Ansible playbook, or juju charms.

• VCS or Version Control System stores the previous artifacts that are built continuously with a continuous integration server. Git can be a good outfit for our VCS. Alternatively, you can use other systems, such as CVS, Subversion, Bazaar, or any other system that you are most familiar with.

• Jenkins is a perfect tool that listens to changes in version control and makes the continuous integration testing automated in production clones for test purposes.

Take a look at the model:

The proposed topology for infrastructure as code consists of infrastructure configuration files (Chef cookbooks, Puppet artifacts, and Vagrant files) that are recorded in a version control system and are built continuously by the means of a continuous integration (CI) server (Jenkins, in our case). Infrastructure configuration files can be used to set up a unit test environment (a virtual environment using Vagrant, for example) and makes use of any system-management tool to provision the infrastructure (Chef, Puppet, and so on). The CI server keeps listening to changes in version control and automatically propagates any new versions to be tested, and then it listens to target environments in production.

Note

Vagrant allows you to build a virtual environment very easily; it is based on Oracle VirtualBox (https://www.virtualbox.org/) to run virtual machines, so you will need these before moving on with the installation in your test environment.

Using such model designs could make our development and integration code infrastructure more valuable. Obviously, the previous OpenStack toolchain highlights the test environment before moving on to production, which is normal! However, you should give a lot of importance to, and care a good deal about, the testing stage, although this might be a very time-consuming task.

Especially in our case, with infrastructure code within OpenStack, things can be difficult for complicated and dependent systems. This makes it imperative to ensure an automated and consistent testing of the infrastructure code.

The best way to do this is to keep testing thoroughly in a repeated way till you gain confidence about your code. When you do, introducing changes to your code when it's needed shouldn't be an issue.

Let's keep on going, get the perfect tool running, and push the button.

If you are not familiar with the Git command line, do not worry, because we will use an integrated development environment (such as Eclipse), which provides a great Git plugin.

Feel free to use any Linux distribution. The next setup will use CentOS 6.5 64 bit as the standard operating system.

Ensure that Java and its dependencies are installed:

packtpub@dev$ sudo yum install java
packtpub@dev$ sudo yum install gcc-c++

You can download Eclipse for CentOS from here:

packtpub@dev$ wget http://mirror.netcologne.de/eclipse/technology/ epp/downloads/ release/juno/SR2/eclipse-automotive-juno-SR2-incubation-linux-gtk-x86_64.tar.gz

Extract the Eclipse to the /opt directory:

packtpub@dev$ sudo tar -xvzf eclipse-automotive-juno-SR2-incubation-linux-gtk-x86_64.tar.gz -C /opt/

Create a symlink:

packtpub@dev$ sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse

To install Ruby, you need to go from the Eclipse menu bar and navigate to Help | Install New Software. From the pending list, navigate to Program Languages | Dynamic Languages Toolkit - Ruby Development Tools:

Install Git:

packtpub@dev$ sudo yum install git

Check the correctness of the Git installation:

packtpub@chef$ git --version

Bring the magic EGit plugin into the action link in order to develop with Git in Eclipse. We do this in the same way from the Eclipse update manager by navigating to the Help | Install new Software menu entry. You will need to add the following URL installation to EGit:

http://download.eclipse.org/egit/updates

You will then see the following screen:

If you are not familiar with Chef, the following section will cover the basic setup and the most important parts of a Chef server, and you can see how it looks.

When you think about a typical chef, you may think of cookbooks, recipes, and knives! These are what a chef needs in order to make awesome dishes. The taste of the food on a plate depends on the creativity of the chef. We do the same thing in terms of cooking: we use a basic cookbook, from which we derive the right recipes. We refine the recipes till we get what fulfills our needs.

Let's see how Chef defines its awesomeness:

Keep the previous terms in your mind, but do not be surprised when you start to find much more than these terms during our deployment. We will cover them in a nutshell.

Prerequisites for settings

Before installing the server, we need to set up the right hostname of our CentOS box, where we can define it as an FQDN with a domain suffix:

1. Open the /etc/sysconfig/network file and modify the HOSTNAME value to match your FQDN hostname:

   packtpub@chef$ sudo nano /etc/sysconfig/network
   HOSTNAME=chef.packtpub.com
   

2. Change the host that is associated to your IP address for your server found in the /etc/hosts file, as follows:

   packtpub@chef$ sudo nano /etc/hosts
   192.168.47.10     chef.packtpub.com   chef
   

3. Check your hostname via the following command:

   packtpub@chef$ hostname -f
   chef.packtpub.com
   

4. Make sure that your changes are persistent on reboot:

   packtpub@chef$ sudo /etc/init.d/network restart
   

   Note

   You will need to adjust the settings, such as the hostname, FQDN, and IP address, to suit your needs.

5. As we are using CentOS, it will be much easier for a smooth installation process in order to modify iptables and SELinux. Note that it is not recommended that you entirely disable the iptables service in a production environment where you will have to create extra iptables rules and update your SELinux as well. We will need to allow access to the following ports:

   • TCP ports: 80, 443 for the Chef server web user interface

   • TCP port: 4000 for the Chef server Knife access

   The following commands will update the running iptables rules in your CentOS box:

   packtpub@chef$ sudo iptables -A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
   packtpub@chef$ sudo iptables -A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
   packtpub@chef$ sudo iptables -A INPUT -p tcp --dport 4000 -m state --state NEW,ESTABLISHED -j ACCEPT
   

6. Save the new policy update and restart the iptables service:

   packtpub@chef$ sudo service iptables save
   packtpub@chef$ sudo service iptables restart
   

7. Set SELinux to the permissive mode:

   packtpub@chef$ sudo setenforce 0
   

The Chef server installation

The next setup describes some simple steps to install the Chef server:

1. In your local shell, run the following command to download the Chef server:

   packtpub@chef$ sudo rpm -ivh https://opscode-omnibus  packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.8-1.el6.x86_64.rpm
   

2. Configure the Chef server:

   packtpub@chef$ sudo chef-server-ctl reconfigure
   

3. Check whether the installation was successful or not:

   packtpub@chef$ sudo chef-server-ctl test
   Finished in 0.11742 seconds
   0 examples, 0 failures
   

4. The server should be up and running. You can access the web interface via https://chef.packtpub.com:443:

   

Tip

The Chef server user interface can be accessed using the FQDN edited previously: https://FQDN:443. You can use https://CHEF_IP_ADDR:443, where CHEF_IP_ADDR is the local IP address of your Chef server.

Enter admin as the username and p@ssw0d1 as the default password.

Workstation installation

We will need additionally one or more Chef workstation(s) as a development toolkit. This node's role is as follows:

• The Chef client is installed and Knife is configured

• It holds the local repository for the Chef server

• It installs Chef on the nodes via the knife bootstrap operation

• It dictates nodes, roles, and infrastructure environments to be uploaded to the Chef server

Our Chef workstation will hold the local repo, and then we can install it on our VCS node, where all the development will be performed and then uploaded to the Chef server.

Note

Knife in Chef refers to a tool which provides a Command-Line Interface (CLI) between a Chef server and the Chef repository. It can be installed in a Chef workstation and used to manage nodes, roles, cookbooks, and environments with the Chef server. To read more about Knife in Chef, refer to this link https://docs.chef.io/knife.html.

Let's install our Chef workstation:

1. Get the Chef client installed:

   packtpub@workstation$ curl -L https://www.opscode.com/chef/ install.sh | bash
   

2. Verify the installation:

   packtpub@workstation$ chef-client -v
   

3. Create your chef-repos for a proper format of the Chef repository from GitHub by cloning the structure in /home/packtpub/:

   packtpub@workstation$ git clone https://github.com/ opscode/chef-repo.git
   packtpub@workstation$ cd /home/packtpub 
   packtpub@workstation$ mkdir -p /chef-repo/.chef
   

   In order to authenticate against the Chef server, we will need to add some inputs for the key files from the Chef server web interface. Go to the Clients tab and click on Edit with the chef-validator client.

4. Copy the value of the Private key field. In your workstation, create a new file for the validator key:

   packtpub@workstation$ vi /home/packtpub/chef-repo/.chef/chef-validator.pem
   

5. Paste the content of the copied key and save and close the file.

We perform the same procedure to generate the admin user's key file. In the Users tab, click on Edit, which is associated with the admin user, and check Regenerate Private Key followed by Save User.

After copying the private key, create a new file admin.pem in /home/packtpub/chef-repo/.chef/ again and paste the content of the admin's private key:

1. Create the Knife configuration file:

   packtpub@workstation$ knife configure -i
   
   Overwrite /root/.chef/knife.rb? (Y/N) y
   Please enter the chef server URL: [https://test.example.com:443] 
   https://chef-server.packtpub.com:443/
   Please enter a name for the new user: [root] packtpub
   Please enter the existing admin name: [admin]  Enter
   Please enter the location of the existing admin's private key: [/etc/chefserver/admin.pem] /home/packtpub/chef-repo/.chef/admin.pem
   Please enter the validation clientname: [chef-validator]
   Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /home/packtpub/chef-repo/.chef/chef-validator.pem  
   Please enter the path to a chef repository (or leave blank):   
   Creating initial API user...     
   Please enter a password for the new user
   Created user[pack-knife]      
   Configuration file written to /home/packtpub/chef-repo/.chef/ knife.rb
   

   At the end, you should have the following list of files:

   packtpub@workstation$  ls  /home/packtpub/chef-repo/.chef/
   admin.pem chef-validator.pem knife.rb pack-knife.pem
   

2. Initialize our Git name and e-mail:

         packtpub@workstation$  git config --global user.email                     "masteropenstack@packtpub.com"
         packtpub@workstation$  git config --global user.name        "masteropenstack"
   

3. Clean up the repository by adding the .chef line to .gitignore:

   packtpub@workstation$ nano /home/packtpub/chef repo/.chef/.gitignore
   
   .rake_test_cache
   .chef/*.pem
   .chef/encrypted_data_bag_secret
   .chef
   

4. Add and commit the current repository:

   packtpub@workstation$ git add .
   packtpub@workstation$ git commit -m 'Finish setting up'
   

5. Make sure that you are using the right Ruby version:

   packtpub@workstation$ vi .bash_profile
   packtpub@workstation$ echo 'export PATH="/opt/chef/embedded/bin:$PATH"' >> ~/.bash_profile
   

6. Populate the bash profile settings:

   packtpub@workstation$  source  ~/.bash_profile
   

7. Test our workstation's Chef server connection:

   packtpub@workstation$  knife user list 
   admin
   packt-knife
   

packtpub@workstation$  rvm system

packtpub@workstation$ nano  /home/packtpub/chef-repo/Berksfile
source "https://supermarket.getchef.com"

cookbook 'apache2', '1.9.6'
cookbook 'apt', '2.3.8'
cookbook 'aws', '2.1.1'
cookbook 'build-essential', '1.4.2'
cookbook 'database', '2.2.0'
cookbook 'erlang', '1.4.2'
cookbook 'memcached', '1.7.2'
cookbook 'mysql', '5.4.4'
cookbook 'mysql-chef_gem', '0.0.4'
cookbook 'openssl', '1.1.0'
cookbook 'postgresql', '3.3.4'
cookbook 'python', '1.4.6'
cookbook 'rabbitmq', '3.0.4'
cookbook 'xfs', '1.1.0'
cookbook 'yum', '3.1.4'
cookbook 'selinux', '0.7.2'
cookbook 'yum-epel', '0.3.4'
cookbook 'galera', '0.4.1'
cookbook 'haproxy', '1.6.6'
cookbook 'keepalived', '1.2.0'
cookbook 'statsd', github: 'att-cloud/cookbook-statsd'
cookbook 'openstack-block-storage', github: 'stackforge/cookbook-openstack-block-storage'
cookbook 'openstack-common', github: 'stackforge/cookbook-openstack-common'
cookbook 'openstack-compute', github: 'stackforge/cookbook-openstack-compute'
cookbook 'openstack-dashboard', github: 'stackforge/cookbook-openstack-dashboard'
cookbook 'openstack-identity', github: 'stackforge/cookbook-openstack-identity'
cookbook 'openstack-image', github: 'stackforge/cookbook-openstack-image'
cookbook 'openstack-network', github: 'stackforge/cookbook-openstack-network'
cookbook 'openstack-object-storage', github: 'stackforge/cookbook-openstack-object-storage'
cookbook 'openstack-ops-database', github: 'stackforge/cookbook-openstack-ops-database'
cookbook 'openstack-ops-messaging', github: 'stackforge/cookbook-openstack-ops-messaging'

packtpub@workstation/home/packtpub/chef-repo $ berks install
Resolving cookbook dependencies...
packtpub@workstation/home/packtpub/chef-repo $ berks upload --no-ssl-verify

$ nano roles/packtpub-os-base.json

{
  "name": "packtpub-os-base.json",
  "description": "PacktPub OpenStack Base Role",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {
  },
  "chef_type": "role",
  "run_list": [
    "recipe[openstack-common]",
    "recipe[openstack-common::logging]",
    "recipe[openstack-common::set_endpoints_by_interface]",
    "recipe[openstack-common::sysctl]"
  ],
  "env_run_lists": {
  }
}

{
  "name": "packtpub-os-compute-worker",
  "description": "PacktPub OpenStack Compute Role",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {
  },
  "chef_type": "role",
  "run_list": [
    "role[packtpub-os-base]",
    "recipe[openstack-compute::compute]"
  ],
  "env_run_lists": {
  }
}

packtpub-os-base-controller

{
  "name": "packtpub-os-base-controller",
  "description": "PacktPub OpenStack Controller Role",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {
  },
  "chef_type": "role",
  "run_list": [
    "role[packtpub-os-base]",
    "role[packtpub-os-ops-database]",
    "recipe[openstack-ops-database::openstack-db]",
    "role[packtpub-os-ops-messaging]",
    "role[packtpub-os-identity]",
    "role[packtpub-os-image]",
    "role[packtpub-os-compute-setup]",
    "role[packtpub-os-compute-conductor]",
    "role[packtpub-os-compute-scheduler]",
    "role[packtpub-os-compute-api]",
    "role[packtpub-os-block-storage]",
    "role[packtpub-os-compute-cert]",
    "role[packtpub-os-compute-vncproxy]",
    "role[packtpub-os-dashboard]"
  ],
  "env_run_lists": {
  }
}

packtpub@workstation$  knife role from file /home/packtpub/chef-cookbooks/roles/*.rb

Asymmetric clustering is mostly used for high availability purposes as well as for the scalability of read/write operations in databases, messaging systems, or files.

In such cases, a standby server is involved to take over only if the other server is facing an event of failure. We may call the passive server the sleepy watcher, where it can include the configuration of a failover.

This is where all nodes are active and a participator handles the process of requests. This setup might be cost-effective by serving active applications and users.

A failed node can be discarded from the cluster, while others take over its workload and continue to handle transactions.

Symmetric clustering can be thought to be similar to a load-balancing cluster situation where all nodes share the workload by increasing the performance and scalability of services running in the cloud infrastructure.

The concept of cloud controllers aims to provide one or many kinds of central management and control over your OpenStack deployments. We can, for example, assume that all authentication and messaging transactions are being managed by the cloud controller by means of our magic hub: the message queue.

Considering a medium- or large-scale infrastructure, we will need, with no doubt, more than a single node. For an OpenStack cloud operator, controllers can be thought of as a service aggregator where the majority of running management services are needed for OpenStack to operate.

Let's see what a cloud controller cloud mainly handles:

Most probably, you have already noticed the main services of the cloud controller in Chapter 1, Designing OpenStack Cloud Architecture, but we did not take a deep look at why such services should run in the controller node in the first place. We will now suggest a second alternative.

We bring, for instance, the cloud controller as a node under the scope. This aggregates the most critical services for OpenStack. Thus, we can start by covering them in a nutshell.

Being a proponent of the physical cloud controller, a machine's clustering effort is considered a step in the right direction: high availability. Several HA topologies will be discussed in Chapter 6, OpenStack HA and Failover.

As we have seen the use cases of several services at this point, which can be separated and clustered, we will extend our logical design of the cloud controller described in Chapter 1, Designing OpenStack Cloud Architecture. Keep in mind that OpenStack is a highly configurable platform and the rest of the description is an example that suits a certain requirement and specific conditions.

The next step is to confirm the first logical design. Questions such as this come up: does it satisfy certain requirements? Are all services in the safe HA zone?

Well, note that we include the MySQL Galera cluster to ensure HA for the database. Eventually, this means we are missing something! Depending on the quorum-based system of Galera, at least a third cloud controller has to join the cloud controller team.

Immediately, you may raise a question: should I add an extra cloud controller to make the replication and database HA achieved? What about a fourth or fifth controller?

Great! Keep this mindset for later. At this level, you assume that logically, your design is on the right path and you already know that some changes have to be made to fulfill some physical constraints.

Then, we extend our cluster setup with a third cloud controller:

At this point, we ensure that our design is deployed in HA at an early stage. Remember, there should be no Single Point Of Failure in any layer!

It is important to prepare how the cloud controllers should be clustered in advance. The previous diagram is an example design that scales as well and takes into account more advanced aspects, such as HA failover and load balancing. You can refer to Chapter 6, OpenStack HA and Failover, to check out more details and practical examples. For instance, the overall OpenStack cloud should expand easily by joining new nodes running several services that require more care. We continue later by adumbrating an automated approach to facilitating the horizontal expansion of the cloud.

Cooking the cloud controller

Once we have identified which service will be deployed in the cloud controller, we jump to the next step by bringing our Chef into action. We have already covered a general overview of the OpenStack cookbooks, which we have based on the Chef community website.

As we are aiming for a large-scale infrastructure, we would rather prepare the roles and recipes and make them more decoupled for service nodes to reach a level of high availability in the second stage. The cookbook design of the cloud controller seems quite complicated, which implies that its implementation might not be intuitively obvious at first glance, but a brief overview of the cookbooks' relationships will make it easier for you to highlight the flexibility of this model. Thus, you may intend to choose on your own how to distribute roles and recipes by maintaining the logic of dependency.

As you may notice in the next figure, we have gathered the majority of services in the cloud controller, except object storage and compute workers. On the other hand, assigned roles and recipes can be detached and reassigned to other nodes. We bring in the cloud controller for it to be deployed first in order to check our cookbooks' consistency.

Keeping in our mindset and whatever system management tools we might choose, underpinning every service component on our OpenStack cloud platform must be a flexible mantra, as much as possible, for the purpose of our first cloud controller deployment. Based on Chapter 2, Deploying OpenStack – DevOps and OpenStack Dual Deal, we have covered how to turn the code of our infrastructure into pieces by means of recipes, while gathering the pieces for a more customized design will form the roles.

This is described in the following cookbook diagram:

You may notice that roles can include not only recipes, but also other roles.

We can go through each role and describe how it is composed in a nutshell by taking a look at the following table:

Role name	Default recipe	Description
packtpub-os-base	openstack-common openstack-common::logging openstack-common::set_endpoints_by_interface openstack-common::sysctl	openstack-common is a set of recipes and attributes describing general OpenStack deployment openstack-common::logging installs and configures common logging attributes openstack-common::set_endpoints_by_interface iterates over the endpoints per node hash and finds any occurrence of bind_interface to set the IP address openstack-common::sysctl iterates over a node hash and updates its entries to /etc/sysctl.d/60-openstack.conf
packtpub-os-ops-database	openstack-ops-database::server openstack-ops-database::openstack-db	openstack-ops-database::server selects the database server configuration using attributes openstack-ops-database::openstack-db defines the required tables and users for OpenStack
packtpub-os-identity	openstack-identity::server openstack-identity::registration	openstack-identity::server installs and configures Keystone services openstack-identity::registration registers identity endpoint and service
packtpub-os-ops-messaging	openstack-ops-messaging::server	This installs a single RabbitMQ server instance
packtpub-os-compute-scheduler	openstack-compute::scheduler	This installs and configures a single instance of nova-scheduler
packtpub-os-compute-conductor	openstack-compute::conductor	This installs and configures a single instance of nova-conductor
packtpub-os-compute-cert	openstack-compute::nova-cert	This installs and configures a single instance of nova-cert
packtpub-os-compute-api	openstack-compute::identity-registration	This registers the identity endpoint for the Nova service
packtpub-os-compute-vncproxy	openstack-compute::vncproxy	This installs and configures a single instance of the Nova VNC service
packtpub-os-image	openstack-image::identity_registration	This registers the identity endpoint for the Glance service
packtpub-os-compute-dashboard	openstack-dashboard::server	This installs and configures a single instance of Horizon
packtpub-os-block-storage	openstack-block-storage::identity_registration	This registers the identity endpoint for the Swift service
packtpub-os-network	openstack-network::identity_registration	This registers the identity endpoint for the Neutron service

Note

It is possible to customize the automation of the cloud controller based on roles and recipes defined in the default cookbooks provided by StackForge, covered in Chapter 2, Deploying OpenStack – DevOps and OpenStack Dual Deal. The main StackForge Chef repository can be found on GitHub at https://github.com/stackforge/openstack-chef-repo.

Once the orchestrator has evaluated the instruments that should be integrated on the stage, we still need the players to accomplish the song. All we need are worker horses where our virtual machines' brains will live. Notice that the brain of this instance refers to where all the thinking processes are done.

The compute node should be separately deployed in the cluster mode as it forms the resources part of the OpenStack infrastructure. Even in another cloud deployment architecture, you may find that the computing part is mostly built in separate farms. It is imperative to give attention to the fact that compute node resources should not be overlooked in processing, memory, network, and storage resources.

From a deployment perspective, an OpenStack compute node might not be complicated to install as it will basically run nova-compute and the network agent for Neutron. However, its hardware and specification choice might not be obvious. The cloud controller presents a wide range of services, but we have agreed that using HA and a separate deployment will crystallize the cloud controller deployment. This way, we suffer less from the issue of service downtime. On the other hand, a compute node will be the space where the virtual machine will run, in other words, the space on which the end user will focus on. They only want to push the button and get the application running on the top of your IaaS layer. It is your mission to guarantee a satisfactory amount of resources.

A good design of cloud controller is needed but is not enough; we need to take care over compute nodes as well: compute resources.

Deciding on the hypervisor

The hypervisor is the heart engine of your OpenStack compute node. This is called the virtual machine monitor (VMM), which provides a set of manageability functions for virtual machines to access the hardware layer. The amazing part about hypervisors in OpenStack is the wide range of VMMs that it can offer, including KVM, VMware ESXi, QEMU, UML, Xen, Hyper-V, LXC, bare metal, and lately, Docker.

If you already have some experience with one or more of these, it will be better to take a look at how they differ at an architectural level. Currently, the last OpenStack release at the time of writing this book was Juno, which has many hypervisor features added or extended. Keep in mind that not all of these support the same features. The Hypervisor Support Matrix (https://wiki.openstack.org/wiki/HypervisorSupportMatrix) is a good reference that can help you to choose what fits your needs.

Obviously, the former hypervisors are not the same, based on their nature and use cases. For example, Quick EMUlator (QEMU) and User Mode Linux (UML) might be used for general development purposes, while Xen requires a nova-compute installation on a paravirtualized platform.

Note

Paravirtualization is an improvement of virtualization technology in which the guest operating system is compiled prior to installation in a virtual machine. Xen and IBM have adopted this technology keeping in mind the high-performance deliverance that it can provide. The operating system and the hypervisor work efficiently in tandem, which helps avoid the overheads imposed by the native system resource emulation.

Most probably, you have heard about most of these previously mentioned hypervisors, but what do you think Docker could be?

It is interesting to discover another attractive point about OpenStack, which has steadily grown and can include any virtual technology in its ecosystem, such as the Docker driver for OpenStack nova-compute.

Out of the box, Docker helps enterprises deploy their applications in highly portable and self-sufficient containers, independent of the hardware and hosting provider. It brings the software deployment in to a secure, automated, and repeatable environment. What makes Docker special is its usage of the terms of several containers, which can be managed on a single machine. Additionally, it becomes more powerful when it is used alongside Nova. Therefore, it would be possible to manage hundreds and even thousands of containers, which makes it the cat's meow. You may wonder about the use cases of Docker, especially in an OpenStack environment. Well, as mentioned previously, Docker is based on containers that are not a replacement for virtual machines, but which are very specific to certain deployments. Containers are very lightweight and fast, which may be a good option for the development of new applications and even to port older application faster. Imagine a virtual machine abstraction that can be shared with any application along with its own specific configuration requirements without them interfering with each other. Docker can do this, but in terms of containers where applications run natively on the Linux kernel and each kernel is segmented from one another to form the operating system. Uniquely, it might be possible to save the state of a container as an image that can be shared though a central image registry. This makes Docker awesome as it creates a portable image across infrastructures and reveals the barrier of building bridges between different clouds, in other words, hybrid clouds.

As this is an introduction to the Havana release, Docker is going to be an important tool for OpenStack, which might stand beside virtual machines in an OpenStack environment.

Note

To read more about Docker, check the following reference: https://www.docker.com/whatisdocker/. The Docker driver documentation for OpenStack can be found here: http://docs.openstack.org/havana/config-reference/content/docker.html.

On the other hand, most OpenStack nova-compute deployments run KVM as the main hypervisor. The fact is that KVM is best suited for workloads that are natively stateless using libvirt.

KVM is the default hypervisor for OpenStack Compute. You can check out your compute node from /etc/nova/nova.conf in the following lines:

compute_driver=libvirt.LibvirtDriver
libvirt_type=kvm

For proper, error-free hypervisor usage, it might be required to first check whether KVM modules are loaded from your compute node:

# lsmod | grep kvm
 kvm_intel or kvm_amd

Otherwise, you may load the required modules via:

# modprobe -a kvm

To make your modules persistent at reboot, which is obviously needed, you can add the following lines to the /etc/modules file when your compute node is an Intel-based processor:

kvm
kvm-intel

Note that kvm-intel can be replaced by kvm-amd in the case of an AMD-based processor.

Our further compute deployments will be based on KVM.

Changing the color of the hypervisor

While we have decided to use KVM for nova-compute, it would be great to check how OpenStack could support this wide range of hypervisors by means of nova-compute drivers. You might be suggested to run your OpenStack environment with two or more hypervisors. It can be a user requirement to choose a typical hypervisor in order to use its native one. This will help the end user resolve the challenge of native platform compatibility, and then we can calibrate the usage of the virtual machine between environments. This would be a great topic in hybrid cloud environment.

The next figure depicts the integration between nova-compute and KVM, QEMU, and LXC by means of libvirt tools and XCP through APIs. On the other hand, vSphere, Xen, or Hyper-V can be managed directly via nova-compute.

Let s take an example and see how such wonderful multihypervisor capability can be factored in to your OpenStack environment. If you already have a VMware vSphere running in your infrastructure, this example will be suitable for you if you plan to integrate vSphere with OpenStack. Practically, the term integration on the hypervisor level refers to the OpenStack driver that will be provided to manage vSphere by nova-compute. Eventually, OpenStack exposes two compute drivers that have been coded:

• vmwareapi.VMwareESXDriver: This allows nova-compute to reach the ESXi host by means of the vSphere SDK

• vmwareapi.VMwareVCDriver: This allows nova-compute to manage multiple clusters by means of a single VMware vCenter server

Imagine the several functions we will gain from such an integration using the OpenStack driver with which we attempt to harness advanced capabilities, such as vMotion, high availability, and Dynamic Resource Scheduler (DRS). It is important to understand how such an integration can do the magic.

Note

vMotion is a component of VMware vSphere that allows the live migration of a running virtual machine from one host to another with no downtime. VMware's vSphere virtualization suite also provides a load-balancing utility called DRS, which moves computing workloads to available hardware resources.

In a vSphere implementation coupled with OpenStack, nova-scheduler will assume each cluster as a single compute node that has the aggregate of resources of all ESXi hosts managed by that cluster, as shown in the previous figure.

A good practice retrieved from this layout implementation is to place the compute node in a separate management vSphere cluster so that nodes that run nova-compute can take advantage of vSphere HA and DRS. vCenter can be managed by the OpenStack compute nodes only if a management vSphere cluster is created outside the OpenStack cluster.

Tip

One common use case for host aggregates is when you want to support scheduling instances to a subset of compute hosts because they have a specific capability.

Our previous example can be thought of as the following if we seek a heterogeneous hypervisor deployment in an OpenStack installation using KVM and vSphere ESXi.

It is important to guarantee that particular VMs are spun up on their specific vSphere cluster, which exposes more hardware requirements. To do this, OpenStack facilitates such requirements by means of host aggregates. They are used with nova-scheduler in order to place VMs on a subset of compute nodes based on their rank capabilities in an automated fashion.

A brief example can be conducted with the following steps:

1. Create a new host aggregate; this can be done through Horizon. Then, select Admin project. Point to the Admin tab and open System Panel. Click on the Host Aggregates category and create new host named vSphere-Cluster_01.

2. Assign the compute nodes managing the vSphere clusters within the newly created host aggregate.

3. Create a new instance flavor and name it vSphere.extra with particular VM resource specifications.

4. Map the new flavor to the vSphere host aggregate.

This is amazing because any user requesting an instance with the vSphere.extra flavor will be forwarded only to the compute nodes in the vSphere-Cluster_01 host aggregate.

Therefore, it will be up to vCenter to decide which ESXi server should host the virtual machine.

At this point, we consider that running multiple hypervisors in a single OpenStack installation is possible using host aggregates or the terminology of cells. Then, if you factor in hypervisors' varieties, do not get confused by the fact that a single hypervisor is running with an individual compute node.

Eventually, the previous figure might consider that the VM instances running on KVM can be hosted directly on a nova-compute node, whereas the vSphere with vCenter on OpenStack requires a separate vCenter server host where the VM instances will be hosted on ESXi.

Cooking the compute node

Deploying the compute node via Chef is much simpler than understanding the resource requirements needed for a node. Basically, the compute node will run nova-compute together with the networking plugin agent. What you should understand at this stage of automated deployment is how to conduct your controller to communicate with the compute node when you run Chef. You got it: create a correct network mapping in your environment file. Let's refresh our memory about the compute spot against the controller:

We have already defined a role in the cookbook that automates the installation of the cloud controller instances in The cloud controller section. We will do this for the compute node as well by defining a new role named packtpub-os-compute.

The next cookbook design will highlight a complete independent compute node setup regardless of the presence of the cloud controller in the environment. Thus, the design might be tempted to show all dependencies as the compute node will be deployed out of the box. As was claimed in our Chef cloud controller installation, many roles can use other recipes within a given role. The same aspect applies to our compute node. Basically, a compute node that runs nova-compute will depend on the image, identity, and network services besides the common services and attributes that describe the OpenStack environment, such as endpoint mapping. If you intend, for example, to start the deployment of the compute node for the first time, its cookbooks must be uploaded. On the other hand, you can create a berks file that defines the list of required cookbook dependencies in your compute node cookbook. Do not be surprised if you find some cookbooks from the Chef community, which may include the same dependencies. This indicates good design as it is considered as a nonmonolithic block with which you can deploy services independently but by sharing the same dependencies. We have our primary cookbooks already uploaded. Adding new ones will depend on the existing ones if you would like to customize your OpenStack deployment. In addition, any updates to recipes will be taken into consideration by Chef. Chef is smart enough to claim that it is already added and there is no need to upload it again, but just to use it. This is another reusability aspect of Chef deployments. You may feel the difference between the first deployment in Chef and the subsequent ones from the perspectives of speed and fewer errors. The Chef compute node cookbook design may look like the following:

Tip

The open vSwitch agent service can also run optionally on compute nodes. In this way, scalability of open vSwitch is achieved in case of compute node failure.

The new packtpub-os-compute Chef role can be defined as the following:

name "packtpub-os-compute"
description "PacktPub scalable compute node role"
run_list(
  "role[packtpub-os-compute-worker]",
  "role[packtpub-os-network-openvswitch]"
) 

To upload the new role to your Chef environment, run the following command from the Chef workstation using the Knife command line:

packtpub@workstation$  knife role from file /home/packtpub/chef-cookbooks/roles/packtpub-os-compute.json

One of the most critical tasks for a system administrator or cloud operator is to plan a backup. Building infrastructure and starting in production without a disaster recovery background is considered highly risky and you will need to start taking immediate actions. We may find a bunch of property software in the cloud computing area that does the job, such as the VMware backup solution.

However, backing up open source clouds will not be that easy. OpenStack does not, for instance, support any special tool for backup. As it is merely a collection of components combined to deliver services, an OpenStack operator should think how to map the components used in its infrastructure and prepare a backup strategy for each; the strategy should be easy, efficient, and autorecovery-enabled.

Thus, you should not miss the first question: what do we need to back up and how do we perform such a mission?

At first glance, you might be tempted to think that backing up the cloud controller will be centered on configuration files and databases.

packtpub@cc01$ sudo rpm -Uvh http://mirrors.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm.

packtpub@cc01$ sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

packtpub@cc01$ sudo yum install backup-manager

export BM_TARBALL_DIRECTORIES="/var/lib/nova /etc/keystone  /etc/cinder /etc/glance /var/lib/glance /var/lib/glance/images  /etc/mysql"

export BM_ARCHIVE_METHOD="tarball mysql"

export BM_REPOSITORY_ROOT="/var/backups/"

export BM_MYSQL_FILETYPE="gzip"

export BM_UPLOAD_SSH_USER="root"

export BM_MYSQL_DATABASES="nova glance keystone dash mysql cinder"
export BM_MYSQL_ADMINPASS="Define the root password in /root/.my.cnf"

For the sake of simplicity, we will start with the nonpersistent storage, which is called ephemeral storage. As its name suggests, a user who actively uses a virtual machine in the OpenStack environment will lose the associated disks once the VM is terminated.

Persistent storage means that the storage resource is always available. Powering off the virtual machine does not affect the data. We can divide it into two persistent storage options in OpenStack—object and block storage with the code names Swift and Cinder, respectively. We did talk about Swift and Cinder in Chapter 1, Designing OpenStack Cloud Architecture, in a nutshell. Let's dive into each storage OpenStack-aware and see how the two different concepts are used to dump different purposes.

Swift was one of the first OpenStack projects. It was developed by NASA and Rackspace, and the former contributed towards the project by developing the code of the block storage of the OpenStack ecosystem. A few major changes to the storage came about in a very short span of time.

Firstly, the emergence of web and mobile applications fundamentally changed data consumption. The second major change was introduced in the Software Defined Storage (SDS), which enables a large distributed storage system to be built by a basic commodity storage. This dramatically reduces the cost of deploying data into an application as the individual component is not reliable.

Swift is an object storage system. This means that it treats immediate consistency before eventual consistency. This allowed Swift to gain HA, redundancy, throughput, and capacity.

By adopting Swift as a cloud storage solution, you can enjoy several benefits, some of which are as follows:

Physical design considerations

The hallmark of Swift usage is that it requires you to look after your data durability and availability. By default, a Swift cluster storage design considers a replica of three.

Therefore, once the data is written on a replica, it is spread across two other replicas, which increases the availability of your data on one hand. On the other hand, you will need more storage capacity. In addition, referring to the first logical design in Chapter 1, Designing OpenStack Cloud Architecture, we have dedicated a network for storage.

That was by purpose firstly for logical network design organization and secondly to mitigate the load on the network by dedicating a separate storage handler. Imagine a situation where one of the storage nodes with 50 TB fails when you need to transfer this huge blob of data remotely to accomplish the required three-replica design. It can take a few hours, but we need it immediately! Thus, take into account the bandwidth precisely between your storage servers and proxies. This is a good reason to put the spotlight on the physical design and the way the data is organized in Swift.

In the first stage, we saw that the accounts, containers, and objects form the term data in Swift, which will need physical storage. In this stage, the storage node will be constructed first. Remember that Swift aims to isolate failures, which makes the cluster wider in terms of grouping according to the nodes. Thus, Swift defines a new hierarchy that helps you abstract the logical organization of data from the physical one:

• Region: Being in a geographically distributed environment, data can be held in multiple nodes that are placed in different regions. This is the case with a multi-region cluster (MRC). A user can suffer due to higher latency that comes with the different servers being placed away from each other in each region. To do so, Swift supports a performance read/write function called read/write affinity. Based on the latency measurements between the connections, Swift will favor the data that is closer to read. On the other hand, it will try to write data locally to transfer the data to the rest of the regions asynchronously.

• Zone: Regions encapsulate zones, which define the availability level that Swift aims to provide. A grouping or a set of hardware items, such as a rack or storage node, can refer to a zone. You can guess the rest—zoning to isolate hardware failure from the other neighbors.

  Tip

  It is recommended to use five zones and start with at least one zone in a given cluster.

• Storage nodes: The logical organization continues the storage abstraction from the region which is the highest level, zones within region until we find the storage servers which define the zone. A set of storage nodes forms a cluster that runs the Swift processes and stores an account, a container, the object data, and its associated metadata. What makes Swift unique is a special storage organizer aware is possibly used to define how your set of nodes would be grouped by criteria.

• Storage criteria: Depending on how the zones are set within the available regions, Swift allows us to customize the way you wish to distribute data across a single region or multiple regions on specific storage hardware or a defined replica cluster.

• Storage device: This is the smallest grain of the Swift abstraction data classification. The storage device can be the internal storage node's device or connected via an external stack of a collection of disks in a drive enclosure.

Note

The drives that are used in Swift can be set in a Just a Bunch of Disks (JBOD) regardless of the configuration and can be accessed from the host computer as a separate drive unlike RAID, which treats a collection of drives as a single storage unit.

The following figure shows the hierarchy in Swift:

Where is my data?

Ultimately, considering an MRC and looking for some sample data across a bunch of storage servers fires up a pertinent question: how could Swift do that?

Whether the request was to read or write, the Swift servers need to map the data names to physical locations, which are called rings. We can summarize the concept of the rings as follows:

• Assign accounts, containers, and objects in separate rings

• Logical partition of the storage device in the ring

• Update the ring by redistributing the partitions in case or add/remove a device to/from the cluster respectively

Tip

It is recommended to use 100 partitions for each device per zone.

Practically, a ring is a bunch of tables that are distributed to every node in the cluster. So, why do these tables exist everywhere? The answer is simple. This is because Swift replicates data everywhere!

Note

There are various rings present in a cluster. When a process needs to find an account, a container, or an object, it first looks for the data in all the locations on every separate ring.

Does this not make sense? When a process needs to find some account-related data, it first starts looking in a local copy of the rings, which points to all the locations on the account ring for the data. For example, the rings in Swift use the hash functions to determine how to retrieve or store an object. When using several drives in a multiregion Swift environment, complicated hashing functions can be used to accomplish such data location.

For example, a simple method to determine where to store an object can use an MD5 algorithm by getting the hash of the object storage location in an account server, as follows:

md5 ("/account_server01/container01/objectID") =  f46aaa8067cbeb944b547a0fbc3012a2

The ring will define the MD5 hash to a hexadecimal representation, which will give a value of 654853167495245315274945238545002450045.

Next, we'll proceed with a modulo operation by dividing its value by the available number of drives. Assuming that we have three drives, it might give the following result:

332115198597019796159838990710599741918 % 3 = 2

The remainder of the former division will map the drive ID, which is 2.

Swift uses the ring-builder tool to create builder files by account/container/object storage that contains information such as the replica count, partition power, and the location of the storage drives within the cluster.

Note

The total number of partitions that exist in your cluster can be obtained by using the following partition power formula:

Total_partitons_per_cluster = 2 [partition power]

Here, the partition power is a random integer.

The following figure shows the ring mechanism:

Let's sum up our understanding of Swift in a real write-show example, as illustrated in the previous figure:

1. Get the list of drives by the proxy servers from the ring.

2. Associate the objects to write the data.

3. Devices acknowledge for the ability to perform write operations.

50 * 5 replicas = 250 TB

250 TB * 1.0526 = 263 TB

[263 / 2.5] = 105.2 => 106 drives

106/30 = 3.533333 => 4 nodes

(27 drives * 3/4 (core.GHz/drive))/2 GHz = 10.125 cores 

The Swift network

Our first network design assumes that an additional network is dedicated for the storage system. In fact, we should remind ourselves that we are talking about a large infrastructure. More precisely, Swift is becoming a big house with small rooms in our OpenStack deployment. However, Cinder can still provide a big room in a fairly small house.

For this reason, we will extend the Swift network by deriving more subnets, as follows:

• The front-cluster network: Proxy servers handle communication with the external clients over this network. Besides, it forwards the traffic for the external API access of the cluster.

• The storage cluster network: It allows communication between the storage nodes and proxies as well as inter-node communication across several racks in the same region.

• The replication network: We do care about the development of our infrastructure size, right? Therefore, we will plan for the same for the multiregion clusters, where we dedicate a network segment for replication-related communication between the storage nodes.

The Swift network is shown in the following figure:

Now, we will talk about block storage. We made a small comparison between Swift and Cinder in Chapter 1, Designing OpenStack Cloud Architecture. Since we are building the infrastructure, we need to decide on the best outfit storage. Without doubt, we have seen that Cinder is fully integrated into OpenStack Compute, where users are able to manage their own storage needs by managing the volumes and the associated snapshots of these volumes.

It is imperative to check the use case of Cinder in our storage design. Like object storage, block storage is mainly a tool for persistent storage. Under the hood, volumes expose a raw block of storage that can be attached to instances and which can store data permanently. On the other hand, Cinder manages snapshots. Keep in mind that the former is a point-in-time copy of a volume, whereas you might be able to make fast and temporary backups by fully copying a volume's data and storing the same in the backup system. However, the concept of the snapshot can be misunderstood when you rely on it purely for long-term backup purposes.

Fundamentally, block storage becomes an essential requirement for virtual infrastructure within OpenStack that is in favor of ephemeral storage. We should be glad that Cinder provides a block device that uses iSCSI, NFS, and Fiber Channel. Alternatively, we can even make it compatible with some other vendor backend storage connectivity. Moreover, Cinder helps you manage the quotas by limiting the tenant's usage. You can limit the quota usage by total storage utilized including snapshots, total of volumes available, or total number of snapshots taken. The following example shows the current default quota for the packtpub_tenant tenant by using the following command line:

# cinder quota-defaults packtpub_tenant
+-----------+-------+
|  Property | Value |
+-----------+-------+
| gigabytes |  1000 |
| snapshots |   50  |
|  volumes  |   50  |
+-----------+-------+

The limiting of the quotas for the packtpub tenant can be done in the following way:

# cinder quota-update --volumes 20 packtpub_tenant
# cinder quota-update --gigabytes 500 packtpub_tenant
# cinder quota-update --snapshots 20 packtpub_tenant
# cinder quota-show packtpub_tenant
+-----------+-------+
|  Property | Value |
+-----------+-------+
| gigabytes |  500 |
| snapshots |   20  |
|  volumes  |   20  |
+-----------+-------+

OpenStack is designed to facilitate the integration of several existing storage architectures in the enterprise. In fact, you may have noticed that we escalated such choices to resolve the CAP theorem.

Eric Brewer of UC Berkeley proposed a theory in 2000, which states the impossibility of a distributed system to guarantee that the following three important points will be implemented:

Therefore, such a choice aspires to bring the challenge that a storage system might face to a great CAP's narrowed implication opportunity.

Swift belongs to the AP class. We have seen how it was architecturally designed to provide HA and be a good option for partition tolerance by cluster zoning.

On the other hand, although Swift and Cinder store slightly different OpenStack data, the volumes of the virtual machines are very critical in the first place. They need a very high performance bias along with a consistency level, which is something that Cinder is good at. Thus, we will not take the risk and wait for a scenario where a write operation to all the nodes in your cloud storage are not reflected simultaneously.

Once validated, the topology for any system design will go through a hardware planning phase. For Swift, we will talk about object storage. We are highly redundant to respect the native requirements of its architecture. Let's examine an example of hardware selection.

Cinder can do more

By relying on Swift to manage object storage on commodity servers instead of specialized vendor hardware, we gain a lot of flexibility at a low cost. However, when we arrive to spend some time on our block storage, we face a few other options.

Block storage differs from object storage with regard to the consistency. In a cloud environment, where machines depend on their volumes to run, it might be obvious to treat this case in a different way. If you already have a special vendor storage solution deployed in your infrastructure, you can change the way of starting from scratch, which might not only be time consuming, but also expensive. The awesome thing about Cinder in OpenStack is that it supports many storage array suppliers. The former exposes block storage by means of Cinder drivers, such as Dell, Hitachi, IBM, VMware, HP, NetApp, and so on.

Note

You can check the Cinder support matrix at https://wiki.openstack.org/wiki/CinderSupportMatrix.

Cinder provides the available block storage driver support by the vendor product. The functions that are enabled by OpenStack release code names. Note that most of the suppliers provide support for protocols such as iSCSI in the first place then Fiber Channel and NFS. In our case, we will deploy block storage with the EMC plugin. As shown in the next figure, you will need an EMC Storage Management Initiative Specification (SMI-S) server to initiate the (CMI) clients operation over HTTP in the backend.

In our case, the integration of an SMI-S server is very useful if we wish to provide a common point to manage the heterogeneous storage devices in our OpenStack environment. Starting with the EMC storage, for example, you will be able to manage the additional SMI-S-enabled storage property from a unique web-based console instead of rushing between each vendor-native management interface.

Note

CMI stands for Clariion Message Interface and is used for communication between the storage processors.

You will need to create a thin PacktPub OpenStack storage pool.

On your SMI-S server, you will need to install some Python dependency packets, which can be done in the following way:

# yum install -y libgcc_s.so.1 glibc.i686 *pywbem* compat-libstdc++-33.x86_64  libstdc++-devel-*

From the EMC website, get the SMI-S install package and install it as follows:

   # tar -xvf se7628-Linux-i386-SMI.tar
   # ./ se7628_install.sh -install -host

Deploy the SMI-S server and configure the storage array in the following way:

  # cd /opt/emc/ECIM/ECOM/bin/
  #  ./TestSmiProvider 
     (localhost:5988) ? addsys
     Add System {y|n} [n]: y
     ArrayType (1=Clar, 2=Symm) [1]: 
     One or more IP address or Hostname or Array ID
     Elements for Addresses
     IP address or hostname or array id 0 (blank to quit): 192.168.1.102
     IP address or hostname or array id 1 (blank to quit): 192.168.1.103
     IP address or hostname or array id 2 (blank to quit): 
     Address types corresponding to addresses specified above.
     (1=URL, 2=IP/Nodename, 3=Array ID) 
     Address Type (0) [default=2]: 
     Address Type (1) [default=2]: 
     User [null]: adminpack 
     Password [null]: adminpack
     ++++ EMCAddSystem ++++

On the Cinder node, check whether you have the following package installed:

# yum install -y libgcc_s.so.1 glibc.i686 *pywbem* compat-libstdc++-33.x86_64  libstdc++-devel-*

Do not forget to tell Cinder about its backend by editing the /etc/cinder/cinder.conf file:

       iscsi_target_prefix = iqn.1992-04.com.emc
       iscsi_ip_address = 192.168.1.104
       volume_driver =  cinder.volume.drivers.emc.emc_smis_iscsi.EMCSMISISCSIDriver
       cinder_emc_config_file = /etc/cinder/cinder_emc_config.xml

Also, we need to tell it which storage pool and which array to use with the following commands:

# touch  /etc/cinder/cinder_emc_config.xml
# Edit  /etc/cinder/cinder_emc_config.xml

Append the following configuration to the XML file:

      <?xml version='1.0' encoding='UTF-8'?>
      <EMC>
      <StorageType>OpenStack</StorageType>
      <EcomServerIp>192.168.1.110</EcomServerIp>
      <EcomServerPort>5985</EcomServerPort>
      <EcomUserName>admin</EcomUserName>
      <EcomPassword>adminpass</EcomPassword>
      </EMC>

After restarting the Cinder service, try to test it as follows:

#   cinder create --display-name packtpub01  5
#   cinder list

For the preceding code, we will get the following output:

The Cinder use case

Managing the different storage pools from one centralized management interface makes Cinder send only the volume management requests to your existing storage system. At this point, you should realize how OpenStack is open to seamlessly integrating the existing pieces in your infrastructure without you having to go through a nightmare when you wish to deploy what you exactly need.

Obviously, you may have a running OpenStack storage with one or multiple backends where Cinder stands happily. However, there are some limitations that you must take into consideration. As a system designer, you may come across different knobs that you might have to twiddle around with in a distributed storage environment. It starts when you move to the production. A database administrator may suddenly discover that its Red Hat box has almost reached 95 percent at home partition. You don't have the time to book a flight and go to Singapore to add a new storage array to the ESX server in the data center, create a new virtual disk, and attach it through the vSphere client. Even worse, it is Christmas! The trading server will expect a peak load the day after, where the database size will increase by gigabytes and you can't go offline. You have an evening to handle the situation and then join the Christmas dinner!

This kind of situation puts a system administrator under tons of pressure, where everybody expects to hear things such as: it will work! Between the words, a lot of words! Stop blaming your monitoring system, which does not send such notifications on time, and look at the situation from a different perspective. Realize that virtualization can remove the limits of hardware access for the endpoint machines, where the cloud computing paradigm just uses it to give a hand and give exactly what you need without wasting resources. You will need to add a new disk and your current case to extend an existing home partition. For example, traditionally creating a new /dev/sda5 primary partition and assigning it to your home partition via LVM will resolve the issue in a few minutes. In a virtual environment, a precondition needs to be satisfied first, which is the price that you have to pay to derive benefits from the cloud technology. For example, if you intend to extend the virtual disk size from your vSphere client while the machine is running, you will need to check whether it is thin provisioned. In case it is thick provisioned, you will need to reboot the machine after resizing the disk in the right partition. For this reason, storage in production should be carefully handled and managed by keeping a margin of surprises that might happen.

In our deployment scenario, it is worth differentiating the types of volume that are provisioned in OpenStack by using Cinder and VNX as a backend, as follows:

• Thin provisioning: In this, the volume is virtually provisioned and can be allocated as needed.

• Thick provisioning: Here, the volume is allocated during the volume creation and is fully provisioned.

• Deduplicated provisioning: Here, the volumes are virtually provisioned and made deduplication-aware. In this case, the storing of volumes in the VNX devices will be done in a more efficient way by eliminating the duplicated segments in the incoming data and storing only the unique one.

• Compressed provisioning: In this, the volumes are virtually compressed and made compression-aware. In this case, the block storage devices may gain more capacity with better, efficient usage by freeing up a greater amount of valuable storage space with lower performance overheads. The compressed provisioning applies to all the volumes, unlike the deduplication provisioning.

The next example depicts how to create a thin volume named ThinVol01 with the storagetype: provisioning=thin spec value. Keep in mind that without specifying the volume type, the driver will create a thick one by default.

$ cinder type-create "ThinVolume"
$ cinder type-key "ThinVolume" set storagetype:provisioning=thin

The preceding command will produce the following output:

Depending on the driver configured against OpenStack, you will be requested to check each approach for what sort of limitations you will face during the production phase. For example, you will not be able to extend a thick volume, which is associated with a snapshot. It is very important to know in advance what kind of volume you are using in order to avoid an error state. On the other hand, the matrix that was cited earlier is very useful to bear in mind all the storage management functionalities. Some of them are not supported directly by Cinder, but you can use them from the native backend resource storage management.

Beyond Cinder – Ceph

Look carefully at the matrix mentioned previously, and you will find Ceph! It is not just a driver that has to be installed and configured as a backend for Cinder. It is more of a standard open source distributed storage. Ceph can be used for object storage through its S3 API as well as the Swift API. If you intend to gather all the pieces from the object and network block devices, you should consider Ceph. Moreover, it is being developed to expose the filesystem interface, which is on the way towards receiving support from the production. The concept of Ceph as a scalable storage solution is almost the same as Swift that replicates data across the commodity storage nodes. Do you think that is all? Of course not. Ceph is a good data consolidator that enables you to grab both the object and block storages in a single system. You can even use it as a backend to glance at images. If it is agreed that Cinder is still recommended in our block storage solution as we need its API, will you go for Ceph rather than Swift for the object storage backend? Well, this will be a difficult question to answer if you do not verify how Ceph is being architected in a nutshell.

Here's the architecture of Ceph:

The main core of Ceph is the Reliable Autonomic Distributed Object Store (RADOS), which is responsible for the distribution and replication of objects across the storage cluster. As illustrated in the previous figure, a block storage layer provides a RADOS Block Device (RBD) for the object's backend. The amazing part in this architecture is that the RBD devices are thinly provisioned within the RADOS objects and thanks to the librbd library, objects can be accessed by means of QEMU drivers, which make the magical link between Ceph and OpenStack. Unlike Swift, Ceph defines other basic components as follows:

• Object Storage Devices (OSDs): This corresponds to the physical disks, which can be a directory residing on a regular filesystem, such as XFS or Btrfs. OSDs run the OSD daemon for the RADOS service, which will take care of the replication, coherency, and recovery of objects.

  Note

  A Linux filesystem such as XFS or ext4 is required for the Ceph production environment, but Btrfs hasn't been proven to be a stable filesystem that is suitable for a production environment. Refer to the official Ceph website, http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/, for recommendation-related updates.

• Placement groups (PGs): A PG helps you map OSDs for performance and scalability reasons. It performs object replication by the pool as well. Every PG that is assigned in a pool will replicate the object into multiple OSDs within the same pool.

• Pool: You can compare a pool in Ceph to the concept of rings in Swift. It defines the number of PGs that are not shared. Furthermore, it provides hash maps for objects in OSDs.

• The CRUSH maps: Based on the defined criteria, a CRUSH algorithm defines how objects in OSDs will be distributed. Its main purpose is to ensure that the replicated objects will not end up on the same disks, hosts, or shelves. Besides OSD, Ceph introduces the following servers:

  • The monitor daemon server (MON) mainly focuses on checking the state of consistency of the data in each node that runs an OSD

  • The metadata server (MDS) is required for the Ceph filesystem to store their metadata if you intend to build a POSIX file on top of objects

Ceph can be integrated seamlessly with OpenStack. It has emerged as a reliable and robust storage backend for OpenStack that defines a new way of provisioning the boot-from-volume instances. This new method of provisioning is named thin provisioning. Eventually, Ceph compromises on a nice concept, the copy-on-write cloning feature, allowing many VMs to start instantly from the templates. This shows a great improvement at the threading level along with an amazing I/O performance boost.

Thousands of VMs can be created from a single master image derived from a Glance image stored in a Ceph block device and booted by using Cinder, which requires only the space needed for their subsequent changes:

Note

To boot the virtual machines in Ceph either from an ephemeral backend or from a volume, you must use a RAW image for Glance, which is the supported format in Ceph.

The creation of the standard Cinder volume and fast copy-on-write clone volume require you to use the Cinder API to forward a create image request from a defined image at the Glance storage (1). The Cinder volume service tries to locate the image under question in the Glance image store (2) and forwards its volume reference back to the API (4). Using the standard way to boot an instance, as shown in the Standard Cinder Volume Creation section in the previous figure, an image has to be pulled from Glance and streamed to the compute node, which is extremely slow (3). The new approach in the Fast Copy on Write Clone Volume section (3) gives the functionality to make snapshot of images while they are being imported. Thus, it might be easier and a more sophisticated to create clones from them as well as for volume from an image.

Ceph in OpenStack

We have been using Cinder and its driver-enabled support for Ceph. We already have an overview of OSDs, which are the workhorses for object and block storage. Moreover, partitions can be created for the OSD nodes and assigned different storage pools. Keep in mind that this setup can be an example from many others. The common point that you should stick to is the way you distribute the Ceph components across the OpenStack infrastructure. In this example, we made the ceph-mon daemon run in the controller node, which makes sense if you intend to centralize all the management services from a logical perspective. The ceph-osd nodes should run in the replica in separate storage nodes. The compute nodes need to know which Ceph node will clone the images or store the volumes that require a Ceph client to run on them.

From the network perspective, the ceph-osd nodes will join the private storage subnetwork while keeping the nodes that are running the Ceph daemons in the management network.

A simple integration model with OpenStack can be depicted in the following way:

Cooking Ceph

Let's go back to our kitchen and check out the recipes that we have to prepare at this stage. As we have many options to handle either the object or block storage, we will try to sum both of them as a storage backend for OpenStack. Cooking time! We will point to the basic Ceph cookbooks from the main repository of the Opscode cookbook market. You can add the Ceph cookbook to your Chef workstation, as follows:

packtpub@workstation$ cd /home/packtpub/chef-repo
packtpub@workstation /home/packtpub/chef-repo $  git clone https://github.com/ceph/ceph-cookbooks.git ceph

Note

The Apache cookbook is required as a cookbook dependency for Ceph. It might not have to be uploaded again because it already exists since the first install of the Berks cookbook dependency.

Upload the Ceph cookbooks again. The Chef server will then take care of it:

packtpub@workstation/home/packtpub/chef-repo$  knife cookbook upload ceph

Let's exploit another flexible feature in Chef—the multienvironment support. We created a basic environment that defines the distribution of the basic components and services of OpenStack in Chapter 2, Deploying OpenStack – DevOps and OpenStack Dual Deal. At this point, you will have two options. You can define a separate environment that was purely written for Ceph, and then proceed by modifying the current OpenStack configuration setup. Alternatively, you can use an environment file, to which you can add a Ceph subenvironment section and make some other changes to this file. Since Chef won't be bothered to rerun as many times as we want for the same environment, we will go for the second option. We would like to make more sense for the automation part. Moreover, when your infrastructure exposes dozens of nodes, you should avoid the nano way. Trusting your environment file will give you an easy life. On the other hand, it might not be wise to adopt such an approach when you only need to modify a few settings or integrate a small plugin that needs only a little modification within a limited number of attributes in your existing environment. Depending on your needs, automation always helps but without a blind eye. Let's bring our new environment file into action and highlight the spot of Ceph that is shown in bold, as follows:

{
  "name": "vagrant-packtpub",
   "description": "PacktPub Testing Environment for Ceph Integration in OpenStack."
  "cookbook_versions": {
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
    "ceph": {
      "config": {
        "mon_initial_members": [
          "controller1"
        ],
        "fsid": " 9ee348be-ef99-ea3e-7a7a-bb133abcef48",
        
        "osd": {
          "osd journal size": "1000"
        },
        "global": {
          "public network": "172.16.24.0/8",
          "storage network": "192.168.47.0/24"
        }
      },
      "openstack": true,
      "monitor-secret": "BASkSuJPonHgFHaaZixurLvTvAz4PRo5IKYGts=="
    }
  },
  "override_attributes": {
    …………
        "image": {
          "api": {
            "bind_interface": "eth1",
         "default_store": "rbd",
         "store_pool": "images"
          },
        …………..
        "block-storage": {
        "volume": {
          "provider": "ceph",
          "rbd_pool": "volumes"
       }
        }
…..
  }
}

We added three Ceph nodes, which have to be deployed within the two interfaces for each node. In the next execution of the Chef client, Cinder and Glance will be aware that we will use Ceph as the backend storage.

Note

To get the filesystem ID (fsid) and the monitor secret key, you will need to run the following commands from the ceph-common package:

# uuidgen -r
# ceph-authtool --name=monitor-secret --gen-key

As we covered in the previous chapter along with Chef deployment, we again intend to create a different set of roles for Ceph.

Basically, we need OSD and MON in the first place, which can be performed from the Chef web interface or via Knife in the following way:

$ nano roles/ceph-osd.json
{
  "name": "ceph-osd",
  "description": "Ceph Object Storage Device",
  "run_list": [
    "recipe[ceph::repo]",
    "recipe[ceph::osd]"
  ]
}
$ nano roles/ceph-mon.json
{
  "name": "ceph-mon",
  "description": "Ceph Monitor",
  "run_list": [
    "recipe[ceph::repo]",
    "recipe[ceph::mon]"
  ]
}

You will need to upload the following roles to the Chef server via the knife command line:

$ knife role from file roles/ceph-osd.rb
$ knife role from file roles/ceph-mon.rb

Note

Do not forget to upload any newly created or updated environment to the Chef server using the Knife command line:

# knife environment from file <path>

For each OSD host, you can assign ceph-osd as well as ceph-mon if you intend to run it in the same or a separate node. In our example, we can fire the ceph-mon daemon to run on the cloud controller by just adding the ceph-mon role to its Chef run-list. On the other hand, it can be useful to run multiple OSDs in the same node by running a ceph-osd daemon for each disk in your box from the global file environment, as follows:

.......
    "ceph": {
         "config": {
         .......
      }
              "osd_devices": {
                "0": {
                    "device": "/dev/sdb",
                    "zap": true
                },
                "1": {
                    "device": "/dev/sdc",
                    "journal": "/dev/sdc"
                }
            }
.......

You can run OSDs with the Ceph node by editing the node settings of each Ceph node, as follows:

{
  "chef_environment": " vagrant-packtpub ",
  "run_list": [
    "recipe[ceph::repo]",
    "role[ceph-osd]"
  ],
  "normal": {
        "ceph": {
    "osd_devices": [
        {
            "device": "/dev/sdb",
            "journal": "/dev/sdb"
        }
    ]
}
  },
  "name": "ceph01"
}

Let's update the Vagrant file, which will include a new ceph-mon role on the cloud controller node. For the sake of simplicity, we will include an additional node that runs three OSDs. The Vagrant file will be updated in the following way:

...
chef_environment = "vagrant-packtpub"

    controller_run_list = [
      "role[packtpub-os-base-controller]",
      ...
      "role[ceph-mon]"   
    ]
...
    Ceph_run_list = [
      "role[ceph-osd]"   
    ]
# Ceph 3 OSDs Node

  config.vm.define :ceph1 do |cephpp|
    cephpp.vm.hostname = "ceph1"
    cephpp.vm.box = "opscode-centos-6.5"
    cephpp.vm.box_url =  "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-6.5_chef-provisionerless.box"
    cephpp.vm.network "private_network", ip: "192.168.47.100" 
     file_to_disk = "./tmp/cephpp.osd_data.vdi"
 (0..2).each do |osd|
        config.vm.provider :virtualbox do |vb|
          vb.customize ['modifyvm', :id, '--memory', '2048',  '--cpus', '2']
          vb.customize ['createhd', '--filename', disk_file,  '--size', 4048]
          vb.customize ['storageattach', :id, '--storagectl',  'SATA Controller', '--port', 3+d, '--device', 0, '--type',  'hdd', '--medium', disk_file]
        end
      end
    cephpp.vm.provision :chef_client do |chef|
      chef.run_list = Ceph_run_list
      chef.environment = chef_environment
      # Where to find our Chef Server by providing the  authorization key
      chef.chef_server_url = "https://chefserver.packtpub.com:443"
     chef.validation_key_path = "/home/packtpub/chef  repo/.chef/chef-validator.pem"
    end
  end

The last thing that you have to do is just push the button. Chef will update the new role for the cloud controller in your Vagrant box, as follows:

# export VAGRANT_VAGRANTFILE=vagrant-packtpub
# vagrant reload --no-provision controller1
# vagrant up ceph

The following is the output for the preceding commands:

A new Ceph node with an address of 192.168.47.100 will be created. This node resides in the same private VB network. This will mimic the storage network in real production. Thus, you will have to change it to fit your network IP address.

You can check out the newly created Ceph node by issuing the following command:

# vagrant ssh ceph01

You can check whether the Ceph service is running in ceph01, as follows:

# ceph –s

For the preceding code, you will get the following result:

Storing images in Ceph

It is possible to use Ceph as a storage backend to store an operating system image for instances.

The following steps show how one can configure Glance to use Ceph as an alternative for the storage of images:

1. On the new Ceph instance, create a new Ceph pool for OpenStack Glance, as follows:

   # ceph osd pool create images 128
   

2. On the cloud controller node, configure OpenStack Glance to use the RBD store in /etc/glance/glance-api.conf, as follows:

   # nano /etc/glance/glance-api.conf
   rbd_store_user=glance
   rbd_store_pool=images
   

   Tip

   To enable the copy-on-write cloning feature, set the direct_url = True directive in /etc/glance/glance-api.conf.

3. Save the configuration file and restart the glance-api service, as follows:

   #/etc/init.d/glance-api restart
   

   Tip

   It is possible to reload the cloud controller configuration by commenting out the rbd_store_user and rbd_store_pool lines in the OpenStack image cookbook's attributes file.

4. On the cloud controller node, download a new image for Glance testing, as follows:

   # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2.xz
   

5. Create a new Glance image from the downloaded image in the following way:

   # glance image-create --name="CentOS-7-image" --is-public=True --disk-format-qcow2 --container-format=ovf < CentOS-7-x86_64-GenericCloud.qcow2.xz
   

   The preceding command yields the following output:

   

6. You can check out the image ID in the images Ceph pool by issuing the following query:

   # rados –p images ls
   

   For the preceding code, we will get the following output:

   

The CentOS image is stored in Ceph, which refers to the CentOS image ID that is shown in the Glance image output. The object that the Glance image recently stored and imported from Ceph is identified with the help of the rbd_id.Image_Glance_ID format.

Tip

Note that it is possible to configure Cinder and Nova to use Ceph as well. You will need to create a new Ceph pool and edit the /etc/cinder/cinder.conf file to specify the RBD driver for Cinder. Instances in OpenStack can be booted directly into Ceph which requires defining optionally in the /etc/nova/nova.conf file the ephemeral backend for Nova. To read more about this specific setup, you may follow this useful link http://ceph.com/docs/master/rbd/rbd-openstack/.

From Horizon, in the Access and Security section, you can add a security group and name it, for example, as PacktPub_SG. Then, a simple click on Edit Rules will do the trick. The following example illustrates how this network security function can help you understand how traffic—both in ingress/egress—can be controlled:

The previous security group contains four rules. The first and the second lines are rules to open all the outgoing traffic for IPv4 and IPv6 respectively. The third line allows the inbound traffic by opening the ICMP port, while the last one opens port 22 for SSH for the inbound interface. You might notice the presence of the CIDR fields, which is essential to know. Based on CIDR, you allow or restrict traffic over the specified port. For example, using CIDR of 0.0.0.0/0 will allow traffic for all the IP addresses over the port that was mentioned in your rule. For example, a CIDR with 32.32.15.5/32 will restrict traffic only to a single host with an IP of 32.32.15.5. If you would like to specify a range of IP addresses in the same subnet, you can use the CIDR notation, 32.32.15.1/24, which will restrict traffic to the IP addresses starting from 32.32.15.*; the other IP addresses will not stick to the latter rule.

The security groups also can be managed by using the Python Neutron command-line interface. Wherever you run the Neutron daemon, you can list, for example, all the present security groups from the command line in the following way:

# neutron security-group-list

The preceding command yields the following output:

To demonstrate how the PacktPub_SG security group rules that were illustrated previously are implemented on the host, we can add a new rule that allows the ingress connections to ping (ICMP) in the following way:

# neutron security-group-rule-create --protocol icmp –-direction ingress PacktPub-SG

The preceding command produces the following result:

The following command line will add a new rule that allows ingress connections to establish a secure shell connection (SSH):

# neutron security-group-rule-create --protocol tcp –-port-range-max 22 –-direction ingress PacktPub-SG

The preceding command gives the following output:

You may conclude from the output of the previous command line that it lists the rules that are associated with the current project ID and not by the security groups.

The Nova command line also does the same trick if you intend to perform the basic security group's control, as follows:

$ nova secgroup-list-rules default

Since we are setting Neutron as our network service controller, we will proceed by using the networking security groups, which will reveal additional traffic control features. If you are still using the compute API to manage the security groups, you can always refer to the nova.conf file for each compute node to set security_group_api = neutron.

To associate the security groups to certain running instances, it might be possible to use the Nova client in the following way:

# nova add-secgroup test-vm1 PacktPub_SG

The following command line illustrates the new association of the PacktPub_SG security group with the test-vm1 instance:

# nova show test-vm1

The following is the result of the preceding command:

One of the best practices to troubleshoot connection issues for the running instances is to start checking the iptables running in the compute node. Eventually, any rule that was added to a security group will be applied to the iptables chains in the compute node. We can check the updated iptables chains in the compute host after applying the security group rules by using the following command:

# iptables-save

The preceding command yields the following output:

The highlighted rules describe the direction of the packet and the rule that is matched. For example, the inbound traffic to the f7fabcce-f interface will be processed by the neutron-openvswi-if7fabcce-f chain.

If you have already created your own security groups, a series of iptables and chains are implemented on every compute node that hosts the instance that is associated within the applied corresponding security group. The following example demonstrates a sample update in the current iptables of a compute node that runs instances within the 10.10.10.0/24 subnet and assigns 10.10.10.2 as a default gateway for the former instances IP ranges:

The last rule that was shown in the preceding screenshot dictates how the flow of the traffic leaving the f7fabcce-f interface must be sourced from 10.10.10.2/32 and the FA:16:3E:7E:79:64 MAC address. The former rule is useful when you wish to prevent an instance from issuing a MAC/IP address spoofing. It is possible to test ping and SSH to the instance via the router namespace in the following way:

# ip netns exec router qrouter-5abdeaf9-fbb6-4a3f-bed2-7f93e91bb904 ping 10.10.10.2

The preceding command provides the following output:

The testing of an SSH to the instance can be done by using the same router namespace, as follows:

# ip netns exec router qrouter-5abdeaf9-fbb6-4a3f-bed2-7f93e91bb904 ssh cirros@10.10.10.2

The preceding command produces the following output:

In the current example, we will show a simple setup of a security group that might be applied to a pool of web servers that are running in the Compute01, Compute02 and Compute03 nodes. We will allow inbound traffic from the Internet to access WebServer01, AppServer01, and DBServer01 over HTTP, HTTPS, and MySQL. This is depicted in the following diagram:

Let's see how we can restrict the traffic ingress/egress via the Neutron API:

$ neutron security-group-create DMZ_Zone --description "allow web traffic from the Internet"

$neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 80 --port_range_max 80 DMZ_Zone --remote-ip-prefix 0.0.0.0/0

$neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 443 --port_range_max 443 DMZ_Zone --remote-ip-prefix 0.0.0.0/0

$neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 3306 --port_range_max 3306 DMZ_Zone --remote-ip-prefix 0.0.0.0/0

From Horizon, we can see the following security rules group have been added:

Creating a new security group, DMZ_Zone, will actually update the iptables rules running on the compute node that hosts the instance. For example, we can see on the compute node the new iptables rules by running the following command line:

# iptables-save

The output of the iptables list is quite long; for the sake of simplicity, we will go only through the fitter chain list.

Eventually the ingress/egress network will traverse first the FORWARD chain as the following:

The rules defined in both chains neutron-filter-top and neutron-openvswi-FORWARD will be processed. Then, iptables return to the calling FORWARD chain, which defines the following chain:

The previous rule will cause to jump to the neutron-openvswi-FORWARD chain. Therefore, it will jump to the neutron-openvswi-sg-chain chain, as the following:

The first rule denotes the direction of the traffic entering the tap6919f23b-34 interface.

The external traffic entering the tap6919f23b-34 interface will be processed and treated by the neurotn-openvsw-i6919f23b-3 chain, as the following:

The previous rules reflect the creation of the DMZ_Zone security group, which was configured previously in the Horizon dashboard.

Like Cinder, several plugins are exposed to relate some other backend implementation of the OpenStack block storage API requests. Similarly, the Neutron API supports a variety of plugins by using the Linux VLANs and firewalls. Initially, FWaaS in Neutron is available, but it needs to be activated. Let's do this.

In the Neutron node conf file named neutron.conf, insert the following lines:

service_plugins = firewall
[service_providers]
service_provider =  FIREWALL:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default
[fwaas]
driver =  neutron.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
enabled = True

To enable firewall management from the dashboard, change the controller node in /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py by using the following setting:

'enable_firewall' = True

Then, you need to restart neutron-server and the web server that runs Horizon in the following way:

root@packtpub# /etc/init.d/neutron-server restart
root@packtpub# service httpd restart

From Horizon, we can see that the new FWaaS section joined the OpenStack management interface, as shown in the following screenshot:

Basically, if you intend to create a firewall, you will need to create a router in the first place, which will apply the firewall setup. Note that FWaaS is a distributed implementation of a firewall per tenant. The firewall will be applied to a router that was created in the same project. You will be able to manage your firewall rules and policies from Horizon or via the Neutron's command line.

Starting the configuration of a firewall in OpenStack is very straightforward. For example, we can apply the rules to allow the ICMP and SSH traffic to go back and forth. You can do the same on security groups level both the ingress/egress directions but will be only in the instance- or port-filtering level. For example, in the VPN site-to-site setup, you will need to allow certain ports in both the L3 devices. We can prepare our rules for the instance and apply them to our first firewall in data center 01.

Let's create the firewall rule, as follows:

$neutron firewall-rule-create --protocol icmp --action allow --name FW01

Next, we will need to specify a firewall policy in the following way:

$neutron firewall-policy-create –-firewall-rules FW01 FWpolicy01

The last step will require the creation of a firewall named FWaaS, as follows:

$neutron firewall-create FWpolicy01 --name FWaaS

Ultimately, the goal of implementing networking plugins in OpenStack is to use the advanced functionalities that the native OpenStack network is not able to perform. You may remember that the basic network model is able to isolate different networks by using VLANs, which is great! But what about exploiting some other features that are present in both layer 2 and 3 by performing the operation of tunneling? One of the supported plugins in OpenStack is Open vSwitch. If anything, it is crucial to understand how to use and configure it in order to validate its use case. Software Defined Network (SDN) is the most recent network paradigm technology that has proven its amazing ease with regard to the network configuration and management. In a nutshell, the SDN concept is based on the separation of the hardware levels, which can be the device that is meant to forward the packets and the network intelligence layer or the decision maker layer that abstracts the network infrastructure to a software controller.

Keep in mind that Open vSwitch is an SDN-aware technology. It is a new, revolutionary way to write a program that can be used to customize and manage your network. Since we are moving in to the new era of network virtualization, it essentially provides switching services to virtual networks within the OpenStack environment.

We have already seen at the start of this chapter the extensions that were brought by the Neutron API. Basically, Open vSwitch is one of the most well-supported and stable plugins together with the Linux Bridge plugin. Both of these plugins provide a layer 2 switching. There is more! If you are wondering how one can make a virtual switch population talk to another physical one, Open vSwitch will support this mix. Open vSwitch officially supports the following:

Empowering the traffic isolation

Now that we measured the network security requirement by using Neutron, we can move ahead and extend the first network design in Chapter 1, Designing OpenStack Cloud Architecture.

Due to the pluggable architecture of the OpenStack networking, we will integrate Open vSwitch and test the network isolation on an instance level. Before we delve into the details, let's refresh our knowledge of the overall network topology by introducing two different logical networks:

• Tenant networks: By default, they are created in isolation and not shared with any other network that is associated within a user project in OpenStack

• Provider networks: In order to connect to the non-OpenStack resources, provider networks can be served by the administrator to map to a specific physical device in the data center

Let's assume that we would like to bring our network design into an existing physical setup. In other words, we want to create provider and tenant networks using the VLAN IDs that correspond to the real VLANs in the data center. We only need one OVS bridge to customize the connection between the following:

• Instances in an OpenStack environment

• Load balancers and firewalls

• A network device residing on the same VLAN layer 2

The next illustration depicts the existence of two tenants, A and B, each has a network with one router and one subnet. The Neutron router connects the tenants to the physical switch to interface the public Internet. Under the hood, it is possible to route securely between tenant networks using VLAN (802.1q tagged) and GRE-based networks. Without the Neutron router, both tenant networks are effectively isolated from each other.

As business grows, it might be needed to expand to the multisite endpoints. In OpenStack, it can be referred to as a multicell endpoint that can provide access to the different regions of the same OpenStack infrastructure. Of course, the implementation of a VPN setup—whether it is a simple SSL one or an IPSEC solution—will empower the security of traffic across the Internet. We have worked around the host security level and brought different isolated networks in order to avoid traffic congestion and improve the security of the internal network of the OpenStack environment. The following case study will sum up the different aspects that were covered previously and take things a step further by protecting the integrity of data by using the tunneling and encryption basis. A fruitful Neutron extension provides VPN as a Service (VPNaaS) again, thanks to virtualization! Now, if you plan to connect two machines in different tenants that are geographically located in different data centers, then you should consider OpenStack, which makes life easier and offers a simple step-by-step configuration. VPNaaS is proven to be extension capable not only to build a site-to-site VPN connection between two private networks, but also to implement several VPN connections per tenant. Thus, you can create as many VPNs as you want, which brings to you a real, extensible network. Let's go to Horizon and study an example that helps you run a hybrid application.

General settings

The next figure depicts two different OpenStack sites, and we intend to link their associated tenants. You may remember that a project in Horizon presents a tenant description that includes its private networks, routers, and subnets.

Tip

Add an admin user to each project in the Admin role. This will allow you to fully perform administrative tasks in Horizon. Additionally, you can create different users per project and assign a service type to it using an admin account.

Let's create our first project named PackPub01 in DC01 and a second project called PacktPub02 in DC02.

Based on the previous illustration, we intend to let the machines in both the OpenStack subnets talk to each other. Note that each subnet is frontended by a gateway router. Each project's network has a defined subnet that serves local IP addresses, whereas a router connects to the external public interface of each network. The connection between both the public IP addresses will be encrypted by the means of VPNaaS. In other words, the traffic between the networks will be tunneled.

The following screenshot shows a basic network topology for DC01:

DC02 will have the same network topology by bridging the 192.168.48.0/24 local subnet to the Internet via an external router interface, which is depicted in the following screenshot:

Note

Subnets in different tenant projects must have nonoverlapping IP address ranges.

We have configured the setup from Horizon. Let's bring VPNaaS shining in our interface by using the following three steps:

1. In order to create a full site-to-site IPsec VPN, we will use Openswan IPsec implementation for Linux. Neutron supports Openswan by providing a driver, which needs to be configured. Additionally, we will need the neutron-plugin-vpn-agent package to be installed on both network nodes in each site as follows:

   # yum install openswan openstack-neutron-vpn-agent
   

2. Start the ipsec service:

   # /etc/init.d/ipsec start
   

3. Configure the Neutron agent file to use the openswan driver as follows:

   # nano /etc/neutron/vpn_agent.ini
   [vpnagent]
   …
   vpn_device_driver=neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver
   …
   

4. Enable VPNaaS in Neutron by activating its plugin in /etc/neutron/neutron.conf, as follows:

   service_plugins =……….. ,vpnaas
   

5. In the same file, enable the VPN service provider to use the Openswan driver in the service_providers section as the following:

   #service_provider=VPN:openswan:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default
   

6. Next, we will add a VPNaaS module interface in /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py in the following way:

       'enable_VPNaaS': True,
   

7. Finally, restart neutron-server and neutron-vpn-agent services, and the web server, as follows:

   # /etc/init.d/httpd restart
   # /etc/init.d/neutron-server restart
   # /etc/init.d/neutron-vpn-agent restart
   

Note

To read more about Openswan, check the official website:

https://www.openswan.org/

VPNaaS configuration

We will start by configuring VPN on DC01.

Creating the Internet Key Exchange policy

In Horizon, we can create the Internet Key Exchange (IKE) policy in the first VPN phase. The following screenshot shows a simple IKE setup of an OpenStack environment that is located in the DC01 site:

Tip

An IKE policy can also be created by using the Neutron command line, as follows:

# neutron vpn-ikepolicy-create --auth-algorithm sha1 --encryption-algorithm aes-256  --ike-version v2  --lifetime units=seconds,value=3600 --pfs group5  --phase1-negotiation-mode  main --name PP-IKE-Policy

Creating an IPSec policy

The creation of an IPSec policy in the OpenStack environment that is located in the DC01 site in Horizon can be done in the following way:

Tip

An IPSec policy can also be created by using the Neutron command line, as follows:

# neutron vpn-ipsecpolicy-create  --auth-algorithm  sha1 --encapsulation-mode tunnel --encryption-algorithm aes-256  --lifetime units=seconds,value=36000  --pfs group5  --transform-protocol esp –name PP_IPSEC_Policy

Standard VPN settings such as Encapsulation mode, Encryption algorithm, Perfect Forward Secrecy, and Transform Protocol should remain the same in both the sites for phase 1.

Tip

If you face a VPN connectivity problem, a best practice of troubleshooting before filtering or debugging the traffic is to begin checking the existence of any mismatch of the phase 1 and phase 2 settings in both sites.

Creating a VPN service

To create a VPN service, you will need to specify the router facing the external interface and attach the web server instance to the private network in the DC01 site. The router will act as a VPN gateway. We can add a new VPN service from Horizon in the following way:

Tip

A VPN service can also be created by using the Neutron command line, as follows:

# neutron vpn-service-create --tenant-id  c4ea3292ca234ddea5d50260e7e58193 --name PP_VPN_Service  Router-VPN public_subnet

Keep in mind that a VPN service is needed to select the router that will perform your VPN gateway. Note that here, we have exposed our local subnet, 192.168.47.0/24.

Creating an IPSec site connection

The last step needs some information. Usually, we need to set up an external IP address of the other peer for a VPN site. In OpenStack, you can check it by logging on as an admin and from the PacktPub02 tenant, clicking on the Router section. Here, you will get all of its details, which includes information regarding the external gateway interface, 172.24.4.227. The Remote peer subnet(s) value is the CIDR notation 192.168.48.0/24. We will finish our first project, the DC01 VPNaaS connection, by setting the secret preshared key to AwEsOmEVPn. The key will be the same for both the sides. The process of setting the key is depicted in the following screenshot:

Tip

A VPN service can also be created by using the Neutron command line, as follows:

# neutron ipsec-site-connection-create --name PP_IPSEC ---vpnservice-id PP_VPN_Service --ikepolicy-id PP-IKE-Policy --ipsecpolicy-id PP_IPSEC_Policy --peer-address 192.168.48.0/24 --peer-id 172.24.4.232 --psk AwEsOmEvPn

The peer gateway public IPv4 address can be obtained from the router details of DC02. We will need to hit the external interface, as shown in the following screenshot:

To finish the VPN setup, you will need to follow the latter steps, but changing the IP gateway addresses of DC01 and the remote subnet to 192.168.47.0/24. Note that the VPN settings, encryption algorithms and protocols, and the shared password must be the same on both the sides.

A small smoke test can evaluate our setup.

From an instance in DC01, we can ping the DC02 site via 192.168.48.12, as follows:

From an instance in DC02, we can ping the DC01 site via 192.168.47.13, as follows:

Tip

Be sure that you have enabled ICMP on the DC02 router to allow pings.

We still remember that one of the several purposes of OpenStack clustering is to make sure that services remain running in the case of a node failure. The HA functionality aims to make sure that the different nodes participating in a given cluster work in tandem to satisfy certain downtime. HA, in fact, is a golden goal for any organization where some useful concepts can be used to reach it with minimum downtime, such as the following:

On the other side, we may find a different terminology, which you may have most likely already experienced, that is, load balancing. In a heavily loaded environment, load balancers are introduced to redistribute a bunch of requests to less loaded servers. This can be similar to the high performance clustering concept, but you should note that this cluster logic takes care of working on the same request, whereas a load balancer aims to relatively distribute the load based on its task handler in an optimal way.

It might be important to understand the context of HA deployments in OpenStack. This makes it imperative to distinguish the different levels of HA in order to consider the following in the cloud environment:

The main focus of the supporting HA in OpenStack has been on L1 and L2, which are covered in this chapter. On the other hand, L3 HA has limited support in the OpenStack community. By virtue of its multistorage backend support, OpenStack is able to bring instances online in the case of host failure by means of live migration. Nova also supports the Nova evacuate implementation, which fires up API calls for VM evacuation to a different host due to a compute node failure. The Nova evacuate command is still limited as it does not provide an automatic way of instance failover. L2 and L3 HA are considered beyond the scope of this book. L4 HA is touched on, and enhanced by, the community in the Havana release. Basically, a few incubated projects in OpenStack, such as Heat, Savana, and Trove, have begun to cover HA and monitoring gaps in the application level. Heat will be introduced in Chapter 8, Extending OpenStack – Advanced Networking Features and Deploying Multi-tier Applications, while Savana and Trove are beyond the scope of this book.

Normally, if you plan to invest time and money in OpenStack clustering, you should refer to the HA architectural approaches in the first place. They guarantee business continuity and service reliability.

At this point, meeting these challenges will drive you to acquire skills you never thought you could master. Moreover, exposing an infrastructure that accepts failures might distinguish your environment as a blockbuster private cloud. Remember that this topic is very important in that all you have built within OpenStack components must be available to your end user.

Availability means that not only is a service running, but it is also exposed and able to be consumed. Let's see a small overview regarding the maximum downtime by looking at the availability percentage or HA as X-nines:

Basically, availability management is a part of IT best practices when it comes to making sure that IT services are running when needed, which reflects your service-level agreement (SLA):

A paradox may appear between the lines when we consider that eliminating the SPOF in a given OpenStack environment will include the addition of more hardware to join the cluster. At this point, you might be exposed to creating more SPOF and, even worse, complicated infrastructure where maintenance turns into a difficult task.

To ease the following sections of this chapter, it might be necessary to remember few terminologies to justify high availability and failover decisions later:

Let's apply our former definition to our OpenStack services:

Any HA architecture introduces an "active/active" or "active/passive" deployment, as covered in Chapter 1, Designing OpenStack Cloud Architecture. This is where your OpenStack environment will highlight its scalability level.

First, let's see the difference between both concepts in a nutshell in order to justify your decision:

Chapter 1, Designing OpenStack Cloud Architecture, provided a few hints on how to prepare for the first design steps: do not lock keys inside your car. At this point, we can go further due to the emerging different topologies, and it is up to you to decide what will fit best. The first question that may come into your mind: OpenStack does not include native HA components; how you can include them? There are widely used solutions for each component that we cited in the previous chapter in a nutshell.

Understanding HAProxy

HAProxy stands for High Availability Proxy. It is a free load balancing software tool that aims to proxy and direct requests to the most available nodes based on TCP/HTTP traffic. This includes a load balancer feature that can be a frontend server. At this point, we find two different servers within an HAProxy setup:

• A frontend server listens for requests coming on a specific IP and port, and determines where the connection or request should be forwarded

• A backend server defines a different set of servers in the cluster receiving the forwarded requests

Basically, HAProxy defines two different load balancing modes:

• Load balancing layer 4: Load balancing is performed in the transport layer in the OSI model. All the user traffic will be forwarded based on a specific IP address and port to the backend servers.

  For example, a load balancer might forward the internal OpenStack system's request to the Horizon web backend group of backend servers. To do this, whichever backend Horizon is selected should respond to the request under scope. This is true in the case of all the servers in the web backend serving identical content. The previous example illustrates the connection of the set servers to a single database. In our case, all services will reach the same database cluster.

• Load balancing layer 7: The application layer will be used for load balancing. This is a good way to load balance network traffic. Simply put, this mode allows you to forward requests to different backend servers based on the content of the request itself.

Many load balancing algorithms are introduced within the HAProxy setup. This is the job of the algorithm, which determines the server in the backend that should be selected to acquire the load. Some of them are as follows:

• Round robin: Here, each server is exploited in turn. As a simple HAProxy setup, round robin is a dynamic algorithm that defines the server's weight and adjusts it on the fly when the called instance hangs or starts slowly.

• Leastconn: The selection of the server is based on the lucky node that has the lowest number of connections.

  Tip

  It is highly recommended that you use the leastconn algorithm in the case of long HTTP sessions.

• Source: This algorithm ensures that the request will be forwarded to the same server based on a hash of the source IP as long as the server is still up.

  Note

  Contrary to RR and leastconn, the source algorithm is considered a static algorithm, which presumes that any change to the server's weight on the fly does not have any effect on processing the load.

• URI: This ensures that the request will be forwarded to the same server based on its URI. It is ideal to increase the cache-hit rate in the case of proxy caches' implementations.

  Note

  Like the source, the URI algorithm is static in that updating the server's weight on the fly will not have any effect on processing the load.

You may wonder how the previous algorithms determine which servers in OpenStack should be selected. Eventually, the hallmark of HAProxy is a healthy check of the server's availability. HAProxy uses health check by automatically disabling any backend server that is not listening on a particular IP address and port.

But how does HAProxy handle connections? To answer this question, you should refer to the first logical design in Chapter 1, Designing OpenStack Cloud Architecture, which is created with virtual IP (VIP). Let's refresh our memory about the things that we can see there by treating a few use cases within a VIP.

Services should not fail

A VIP can be assigned to the active servers running all the OpenStack services that need to be configured to use the address of the server. For example, in the case of a failover of the nova-api service in controller node 1, the IP address will follow the nova-api in controller node 2, and all clients' requests, which are the internal system requests in our case, will continue to work:

The load balancer should not fail

The previous use case assumes that the load balancer never fails! But in reality, this is an SPOF that we have to arm by adding a VIP on top of the load balancer's set. Usually, we need a stateless load balancer in OpenStack services. Thus, we can undertake such challenges using software similar to Keepalived:

Keepalived is a free software tool that provides high availability and load balancing facilities based on its framework in order to check a Linux Virtual Server (LVS) pool state.

Note

LVS is a highly available server built on a cluster of real servers by running a load balancer on the Linux operating system. It is mostly used to build scalable web, mail, and FTP services.

As shown in the previous illustration, nothing is magic! Keepalived uses the Virtual Router Redundancy Protocol (VRRP) protocol to eliminate SPOF by making IPs highly available. VRRP implements virtual routing between two or more servers in a static, default routed environment. Considering a master router failure event, the backup node takes the master state after a period of time.

Note

In a standard VRRP setup, the backup node keeps listening for multicast packets from the master node with a given priority. If the backup node fails to receive any VRRP advertisement packets for a certain period, it will take over the master state by assigning the routed IP to itself. In a multibackup setup, the backup node with the same priority will be selected within its highest IP value to be the master one.