Puppet+Hiera+Jenkins. Concepts

Wouldn't it be great if we could transfer large part of the application configuration from Development to Ops? For instance, typical Java EE application hold tons of external references (such as DB Url, User, Password, Security Certificates, etc). Typical configuration change often requires hours and distracts developers, build engineers and QA. Lots of waste for trivial change.

In this series of three posts we will show a configuration management framework, build of Puppet, Hiera and Jenkins that supports multiple environments:

• In this post we will review concepts and Puppet's configuration files
• Second post will be devoted to custom modules and interaction with Hiera
• Final, third post will focus on Jenkins integration

 Our target environment could be pictured as following:

Fig 1: Deployment schema for two environments

We will build on top of practices [1], and I strongly advice to read that article before proceeding further.

Conceptually, the workflow consist of the following steps:
1. Platform setup: during this step system interfaces and services of the nodes are configured (networking, file-system permissions, users and groups creations, rdbms, http servers, etc)

Fig 2: Puppet workflow

2.  Application setup: during this step business applications are installed and configured on the nodes

Fig 3: Jenkins workflow

Our skeleton's file structure will look as follow:


Three folders form a top-level hierarchy:

• /etc/puppet soon after puppet package is installed this folder contains standard puppet modules
  In addition, we will place there two configuration files:
  - puppet.conf describing puppet search paths and config
  - hiera.yaml describing Hiera search paths and config
  
• /biz/puppet/modules contains all of the custom modules
• /biz/puppet/hieradata holds Hiera data files grouped by environment

Lets review puppet configuration files from /etc/puppet

puppet.conf

This file should be normally divided into two: one with [main] and [master] sections deployed at Puppet Master node and another with [main] and [agent] sections deployed to Puppet Agent nodes.
puppet.conf provides wide spectrum of settings [3], but here we define only most critical ones, such as:

• modulepath defines path to the custom modules
• manifest defines where site.pp files are located
  Note that in our case site.pp files are environment-specific. For instance site.pp for QA environment is stored in /biz/puppet/hieradata/qa/ folder
• server identifies the Puppet Master domain name
• environment defines environment of the Agent node. For instance: qa, ci, etc
  Details on the environment setting could be found at [2]

hiera.yaml
Above, we declared that configuration files for particular nodes in the cluster will be in JSON format, grouped by environment and identified by the full domain name.
Note that the attribute -common under :hierarchy: denotes common settings for the environment and in our case refers to the /biz/puppet/hieradata/${environment}/common.json file.

Continue with custom modules.

[1] Craig Dunn: Designing Puppet – Roles and Profiles
http://www.craigdunn.org/2012/05/239/

[2] Defining environment in Puppet/Hiera
http://docs.puppetlabs.com/guides/environment.html

[3] Puppet.conf in details
http://docs.puppetlabs.com/references/latest/configuration.html