Puppet+Hiera+Jenkins. Jenkins Integration

Continuation. By now we should have a working bundle of Puppet and Hiera, and in this post we will focus on Jenkins integration, as well as management of the apps configuration.

First, let's recall that roles contain class {'configuration':} declaration. This tiny module allows us to keep a current copy of the apps configuration on the target nodes.
Module's file tree looks as follow:

	.
	├── files
	│   ├── ci
	│   │   ├── enterprise_app_1
	│   │   │   ├── dir.1
	│   │   │   │   └── file.1
	│   │   │   ├── file.1
	│   │   │   └── file.2
	│   │   └── web_app_1
	│   │   ├── dir.1
	│   │   │   └── file.1
	│   │   ├── file.1
	│   │   └── file.2
	│   └── qa
	│   ├── enterprise_app_1
	│   │   ├── dir.1
	│   │   │   └── file.1
	│   │   ├── file.1
	│   │   └── file.2
	│   └── web_app_1
	│   ├── dir.1
	│   │   └── file.1
	│   ├── file.1
	│   └── file.2
	└── manifests
	└── init.pp

view raw configuration_tree.txt hosted with ❤ by GitHub

The manifest file itself (init.pp) is as follows:

	# file /biz/puppet/modules/configuration/manifests/init.pp
	class configuration ($application, $env){
	# creating directory structure first
	file { ['/biz', '/biz/configuration']:
	ensure => directory,
	mode => 0640,
	}
	file { "/biz/configuration/${application}":
	ensure => directory, # so make this a directory
	recurse => true, # enable recursive directory management
	purge => true, # purge all unmanaged junk
	force => true, # also purge subdirs and links etc.
	mode => 0640,
	source => "puppet:///modules/configuration/${env}/${application}",
	}
	}

view raw configuration hosted with ❤ by GitHub

It can be summarized as procedure that base on ${environment} name and ${application} name copies configuration files from /biz/puppet/hieradata/${environment}/${application} on Puppet Master to /biz/configuration/${application} on Puppet Agent node.

It is important that destination point has no reference of the ${environment}. This later allows Jenkins to perform configuration update in a complete agnostic way - with only ${application} name being required.

Consider the file tree under files folder. It is grouped by environment and application name. Should you like to add a new application, let's say HelloWorld to the CI env, you would have to:

1. Make sure that folder /biz/puppet/hieradata/ci exist
2. Create new subfolder /biz/puppet/hieradata/ci/HelloWorld
3. Place HelloWorld configuration files into /biz/puppet/hieradata/ci/HelloWorld 
4. Create/update Jenkins script to take the application configuration files from /biz/configuration/${application}
5. Create/update a role to reference the HelloWorld:

	class role::hello_world_server inherits role {
	# ...
	class { 'configuration' :
	env => "${environment}",
	application => 'HelloWorld',
	}
	}

view raw hello_world_role.pp hosted with ❤ by GitHub

The flow can be illustrated as:

Fig 1: Deployment process

Last, but not least - let's describe development process on Puppet+Hiera+Jenkins govern cluster:

Fig 2: Development process

Fig 2 illustrates that the framework significantly fast-tracks configuration changes. In summary, we have shown a configuration management framework targeted for small-to-medium sized project, where you are comfortable with file-based configuration for your applications.

Cheers!

Puppet+Hiera+Jenkins. Custom Modules

Continuation. In this second post of the series we will focus on custom modules for the Puppet+Hiera+Jenkins framework. For the sake of brevity we will illustrate our efforts at an example of a single QA environment.

As it is defined in puppet.conf, site.pp file could be found at /biz/puppet/hieradata/${environment}. In terms of QA environment it is /biz/puppet/hieradata/qa:

	node ip-10-0-8-11 {
	include role::web_portal
	}
	node ip-10-0-8-12 {
	include role::enterprise_server
	}

view raw site.pp hosted with ❤ by GitHub

Here, manifest (aka site.pp) assigns a single role to every node in the cluster. Nodes are identified by the full domain name. Let's assume that:

• Puppet Master lives at ip-10-0-8-10.sampup.com
• ip-10-0-8-11.sampup.com and ip-10-0-8-12.sampup.com are Puppet Agents. 

Roles are defined in /biz/puppet/modules/role module:

	# file /biz/puppet/modules/role/manifests/init.pp
	class role {
	include profile::base
	}
	# file /biz/puppet/modules/role/manifests/web_portal.pp
	class role::web_portal inherits role {
	include profile::web_server
	class { 'configuration' :
	env => "${environment}",
	application => 'web_app_1',
	}
	}
	# file /biz/puppet/modules/role/manifests/enterprise_server.pp
	class role::enterprise_server inherits role {
	# out of the blog post scope - install tomcat+java
	# include profile::tomcat_server
	class { 'configuration' :
	env => "${environment}",
	application => 'enterprise_app_1',
	}
	}

view raw roles hosted with ❤ by GitHub

You might be asking what class {'configuration':} is about? This where actual application configuration is delivered to agent nodes, and we will address this module in the last post of the series.

Corresponding profiles are defined in /biz/puppet/modules/profile module:

	# file /biz/puppet/modules/profile/manifests/init.pp
	class profile {
	}
	# file /biz/puppet/modules/profile/manifests/web_server.pp
	class profile::web_server {
	class { "apache": }
	}
	# file /biz/puppet/modules/profile/manifests/base.pp
	class profile::base {
	include networking
	include timezone::utc
	include users
	}

view raw profiles hosted with ❤ by GitHub

Lets discuss Hiera configuration. There is a lot of reading available [1], however we can summarize following:

1. Hiera data files are named after node's fully qualified domain name
   For instance: ip-10-0-8-12.sampup.com.json
2. Files contain key-values pairs
   For instance: "ntp::restrict" : true
3. Each key comprise of the module name and the property name, joined by ::
   For instance, in "ntp::restrict" : true - ntp is the module name
   - restrict is the property name
   - true is the value
4. Declared key-values are applied during invocation of the parametrized Puppet classes
5. In case Hiera finds no filename matching the node's domain name, it will first look in common.json
   Puppet will resolve to the default module parameters, should it find no relevant key-values pairs.

/biz/puppet/hieradata/qa/common.json

	{
	"ntp::restrict" : true,
	"ntp::autoupdate" : true,
	"ntp::enable" : true,
	"ntp::servers" : [
	"0.centos.pool.ntp.org iburst",
	"1.centos.pool.ntp.org iburst",
	"2.centos.pool.ntp.org iburst"
	]
	}

view raw common.json hosted with ❤ by GitHub

/biz/puppet/hieradata/qa/ip-10-0-8-12.sampup.com.json

	{
	"apache::vhost::priority": "10",
	"apache::vhost::vhost_name": "web_server.sampup.com",
	"apache::vhost::port": "80",
	"apache::vhost::docroot": "/var/www",
	}

view raw ip-10-0-8-12.sampup.com.json hosted with ❤ by GitHub

Continue with Jenkins and application configuration.

[1] Hiera reading
http://docs.puppetlabs.com/hiera/1/puppet.html#hiera-lookup-functions

Puppet+Hiera+Jenkins. Concepts

Wouldn't it be great if we could transfer large part of the application configuration from Development to Ops? For instance, typical Java EE application hold tons of external references (such as DB Url, User, Password, Security Certificates, etc). Typical configuration change often requires hours and distracts developers, build engineers and QA. Lots of waste for trivial change.

In this series of three posts we will show a configuration management framework, build of Puppet, Hiera and Jenkins that supports multiple environments:

• In this post we will review concepts and Puppet's configuration files
• Second post will be devoted to custom modules and interaction with Hiera
• Final, third post will focus on Jenkins integration

 Our target environment could be pictured as following:

Fig 1: Deployment schema for two environments

We will build on top of practices [1], and I strongly advice to read that article before proceeding further.

Conceptually, the workflow consist of the following steps:
1. Platform setup: during this step system interfaces and services of the nodes are configured (networking, file-system permissions, users and groups creations, rdbms, http servers, etc)

Fig 2: Puppet workflow

2.  Application setup: during this step business applications are installed and configured on the nodes

Fig 3: Jenkins workflow

Our skeleton's file structure will look as follow:

	.
	├── biz
	│ └── puppet
	│ ├── hieradata
	│ │ ├── ci
	│ │ │ ├── node
	│ │ │ │ ├── ip-10-0-8-11.sampup.com.json
	│ │ │ │ └── ip-10-0-8-12.sampup.com.json
	│ │ │ ├── common.json
	│ │ │ └── site.pp
	│ │ └── qa
	│ │ ├── node
	│ │ │ ├── ip-10-0-8-11.sampup.com.json
	│ │ │ └── ip-10-0-8-12.sampup.com.json
	│ │ ├── common.json
	│ │ └── site.pp
	│ └── modules
	│ ├── configuration
	│ │ ├── files
	│ │ │ ├── ci
	│ │ │ │ ├── enterprise_app_1
	│ │ │ │ │ ├── dir.1
	│ │ │ │ │ │ └── file.1
	│ │ │ │ │ ├── file.1
	│ │ │ │ │ └── file.2
	│ │ │ │ └── web_app_1
	│ │ │ │ ├── dir.1
	│ │ │ │ │ └── file.1
	│ │ │ │ ├── file.1
	│ │ │ │ └── file.2
	│ │ │ └── qa
	│ │ │ ├── enterprise_app_1
	│ │ │ │ ├── dir.1
	│ │ │ │ │ └── file.1
	│ │ │ │ ├── file.1
	│ │ │ │ └── file.2
	│ │ │ └── web_app_1
	│ │ │ ├── dir.1
	│ │ │ │ └── file.1
	│ │ │ ├── file.1
	│ │ │ └── file.2
	│ │ └── manifests
	│ │ └── init.pp
	│ ├── networking
	│ │ └── manifests
	│ │ └── init.pp
	│ ├── profile
	│ │ └── manifests
	│ │ ├── base.pp
	│ │ ├── init.pp
	│ │ └── web_server.pp
	│ ├── role
	│ │ └── manifests
	│ │ ├── enterprise_server.pp
	│ │ ├── init.pp
	│ │ └── web_portal.pp
	│ └── timezone
	│ └── manifests
	│ ├── init.pp
	│ └── utc.pp
	├── etc
	│ └── puppet
	│ ├── hiera.yaml
	│ └── puppet.conf
	├── agent_deploy.sh
	├── deploy.sh
	└── install_modules.sh

view raw file_tree.txt hosted with ❤ by GitHub

Three folders form a top-level hierarchy:

• /etc/puppet soon after puppet package is installed this folder contains standard puppet modules
  In addition, we will place there two configuration files:
  - puppet.conf describing puppet search paths and config
  - hiera.yaml describing Hiera search paths and config
  
• /biz/puppet/modules contains all of the custom modules
• /biz/puppet/hieradata holds Hiera data files grouped by environment

Lets review puppet configuration files from /etc/puppet

puppet.conf

	[main]
	server = ip-10-0-8-10.sampup.com
	certname = ip-10-0-8-10.sampup.com
	#user = puppet
	#group = puppet
	vardir = /var/lib/puppet
	factpath = $vardir/lib/facter
	templatedir = $confdir/templates
	# The Puppet log directory.
	# The default value is '$vardir/log'.
	logdir = /var/log/puppet
	# Where Puppet PID files are kept.
	# The default value is '$vardir/run'.
	rundir = /var/run/puppet
	# Where SSL certificates are kept.
	# The default value is '$confdir/ssl'.
	ssldir = $vardir/ssl
	[master]
	dns_alt_names = ip-10-0-8-10.sampup.com, puppet
	# overriding location of the hierra.yaml
	hiera_config = /etc/puppet/hiera.yaml
	# site.pp path
	manifest = /biz/puppet/hieradata/$environment/site.pp
	# a multi-directory modulepath:
	modulepath = /etc/puppet/modules:/usr/share/puppet/modules:/biz/puppet/modules
	[agent]
	pluginsync = true
	# The file in which puppetd stores a list of the classes
	# associated with the retrieved configuratiion. Can be loaded in
	# the separate ``puppet`` executable using the ``--loadclasses``
	# option.
	# The default value is '$confdir/classes.txt'.
	classfile = $vardir/classes.txt
	# Where puppetd caches the local configuration. An
	# extension indicating the cache format is added automatically.
	# The default value is '$confdir/localconfig'.
	localconfig = $vardir/localconfig
	certname = ip-10-0-8-11.sampup.com
	dns_alt_names = ip-10-0-8-11.sampup.com
	report = true
	archive_files = true
	environment = qa

view raw puppet.conf hosted with ❤ by GitHub

This file should be normally divided into two: one with [main] and [master] sections deployed at Puppet Master node and another with [main] and [agent] sections deployed to Puppet Agent nodes.
puppet.conf provides wide spectrum of settings [3], but here we define only most critical ones, such as:

• modulepath defines path to the custom modules
• manifest defines where site.pp files are located
  Note that in our case site.pp files are environment-specific. For instance site.pp for QA environment is stored in /biz/puppet/hieradata/qa/ folder
• server identifies the Puppet Master domain name
• environment defines environment of the Agent node. For instance: qa, ci, etc
  Details on the environment setting could be found at [2]

hiera.yaml

	---
	:backends:
	- json
	:json:
	:datadir: /biz/puppet/hieradata/%{::environment}
	:hierarchy:
	# A single node/ directory will contain any number of files named after some node’s fqdn (fully qualified domain name) fact.
	# (E.g. /etc/puppet/hiera/node/grover.example.com.json) This lets us specifically configure any given node with Hiera.
	# Not every node needs to have a file in node/ — if it’s not there, Hiera will just move onto the next hierarchy level.
	- node/%{::fqdn}
	- common
	:logger:
	- puppet

view raw hiera.yaml hosted with ❤ by GitHub

Above, we declared that configuration files for particular nodes in the cluster will be in JSON format, grouped by environment and identified by the full domain name.
Note that the attribute -common under :hierarchy: denotes common settings for the environment and in our case refers to the /biz/puppet/hieradata/${environment}/common.json file.

Continue with custom modules.

[1] Craig Dunn: Designing Puppet – Roles and Profiles
http://www.craigdunn.org/2012/05/239/

[2] Defining environment in Puppet/Hiera
http://docs.puppetlabs.com/guides/environment.html

[3] Puppet.conf in details
http://docs.puppetlabs.com/references/latest/configuration.html