If you've used Puppet, you probably know difficult is develop the modules needed for your requirements. Large applications probably have hundreds of configurable params and handle all of them can be tedious, horrifying or with some hints, simply awesome.

In this post I'm going to show you how I handle a very large application, composed by multiple services which is configured via multiple configuration files for each service. We will define a common hiera configuration, create modules for each service and deploy & configure multiple instances for ensuring High Availability

Hiera: Where our configs values come from

Puppet uses file hierarchy for reading dynamically all the properties defined in these files. The hierarchy must be defined in the file /etc/puppet/hiera.yaml or C:\ProgramData\PuppetLabs\puppet\etc\hiera.yaml in a Windows System.

In this file the hierarchy it's defined in a top-down approach the upper entries are used to resolve the configurations with higher priority.

This is an example of the hiera.yaml file

---
# Hiera 5 Global configuration file

version: 5

# defaults:
#   data_hash: yaml_data
# hierarchy:
#  - name: Common
#    data_hash: yaml_data
hierarchy:

  - name: "Node specific configuration"
    path: "C:/ProgramData/PuppetLabs/code/environments/hieradata/%{trusted.certname}.yaml"

  - name: "ODBC Common Configuration"
    path: "C:/ProgramData/PuppetLabs/code/environments/%{environment}/hieradata/odbc.yaml"

  - name: "AppServer specific configuration"
    path: "C:/ProgramData/PuppetLabs/code/environments/%{environment}/hieradata/appconfig.yaml"

  - name: "Website specific configuration"
    path: "C:/ProgramData/PuppetLabs/code/environments/%{environment}/hieradata/website.yaml"

This will be our lookup order:

  1. Node name custom configs %{trusted.certname}
  2. Common properties for each service (different for each environment) %{environment}

Creating our first module

In Puppet, one module is a directory under ./environments/$environment/modules directory which have the following structure:

  • files Contains the plain files to copied during the deployment process.
  • manifests Puppet DSL files describing the deployment process with resources. If we want to apply one module via UI we should create a main file named init.pp having as class name the name of the parent directory.
  • templates *.ERB Ruby templates for configurations advanced replacing using hiera.
  • tests Directory containing test files.

As we want to apply profiles instead modules, the modules shouldn't be available to apply via UI (Foreman), to achive this behaviour, the file init.pp shouldn't be present under ./modules/$modulename/manifests directory.

Dynamic Parameters in Puppet classes

As we don't want init.pp file, we will start from config_init.pp file name as standard.

This file will contain the main parameters of the service to deploy like instances to deploy & configure or the installation path if needed.

We have to methods to lookup for parameters using hiera.

Specifying custom lookup

If you want to specify what token search for, you can define the paramater as usually:

class m4gateway::configs_init (
  $install_path = lookup('profile::gateway::install_path'),
  $instances = lookup('gateway')
) {
  # Class specific implementation
}

This class will use hiera for searching for the specified tokens profile::gateway::install_path and gateway

But, there's another automatic option, which is recommended if you're going to manage large deployments.

Automatic parameters binding

Puppet uses the name of the class plus the name of the parameter for automatically search for the appropiate token in a automagically way. Let's see how this Puppet function works.

Taking the same previous class definition:

class gateway::configs_init (
  $install_path = {},
  $instances = {}
) {
  # Class specific implementation
}

We need to declare the desired parameters as empty objects, and Puppet once the class is instantiated, will lookup for the appended token class name + parameter.

Taking the given example, internally, Puppet will lookup for the tokens gateway::configs_init::install_path and gateway::configs_init::instances, making them accesible from the variables specified in the class parameters.

Yaml Subkeys to the rescue!

This feature makes extremely easy automatically lookup configurations in hiera. It's common to configure multiple parameters in each file. I suggest organize our puppet modules and hiera for specify each file configuration in a parent yaml object for lookup only for the parent object in the puppet class.

Let's view an example of a custom node configuration yaml with this approach:

gateway:
    gateway1:
        fileservice:
            userid: 'USER'
            serverport: 9999
        properties:  
            sockettimeout: 28800000
        log:
            tracelevel: 'debug'
            
     gateway2:
        fileservice:
            userid: 'USER2'
            serverport: 8888
        properties:  
            sockettimeout: 28800000
        log:
            tracelevel: 'debug'

gateway holds all the instances to configure and gatewa1 and gateway2 are the 2 differents instances to be configured.

I've divided the different parameters in multiple objects, fileservice , properties and log, this objects represents the different files to be configured. In this example we have few configs per file, but as I said, in large configuration files, this approach will save us a lot of time.

Looking up for the parent keys in the Puppet classes

Now, we've to declare the parent objects which represents the different files in the class parameters like this:

define gateway::config_instances (
  $fileservice = {},
  $properties = {},
  $log = {}
) {
  # Using this variables
  # As we are in a define puppet resource called from create_resources
  # we can use the variable $name which hold the name of the current
  # instance
  
  file { "${install_path}\\${name}\\WEB-INF\\classes\\properties\\fileservice.properties":
    ensure             => file,
    content            => template('gateway/fileservice.properties.erb'),
    source_permissions => ignore,
  }
}

With the previous ruby code, the create_resources method declared in the gateway::configs_init class will call this define resource with the different instsances parameters, because create_resources is a Puppet method for implementing recursion.

Writing our ERB template

Now it's time to write our ERB template and make it work with puppet automatic configuration:

/* -- Previous contents -- */
* Connection User ID: 
	The user id for the server connection. */
fileservice.connectionpool.UserId=<%= @fileservice['userid'] %>

/* Private Key file: 
	The file contains the encrypted password for the key repository file. */
fileservice.connectionpool.PrivateFile=pass.key

/* -- More contents -- */

If you read the code, the magic part here is the <%= @fileservice['userid'] %> section. This ruby expression will use the fileservice variable and will acces the userid object to retrieve its value, which will be injected in that code line. This, will generate the following file:

/* -- Previous contents -- */
* Connection User ID: 
	The user id for the server connection. */
fileservice.connectionpool.UserId=USER

/* Private Key file: 
	The file contains the encrypted password for the key repository file. */
fileservice.connectionpool.PrivateFile=pass.key

/* -- More contents -- */

And that's all mates!

This is the way I configure medium-large deployments with Puppet!