Since some days we had been experiencing some weird behaviours with our internal environments deployed with Puppet. The issues appeared when we updated a custom module with its own types, providers and functions.
Somehow the Puppet agent doesn't get adviced the module had changed the types definition and started to show errors about parameters not defined in the type:
We "solved" it restarting the Puppet server service on the master node, but soon the problems started to happen again and we didn't know why.
After a exhaustive research we found that if we launched an agent with a previous version of the updated module, every else node with the newer version couldn't be capable of found the new params.
Puppet official documentation references the environment bleed action when the same custom type has different type definitions in different environments, which causes Ruby cached the first run type metadata and use it for every environment, causing the issue described. You can check more about this here: https://puppet.com/docs/puppet/latest/environment_isolation.html
Generating static types for every environment! Puppet offers a command for statically generate the types definitions for custom types on a given environment with the following command:
sudo -u puppet /opt/puppetlabs/bin/puppet generate types --confdir /etc/puppetlabs/puppet --codedir /etc/puppetlabs/code --vardir /opt/puppetlabs/puppet/cache --environment 2019_u3
So, as we use r10k triggered by a webhook the simplest solution is to add this snippet at the end of our webhook-launched script for every environment under
for d in `cd /etc/puppetlabs/code/environments && ls -d *` ; do echo "Generating static types for $d environment" /opt/puppetlabs/bin/puppet generate types --confdir /etc/puppetlabs/puppet --codedir /etc/puppetlabs/code --vardir /opt/puppetlabs/puppet/cache --environment "$d" done