Tom Duckering

Icon

a web log of my technical stumblings

Puppet Doesn’t Play Nice With Really Big Files

To kick things off, I will share some of my learnings about puppet. To set the scene I’m using it with a very small number (<10) of AWS EC2 instances. Given a vanilla EC2 instance running a rightscale CentOS image I do a quick set up of puppet:

yum install -y puppet
echo "server=<hostname of my puppet master>" >> /etc/puppet/puppet.conf
puppetd --test

This first invocation of puppet will not do anything as there is no signed certificate on the puppetmaster. So I jump back to my puppet master server to check for the newly received, but as yet unsigned, certificate:

puppetca --list
Find the unsigned cert and sign it with:
puppetca --sign <cert name>

Back on the new instance I call puppet again:
puppetd --test

Now with the cert signed it can do its thing. It always seems to run ok the first time, then subsequent runs are subject to “random” errors. I think it’s down to the fact that I’m getting puppet to serve up a couple of reasonably large installation binaries (c. 350MB)

file { "/var/src/biginstaller" :
source => "puppet://my_puppet_master.compute.amazonaws.com/infra/config/biginstaller",
mode => 755,
owner => root,
group => root
}

My suspicion – based on some googling – is that my puppet master is running low on resources as puppet computes the hash of any files it serves. It does this, sensibly, to see if it needs to refresh that file, but doesn’t have to on the first go as the file is initially absent. However I understand that it computes this hash – naively – by loading the entire file into memory – it seems that this has a knock on effect on the machine (which is an m1.small) and the file server starts to time out.

So I have now switched the installers to be fetched from an AWS S3 bucket using an exec class using wget – like so:

exec { "Fetch Big Installer" :
path => "/usr/bin:/usr/sbin/:/bin",
cwd => "/var/src",
command => "wget --no-check-certificate https://s3.amazonaws.com/my-installers/biginstaller",
creates => "/var/src/bigintstaller",
require => File["/var/src"]
}

It seems to be working ok so far. I’m keeping other bits of config that are a bit more tailored than installers to be served using the normal puppet route. This keeps my custom config and other small scripts under slightly tighter control.

Filed under: devops,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: