asemanfar - a blog about programming

Monitoring Unicorn with Bluepill

November 06, 2009

At Serious Business, we recently switched to using unicorn and bluepill. One of the features we implemented in bluepill was the ability to monitor child processes. This was built with Unicorn's workers in mind. Here's the config file we're using to monitor our Unicorn workers:

   1  Bluepill.application("app_name-production") do |app|
   2    app.process("unicorn") do |process|
   3      process.pid_file = File.join(RAILS_ROOT, 'tmp', 'pids', 'unicorn.pid')
   4      process.working_dir = RAILS_ROOT
   5    
   6      process.start_command = "unicorn -Dc config/unicorn.rb -E production"
   7      process.stop_command = "kill -QUIT {{PID}}"
   8      process.restart_command = "kill -USR2 {{PID}}"
   9    
  10      process.uid = process.gid = 'deploy'
  11    
  12      process.start_grace_time = 8.seconds
  13      process.stop_grace_time = 5.seconds
  14      process.restart_grace_time = 13.seconds
  15    
  16    
  17      process.monitor_children do |child_process|
  18        child_process.stop_command = "kill -QUIT {{PID}}"
  19      
  20        child_process.checks :mem_usage, :every => 10.seconds, :below => 150.megabytes, :times => [3,4], :fires => :stop
  21        child_process.checks :cpu_usage, :every => 10.seconds, :below => 20, :times => [3,4], :fires => :stop
  22      end
  23    end
  24  end

One new feature we recently implemented was the support for working_dir, similar to god's config. There is one important difference though: we also set the PWD environmental variable. This is used to reload the unicorn app with a USR2 signal. In order for Unicorn to properly reload the app with the new current symlink, the env variable needs to be set.

Typically, when you chdir into a symlink'd folder, you are stuck in whatever the symlimk referenced, but if we pass the symlink as an environmental variable, unicorn chdir's into the symlink. The alternative would be to put the full path to the config file in the start command.

The montioring of child processes is still fairly new, as is all of bluepill, so please report any bugs you encounter to the github repos' issue page.

Read more about Bluepill.

Comments

posted by Michael Guterl on 12/01/09 03:09 PM UTC

I'm not sure, but I think http://www.mail-archive.com/mongrel-unicorn@rubyforge.org/msg00163.html resolves some of the issues with the symlink'd relative path to one's configuration file.

posted by Arya Asemanfar on 12/01/09 06:22 PM UTC

It does indeed solve a problem with the relative path, but specifically with the config, not the working directory that the process things the application code is in. Unicorn captures the value of the environmental variable PWD and then uses that when it re-execs instead of just Dir.pwd (which is absolute non-symlink path). So setting the symlink version of the working directory in the ENV variable allows unicorn to chdir into the correct directory when you re-deploy.

posted by david on 04/07/10 05:20 AM UTC

what's the command to restart the application given this config? in fact, it doesn't stop/restart when I do bluepill stop/restart application

posted by Arya Asemanfar on 04/07/10 05:32 AM UTC

@david sudo bluepill restart unicorn

posted by david on 04/07/10 06:26 AM UTC

my bad, it was a problem with the process.working_dir, it's working very well now :)

posted by david on 04/07/10 08:47 PM UTC

i am wondering.. how do you rotate the bluepill logs ? logrotate?

posted by Arya Asemanfar on 04/08/10 04:32 AM UTC

Yep, logrotate. It's described a little in the README at http://github.com/arya/bluepill

posted by david on 04/09/10 11:02 PM UTC

thanks Arya!

last question I promise.

when I restart the process I can see there's a process which says "... (old)". am I supposed to kill it manually? or what?

posted by Arya Asemanfar on 04/17/10 08:32 PM UTC

You mean where it says "killing previous bluepill[1234]"? I'm not sure which "(old)" you're referring to? Can you paste the command you run and its output to see that?


Leave a Comment