Mina: problems with restarting puma

Created on 25 Oct 2014  ·  10Comments  ·  Source: mina-deploy/mina

I'm very often left with non-working web app after deploy...

puma somehow dies...

-----> DB schema unchanged; skipping DB migration        
-----> Skipping asset precompilation        
-----> Cleaning up old releases (keeping 5)        
-----> Build finished        
-----> Moving build to releases/42        
-----> Updating the current symlink        
-----> Launching        
       Command phased-restart sent success 
-----> Done. Deployed v42        
       Elapsed time: 3.00 seconds

(here the website is down)

david@eclipse:~/Projects/meetcasts (master)$ mina puma:restart
-----> Loading rbenv
Puma is not running!
[23126] Puma starting in cluster mode...
[23126] * Version 2.9.0 (ruby 2.1.2-p95), codename: Team High Five
[23126] * Min threads: 1, max threads: 16
[23126] * Environment: production
[23126] * Process workers: 2
[23126] * Preloading application
[23126] * Listening on unix:///var/www/meetcasts/shared/tmp/sockets/puma.sock
[23126] * Listening on tcp://0.0.0.0:3003
[23126] * Daemonizing...
Connection to example.com closed.
       Elapsed time: 4.00 seconds

(note the line Puma is not running)

Sometimes I have to repeat mina puma:restart multiple times to get it working...

Is this a puma problem? I have some sinatra apps and when I send SIGUSR2, it always restarts correctly... Maybe on Rails it has problems and it's not a mina issue? How can I get more debug data about this?

Most helpful comment

It's puma issue. The problem i that puma is not actually starts daemon immediately after start. System kills puma process when ssh session is closed. You can fix it by adding sleep 1 after puma start.

For example:

namespace :puma do
  task :custom_start => :environment do
    command %[
      if [ -e '#{fetch(:pumactl_socket)}' ]; then
        echo 'Puma is already running!';
      else
        if [ -e '#{fetch(:puma_config)}' ]; then
          cd '#{fetch(:current_path)}' && #{fetch(:puma_cmd)} -d -e #{fetch(:puma_env)} -C #{fetch(:puma_config)}
          sleep 1
        else
          echo 'Puma config is required'
          exit 1
        fi
      fi
    ]
  end
end

All 10 comments

So you say when your deploying using mina it doesn't run, but as I can see, when you ssh to your server and run mina puma:restart everything seems to be ok? Can you please copy your deploy.rb file (only the deploy task), and can you run deploy with the -v parameter (mina deploy -v) and also write me the response down.

I actually run mina puma:restart locally (so I don't ssh there, but mina does), you probably ment that.

I have to run this repeteatly until it suddenly works (page loads). So again after a normal deploy, the site doesn't load and after running mina puma:restart it says that puma was not running and it is starting it now. Then sometimes page loads but many times I have to repeat up to 5 to 7 times before it works.

Here is the deploy file for one project:
https://gist.github.com/davidhq/fe0decad29bb005e9a6d

I thought it actually happens only on this project, but just now it happened on another.

I have examined puma restarting code

cd #{deploy_to}/#{current_path} && #{pumactl_cmd} -S #{puma_state} phased-restart

and it seems ok.. but it's here that things break in the first place..

I don't know how to proceed in finding the fault, but it looks like it's puma's problem, right?

now I have the webapp in not-running state after a deploy:

david@theta:/var/www/tb/shared/tmp/sockets$ cat puma.state 
---
pid: 30362
config: !ruby/object:Puma::Configuration
  options:
    :min_threads: 1
    :max_threads: 16
    :quiet: true
    :debug: false
    :binds:
    - unix:///var/www/tb/shared/tmp/sockets/puma.sock
    - tcp://0.0.0.0:3007
    :workers: 1
    :daemon: true
    :worker_directory: "/var/www/tb/current"
    :environment: production
    :state: "/var/www/tb/shared/tmp/sockets/puma.state"
    :control_url: unix:///var/www/tb/shared/tmp/sockets/pumactl.sock
    :config_file: config/puma.rb
    :mode: :http
    :worker_timeout: 60
    :preload_app: true
    :rackup: config.ru
    :control_auth_token: 7d18137e9e5e7ea13dc2d3758397e3a
david@theta:/var/www/tb/shared/tmp/sockets$ ps aux | grep 30362
[no process with such PID]

Maybe it has something to do with rsync I'm using? Because my friends who deploy with mina don't have any issues, but they don't use rsync method... I was thinking that because I haven't been excluding /tmp (as I probably should) when rsyncing, that this got on the server (and puma socket and state are there).. but I don't know how it could be in the way because symlinking to /shared/tmp is working and it happens before puma restart.

It wasn't that because today it happened again and those directories were excluded... I also upgraded puma from 2.9.0 to 2.9.2... it's a bit frustrating :( Should I ask at Puma's side as well?

I now tried replacing invoke :'puma:phased_restart' with:

queue "kill -SIGUSR2 `ps aux | grep tcp://0.0.0.0:3003 | grep puma | awk '{ print $2 }'`"

And it happened again on 3rd deploy... so it seems that's not a mina problem in any way...

I have some new info.... https://github.com/puma/puma/issues/598 ... still not sure where is the problem but apparently puma rejects phased restart and after normal restart it cannot find the Gemfile for some reason...

Turns out you cannot use phased restart with preloading the app... I hope it will be ok now and that this helps others ... more about this: https://github.com/puma/puma/blob/master/DEPLOYMENT.md#restarting

It's puma issue. The problem i that puma is not actually starts daemon immediately after start. System kills puma process when ssh session is closed. You can fix it by adding sleep 1 after puma start.

For example:

namespace :puma do
  task :custom_start => :environment do
    command %[
      if [ -e '#{fetch(:pumactl_socket)}' ]; then
        echo 'Puma is already running!';
      else
        if [ -e '#{fetch(:puma_config)}' ]; then
          cd '#{fetch(:current_path)}' && #{fetch(:puma_cmd)} -d -e #{fetch(:puma_env)} -C #{fetch(:puma_config)}
          sleep 1
        else
          echo 'Puma config is required'
          exit 1
        fi
      fi
    ]
  end
end
Was this page helpful?
0 / 5 - 0 ratings

Related issues

thaichor picture thaichor  ·  8Comments

sosedoff picture sosedoff  ·  9Comments

Epigene picture Epigene  ·  4Comments

Bilge picture Bilge  ·  9Comments

thelucid picture thelucid  ·  8Comments