Project

General

Profile

Thin timing out Nginx cannot connect

Added by Andrew Parker over 8 years ago

I have been trying different configurations and cannot figure out why Thin is timing out. At least this is what I think is happening. Thin is not available after the server is idle for a while (i.e. overnight).

Environment:
Ubuntu 14.04 running on AWS t2.nano
Redmine: Redmine 2.6.10.stable.15251
Ruby: 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
Rails: 3.2.22.2
Thin: 1.3.1 codename Triple Espresso
Database: mysql

No Thin error log. The error log in Nginx is:

2016/03/24 07:01:47 [error] 18432#0: *1376 connect() to unix:/ebs_001/redmine/run/thin/redmine.0.sock failed (111: Connection refused) while connecting to upstream, client: 184.166.153.12, server: pm.source3.com, request: "GET / HTTP/1.1", upstream: "http://unix:/ebs_001/redmine/run/thin/redmine.0.sock:/", host: "pm.source3.com" 

I can restart Thin (more on this later) and connect without restarting Ngnix.

When I try to stop Thin I get the following message

user1@ip-172-31-18-58:/var/log/nginx$ sudo service thin stop
[stop] /etc/thin1.9.1/redmine.yml ...
Stopping server on /ebs_001/redmine/run/thin/redmine.0.sock ... 
Sending QUIT signal to process 18385 ... 
process not found!
Sending KILL signal to process 18385 ... 
/usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `kill': No such process (Errno::ESRCH)
    from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `force_kill'
    from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:134:in `rescue in send_signal'
    from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:118:in `send_signal'
    from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:107:in `kill'
    from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:93:in `block in stop'
    from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:134:in `tail_log'
    from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:92:in `stop'
    from /usr/lib/ruby/vendor_ruby/thin/runner.rb:185:in `run_command'
    from /usr/lib/ruby/vendor_ruby/thin/runner.rb:151:in `run!'
    from /usr/bin/thin:6:in `<main>'

After I stop Thin, I can then start Thin (sudo service Thin start) and connect to my redmine project without restarting nxinx.
I do not see any error logs in redmine or Thin.

My /etc/thin/redmine.yml file:

---
user: user1
group: group1
pid: /ebs_001/redmine/run/thin/redmine.pid
timeout: 30
wait: 30
log: /ebs_001/redmine/logs/thin/redmine.log
max_conns: 1024
require: []
environment: production
max_persistent_conns: 512
servers: 1
daemonize: true
socket: /ebs_001/redmine/run/thin/redmine.sock
chdir: /ebs_001/redmine/redmine-2.6
tag: redmine

Portions of my /etc/nginx/sites-available/redmine.conf:

# Upstream Ruby process cluster for load balancing
upstream thin_cluster {
    server unix:/ebs_001/redmine/run/thin/redmine.0.sock;
    # server unix:/ebs_001/redmine/run/thin/redmine.1.sock max_fails=1 fail_timeout=15s;
    # server unix:/ebs_001/redmine/run/thin/redmine.2.sock;
    # server unix:/ebs_001/redmine/run/thin/redmine.3.sock;
}

### REDMINE - serve all pages via ssl (https)

server {
    listen 80;
    server_name pm.source3.com;
    return 301 https://$host$request_uri;
}

server {
    listen       443 ssl;
    server_name  pm.source3.com;
    ssl on;
    ssl_certificate /etc/nginx/ssl/redmine.crt;
    ssl_certificate_key /etc/nginx/ssl/redmine.key;

    include /etc/nginx/includes/redmine.include;
    proxy_redirect off;
    root   /ebs_001/redmine/redmine-2.6;

    # An alias to your upstream app
    location @cluster {
        proxy_pass http://thin_cluster;
        # Define what a "failure" is, so it can try the next server
        proxy_next_upstream error timeout http_502 http_503;
        # If the upstream server doesn't respond within n seconds, timeout
        proxy_read_timeout 60s;
    }    

    location / {
        try_files $uri/index.html $uri.html $uri @cluster;
    }
}
...

And the ../includes/redmine.include

    proxy_set_header   Host $http_host;                                                                                          
    proxy_set_header   X-Real-IP $remote_addr;                                                                                   
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;

    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;

It is not a permission issue as I can restart Thin and connect to Redmine.

I only have about 15M of free memory. Maybe this is the issue. But if it was, then Redmine would crash as I am using it throughout the day.

Any help on figuring out the timeout is much appreciated.


Replies (1)

RE: Thin timing out Nginx cannot connect - Added by Andrew Parker over 8 years ago

The problem was out of memory on the server. I was running to many applications on a AWS t2.nano. Redmine was the only one crashing. No error logs was a hint it was a memory issue. And the "free -h" command was a big clue.

I was running:
  • One Django app (running Postgres vis AWS RDS)
  • WordPress
  • Redmine (running MySQL locally).

Migrate to a AWS t2.mircro for more memory and all is fine.

    (1-1/1)