Thin timing out Nginx cannot connect
Added by Andrew Parker over 8 years ago
I have been trying different configurations and cannot figure out why Thin is timing out. At least this is what I think is happening. Thin is not available after the server is idle for a while (i.e. overnight).
Environment:
Ubuntu 14.04 running on AWS t2.nano
Redmine: Redmine 2.6.10.stable.15251
Ruby: 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
Rails: 3.2.22.2
Thin: 1.3.1 codename Triple Espresso
Database: mysql
No Thin error log. The error log in Nginx is:
2016/03/24 07:01:47 [error] 18432#0: *1376 connect() to unix:/ebs_001/redmine/run/thin/redmine.0.sock failed (111: Connection refused) while connecting to upstream, client: 184.166.153.12, server: pm.source3.com, request: "GET / HTTP/1.1", upstream: "http://unix:/ebs_001/redmine/run/thin/redmine.0.sock:/", host: "pm.source3.com"
I can restart Thin (more on this later) and connect without restarting Ngnix.
When I try to stop Thin I get the following message
user1@ip-172-31-18-58:/var/log/nginx$ sudo service thin stop [stop] /etc/thin1.9.1/redmine.yml ... Stopping server on /ebs_001/redmine/run/thin/redmine.0.sock ... Sending QUIT signal to process 18385 ... process not found! Sending KILL signal to process 18385 ... /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `kill': No such process (Errno::ESRCH) from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `force_kill' from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:134:in `rescue in send_signal' from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:118:in `send_signal' from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:107:in `kill' from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:93:in `block in stop' from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:134:in `tail_log' from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:92:in `stop' from /usr/lib/ruby/vendor_ruby/thin/runner.rb:185:in `run_command' from /usr/lib/ruby/vendor_ruby/thin/runner.rb:151:in `run!' from /usr/bin/thin:6:in `<main>'
After I stop Thin, I can then start Thin (sudo service Thin start) and connect to my redmine project without restarting nxinx.
I do not see any error logs in redmine or Thin.
My /etc/thin/redmine.yml file:
--- user: user1 group: group1 pid: /ebs_001/redmine/run/thin/redmine.pid timeout: 30 wait: 30 log: /ebs_001/redmine/logs/thin/redmine.log max_conns: 1024 require: [] environment: production max_persistent_conns: 512 servers: 1 daemonize: true socket: /ebs_001/redmine/run/thin/redmine.sock chdir: /ebs_001/redmine/redmine-2.6 tag: redmine
Portions of my /etc/nginx/sites-available/redmine.conf:
# Upstream Ruby process cluster for load balancing upstream thin_cluster { server unix:/ebs_001/redmine/run/thin/redmine.0.sock; # server unix:/ebs_001/redmine/run/thin/redmine.1.sock max_fails=1 fail_timeout=15s; # server unix:/ebs_001/redmine/run/thin/redmine.2.sock; # server unix:/ebs_001/redmine/run/thin/redmine.3.sock; } ### REDMINE - serve all pages via ssl (https) server { listen 80; server_name pm.source3.com; return 301 https://$host$request_uri; } server { listen 443 ssl; server_name pm.source3.com; ssl on; ssl_certificate /etc/nginx/ssl/redmine.crt; ssl_certificate_key /etc/nginx/ssl/redmine.key; include /etc/nginx/includes/redmine.include; proxy_redirect off; root /ebs_001/redmine/redmine-2.6; # An alias to your upstream app location @cluster { proxy_pass http://thin_cluster; # Define what a "failure" is, so it can try the next server proxy_next_upstream error timeout http_502 http_503; # If the upstream server doesn't respond within n seconds, timeout proxy_read_timeout 60s; } location / { try_files $uri/index.html $uri.html $uri @cluster; } } ...
And the ../includes/redmine.include
proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k;
It is not a permission issue as I can restart Thin and connect to Redmine.
I only have about 15M of free memory. Maybe this is the issue. But if it was, then Redmine would crash as I am using it throughout the day.
Any help on figuring out the timeout is much appreciated.
Replies (1)
RE: Thin timing out Nginx cannot connect - Added by Andrew Parker over 8 years ago
The problem was out of memory on the server. I was running to many applications on a AWS t2.nano. Redmine was the only one crashing. No error logs was a hint it was a memory issue. And the "free -h" command was a big clue.
I was running:- One Django app (running Postgres vis AWS RDS)
- WordPress
- Redmine (running MySQL locally).
Migrate to a AWS t2.mircro for more memory and all is fine.