Project

General

Profile

Crash frequently

Added by dean chen over 11 years ago

My redmine server crashes a few times every day. Don't know why. Below is the records from production.log when accessing /issue/14, it crashed again.

Completed 200 OK in 61ms (Views: 34.4ms | ActiveRecord: 9.9ms)
Started GET "/issues/14" for 10.112.18.179 at 2013-07-31 09:50:47 +0800
Processing by IssuesController#show as HTML
Parameters: {"id"=>"14"}
Current user: shu6889 (id=3)
Rendered issues/_action_menu.html.erb (8.2ms)
Rendered issue_relations/_form.html.erb (3.0ms)
Rendered issues/_relations.html.erb (5.2ms)
Rendered issues/_action_menu.html.erb (5.5ms)
Rendered issues/_form_custom_fields.html.erb (0.1ms)
Rendered issues/_attributes.html.erb (47.8ms)
Rendered issues/_form.html.erb (100.4ms)
Rendered attachments/_form.html.erb (3.3ms)
Rendered issues/_edit.html.erb (123.9ms)
Rendered issues/_sidebar.html.erb (126.5ms)
Rendered watchers/_watchers.html.erb (8.1ms)
Rendered issues/show.html.erb within layouts/base (456.9ms)
Rendered plugins/redmine_backlogs/app/views/hooks/_rb_include_scripts.html.erb (7.2ms)
Completed 200 OK in 858ms (Views: 715.3ms | ActiveRecord: 89.8ms)
Connecting to database specified by database.yml
OpenIdAuthentication.store is nil. Using in-memory store.
Creating scope :system. Overwriting existing method Enumeration.system.
Creating scope :sorted. Overwriting existing method User.sorted.
Creating scope :sorted. Overwriting existing method Group.sorted.

My redmine info:

Information

Redmine 2.3.2.stable.12037

Default administrator account changed
Attachments directory writable
Plugin assets directory writable
RMagick available (optional)

Environment:
Redmine version 2.3.2.stable.12037
Ruby version 1.9.3-p194 (2012-04-20) [x86_64-linux]
Rails version 3.2.13
Environment production
Database adapter Mysql2
Redmine plugins:
redmine_backlogs v1.0.4


Replies (6)

RE: Crash frequently - Added by dean chen over 11 years ago

Any idea for help me to profile this problem?
Thanks.

RE: Crash frequently - Added by Jan Niggemann (redmine.org team member) over 11 years ago

What's the problem, I can't spot the error? What do you mean "My redmine server crashes", is it the system, the webserver, the application server, ...?

RE: Crash frequently - Added by dean chen over 11 years ago

I am using Nginx, Nginx talks to redmine through FastCGI.
OS, Nginx are stable, just fascgi process sometimes is dead.

2013/07/31 09:50:48 [error] 1075#0: *83 connect() failed (111: Connection refused) while connecting to upstream, client: 10.112.18.179, server: localhost, request: "GET /favico\
n.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9001", host: "10.112.18.45"

Below is my /etc/init.d/redmine script:

#!/bin/sh                                                                                                                                                                        

### BEGIN INIT INFO                                                                                                                                                              
# Provides:     Dean Chen                                                                                                                                                        
# Required-Start:                                                                                                                                                                
# Required-Stop:                                                                                                                                                                 
# Default-Start:        2 3 4 5                                                                                                                                                  
# Default-Stop:         0 1 6                                                                                                                                                    
# Short-Description: redmine                                                                                                                                                     
# Description: Redmine+Backlogs server                                                                                                                                           
### END INIT INFO                                                                                                                                                                

. /lib/lsb/init-functions

SPAWN_FCGI=/usr/bin/spawn-fcgi

case "$1" in
  start)
     log_begin_msg "Starting redmine server" 
     $SPAWN_FCGI -p 9001 -f /root/redmine-2.3/public/dispatch.fcgi
     log_end_msg 0
     ;;
  stop)
     PID=`ps -ef | grep 'fcgi' | grep -v grep | awk '{print $2}'`
     log_begin_msg "Stopping redmine server" 
     if [ ! -z "$PID" ]; then
        kill -15 $PID
     fi
     log_end_msg 0
     ;;
  restart)
     $0 stop
     $0 start
     ;;
  *)
     log_success_msg "Usage: service redmine {start|stop|restart}" 
     exit 1
esac

exit 0

RE: Crash frequently - Added by Jan Niggemann (redmine.org team member) over 11 years ago

I don't use nginx, but I suspect the problem is there and not in redmine itself,,,

RE: Crash frequently - Added by dean chen over 11 years ago

From what I have seen, the dispatch.fcgi process is terminated without any log.
what can I do to find more detail logs to analyse this problem?
In default.conf from Nginx, my configuration about FastCGI looks below:

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9001                                                                                                         
    location / {                                                                                                                                            
    fastcgi_pass   127.0.0.1:9001;                                                                                                                                               
    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    include        fastcgi_params;
    }

RE: Crash frequently - Added by dean chen over 11 years ago

I figured out it.
It's about spawn-fcgi, it has -u argument, if I specify -u root, it rejects this and outputs one message:
spawn-fcgi: I will not set uid to 0

If do not use -u argument, it launches dispatch.fcgi successfully, but the process will crash frequently.

I just modified the spawn-fcgi source codes to remove the restriction and tried for a few hours.

It's stable now.

And I will create a user for redmine instead of root when I am free, because I am not gonna be a bad administrator. :)

    (1-6/6)