Redmine

Recently, I often get "500 Internal Server Error" on Redmine.
Usually, when I call this page: https://redmine/issues?query_id=xx

Few days earlier, I did an apt update && apt upgrade without major change, either for lighttpd / fastcgi / ruby.

$ dpkg -l | egrep "(ruby|lighttpd|fcgi)" 
ii  libruby3.0:amd64                       3.0.2-7ubuntu2.4                        amd64        Libraries necessary to run Ruby 3.0
ii  lighttpd                               1.4.63-1ubuntu3.1                       amd64        fast webserver with minimal memory footprint
ii  lighttpd-mod-deflate                   1.4.63-1ubuntu3.1                       amd64        HTTP response compression module for lighttpd
rc  lighttpd-mod-magnet                    1.4.55-1ubuntu1.20.04.2                 amd64        control the request handling module for lighttpd
ii  lighttpd-mod-openssl                   1.4.63-1ubuntu3.1                       amd64        TLS support using OpenSSL module for lighttpd
ii  lighttpd-modules-lua                   1.4.63-1ubuntu3.1                       amd64        LUA-based modules for lighttpd
ii  rake                                   13.0.6-2                                all          ruby make-like utility
ii  ruby                                   1:3.0~exp1                              amd64        Interpreter of object-oriented scripting language Ruby (default version)
ii  ruby-dev:amd64                         1:3.0~exp1                              amd64        Header files for compiling extension modules for Ruby (default version)
ii  ruby-net-telnet                        0.1.1-2                                 all          telnet client library
ii  ruby-rubygems                          3.3.5-2                                 all          Package management framework for Ruby
ii  ruby-webrick                           1.7.0-3                                 all          HTTP server toolkit in Ruby
ii  ruby-xmlrpc                            0.3.2-1ubuntu0.1                        all          XMLRPC library for Ruby
ii  ruby3.0                                3.0.2-7ubuntu2.4                        amd64        Interpreter of object-oriented scripting language Ruby
ii  ruby3.0-dev:amd64                      3.0.2-7ubuntu2.4                        amd64        Header files for compiling extension modules for the Ruby 3.0
ii  ruby3.0-doc                            3.0.2-7ubuntu2.4                        all          Documentation for Ruby 3.0
ii  rubygems-integration                   1.18                                    all          integration of Debian Ruby packages with Rubygems
ii  libfcgi-bin                            2.4.2-2build2                           amd64        FastCGI bridge from CGI
ii  libfcgi-dev:amd64                      2.4.2-2build2                           amd64        header files of FastCGI
ii  libfcgi-perl:amd64                     0.82+ds-1build1                         amd64        helper module for FastCGI
ii  libfcgi0ldbl:amd64                     2.4.2-2build2                           amd64        shared library of FastCGI
ii  spawn-fcgi                             1.6.4-2                                 amd64        FastCGI process spawner

I found this in the lighttpd error log:

2023-11-07 10:59:09: (gw_backend.c.2398) response not received, request sent: 2001 on socket: unix:/var/tmp/lighttpd/rails-application.sock-0 for /dispatch.fcgi?query_id=61, closing connection
2023-11-07 10:59:13: (gw_backend.c.373) child exited: 1 unix:/var/tmp/lighttpd/rails-application.sock-0

and the syslog has:

Nov  7 10:59:13 server lighttpd[3997811]: /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:89:in `flush': unknown error (syscall error) (FCGI::Stream::Error)
Nov  7 10:59:13 server lighttpd[3997811]: #011from /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:89:in `send_headers'
Nov  7 10:59:13 server lighttpd[3997811]: #011from /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:70:in `serve'
Nov  7 10:59:13 server lighttpd[3997811]: #011from /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:28:in `block in run'
Nov  7 10:59:13 server lighttpd[3997811]: #011from /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:27:in `each'
Nov  7 10:59:13 server lighttpd[3997811]: #011from /opt/redmine-5.0.6/vendor/cache/ruby/3.1.0/gems/rack-2.2.8/lib/rack/handler/fastcgi.rb:27:in `run'
Nov  7 10:59:13 server lighttpd[3997811]: #011from /var/www/redmine/public/dispatch.fcgi:22:in `<main>'

Here is the content of /var/www/redmine/public/dispatch.fcgi :

#!/usr/local/rvm/rubies/default/bin/ruby

require File.dirname(__FILE__) + '/../config/boot'
require File.dirname(__FILE__) + '/../config/environment'
require 'rubygems'
require 'fcgi'

class Rack::PathInfoRewriter
  def initialize(app)
    @app = app
  end

  def call(env)
    env.delete('SCRIPT_NAME')
    parts = env['REQUEST_URI'].split('?')
    env['PATH_INFO'] = parts[0]
    env['QUERY_STRING'] = parts[1].to_s
    @app.call(env)
  end
end

Rack::Handler::FastCGI.run Rack::PathInfoRewriter.new(RedmineApp::Application)

on my /opt/redmine-5.0.6/Gemfile.local

gem 'fcgi', :source => 'https://rubygems.org/'
gem 'sidekiq', '<7'