Defect #31241
closed
intermittent 500s ActiveRecord::ConnectionTimeoutError
0%
Description
after upgrading from redmine from 3.0-stable to 4.0-stable, i started getting intermittent failures:
ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.005 seconds); all pooled connections were in use): app/models/setting.rb:235:in `check_cache' app/controllers/application_controller.rb:90:in `user_setup'
using uwsgi/nginx - journalctl shows this error:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /issues/227> uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /issues/>
i tried increasing the number of rails db pool connections and timeout, but that did not seem to help
$ bin/about sh: hg: command not found sh: cvs: command not found sh: bzr: command not found Environment: Redmine version 4.0.2.stable Ruby version 2.6.1-p33 (2019-01-30) [x86_64-linux] Rails version 5.2.2 Environment production Database adapter PostgreSQL Mailer queue ActiveJob::QueueAdapters::AsyncAdapter Mailer delivery sendmail SCM: Subversion 1.11.1 Git 2.21.0 Filesystem Redmine plugins: parabola 0.0.1
Related issues
Updated by bill auger over 5 years ago
it often coincides with the firing of an ansynchronous outgoing mail event with 10 or so recipents
Performing ActionMailer::DeliveryJob (Job ID: N) from Async(mailers)
Updated by
Updated by pasquale [:dedalus] about 5 years ago
The same issue here.
Environment: Redmine version 4.0.5.stable Ruby version 2.5.7-p206 (2019-10-01) [x64-mingw32] Rails version 5.2.3 Environment production Database adapter Mysql2 Mailer queue ActiveJob::QueueAdapters::AsyncAdapter Mailer delivery smtp SCM: Mercurial 5.1.2 Git 2.19.1 Filesystem
maybe there is related to #32314?
Updated by Marius BĂLTEANU over 4 years ago
- Status changed from New to Needs feedback
pasquale [:dedalus] wrote:
The same issue here.
[...]
maybe there is related to #32314?
@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?
Andreas Bill auger, please see the notes from #32314#note-15 and #32314#note-16, you should not use in production the AsyncAdapter. Maybe your server remains without memory because of the mail queue (it's just a supposition).
Updated by Marius BĂLTEANU over 4 years ago
- Has duplicate Defect #31363: 500 Internal Server Error added
Updated by pasquale [:dedalus] over 4 years ago
Updated by Marius BĂLTEANU over 4 years ago
pasquale [:dedalus] wrote:
Marius BALTEANU wrote:
pasquale [:dedalus] wrote:
The same issue here.
[...]
maybe there is related to #32314?
@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?
No, It works likes a charm!
Thanks for your quick response.
Andreas Bill auger, please try with another adapter and let us how it works.
Updated by jianmin Zhang over 4 years ago
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!
Updated by Marius BĂLTEANU over 4 years ago
- Status changed from Needs feedback to Closed
- Resolution set to Invalid
jianmin Zhang wrote:
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!
You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).
I'm closing this for now because it seems to be a configuration problem.
Updated by jianmin Zhang over 4 years ago
Marius BALTEANU wrote:
jianmin Zhang wrote:
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!You can change your connection pool from
database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).I'm closing this for now because it seems to be a configuration problem.
Thank you for your advice
Updated by Robin Kluth over 4 years ago
That seem to work, I added pool: 10 to the database.yml. So 5 seems a bit too low for redmine? In some cases?
Would be great, if one could change that via the official docker image variables.
For those with docker:
- "./data/database.yml:/usr/src/redmine/config/database.yml"
and put the changes inside the database.yml.
Updated by Elvis Stansvik about 2 years ago
Hm, is not switching the mail queue adapter or increasing the database connection pool size just workarounds for an issue that should ideally be fixed?
Surely Redmine should not need to keep a bunch of database connections open at the same time just to send out e-mails e.g. when there are many subscribes to an issue, even if the "not for production" mail queue adapter is used? If a database connection needs to be held open while the e-mail is being sent out, then either the same connection should be used or the mails to be sent out should be batched so that they can be sent out with a reasonable number of db conns.