Defect #31241
closed
intermittent 500s ActiveRecord::ConnectionTimeoutError
Added by bill auger over 5 years ago.
Updated over 2 years ago.
Description
after upgrading from redmine from 3.0-stable to 4.0-stable, i started getting intermittent failures:
ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.005 seconds); all pooled connections were in use):
app/models/setting.rb:235:in `check_cache'
app/controllers/application_controller.rb:90:in `user_setup'
using uwsgi/nginx - journalctl shows this error:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /issues/227>
uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /issues/>
i tried increasing the number of rails db pool connections and timeout, but that did not seem to help
$ bin/about
sh: hg: command not found
sh: cvs: command not found
sh: bzr: command not found
Environment:
Redmine version 4.0.2.stable
Ruby version 2.6.1-p33 (2019-01-30) [x86_64-linux]
Rails version 5.2.2
Environment production
Database adapter PostgreSQL
Mailer queue ActiveJob::QueueAdapters::AsyncAdapter
Mailer delivery sendmail
SCM:
Subversion 1.11.1
Git 2.21.0
Filesystem
Redmine plugins:
parabola 0.0.1
it often coincides with the firing of an ansynchronous outgoing mail event with 10 or so recipents
Performing ActionMailer::DeliveryJob (Job ID: N) from Async(mailers)
The same issue here.
Environment:
Redmine version 4.0.5.stable
Ruby version 2.5.7-p206 (2019-10-01) [x64-mingw32]
Rails version 5.2.3
Environment production
Database adapter Mysql2
Mailer queue ActiveJob::QueueAdapters::AsyncAdapter
Mailer delivery smtp
SCM:
Mercurial 5.1.2
Git 2.19.1
Filesystem
maybe there is related to #32314?
- Status changed from New to Needs feedback
pasquale [:dedalus] wrote:
The same issue here.
[...]
maybe there is related to #32314?
@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?
Andreas Bill auger, please see the notes from #32314#note-15 and #32314#note-16, you should not use in production the AsyncAdapter. Maybe your server remains without memory because of the mail queue (it's just a supposition).
Marius BALTEANU wrote:
pasquale [:dedalus] wrote:
The same issue here.
[...]
maybe there is related to #32314?
@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?
No, It works likes a charm!
pasquale [:dedalus] wrote:
Marius BALTEANU wrote:
pasquale [:dedalus] wrote:
The same issue here.
[...]
maybe there is related to #32314?
@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?
No, It works likes a charm!
Thanks for your quick response.
Andreas Bill auger, please try with another adapter and let us how it works.
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!
- Status changed from Needs feedback to Closed
- Resolution set to Invalid
jianmin Zhang wrote:
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!
You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).
I'm closing this for now because it seems to be a configuration problem.
Marius BALTEANU wrote:
jianmin Zhang wrote:
Hello,
The error dispaly that “all pooled connections were in use”，Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000，The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!
You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).
I'm closing this for now because it seems to be a configuration problem.
Thank you for your advice
That seem to work, I added pool: 10 to the database.yml. So 5 seems a bit too low for redmine? In some cases?
Would be great, if one could change that via the official docker image variables.
For those with docker:
- "./data/database.yml:/usr/src/redmine/config/database.yml"
and put the changes inside the database.yml.
Hm, is not switching the mail queue adapter or increasing the database connection pool size just workarounds for an issue that should ideally be fixed?
Surely Redmine should not need to keep a bunch of database connections open at the same time just to send out e-mails e.g. when there are many subscribes to an issue, even if the "not for production" mail queue adapter is used? If a database connection needs to be held open while the e-mail is being sent out, then either the same connection should be used or the mails to be sent out should be batched so that they can be sent out with a reasonable number of db conns.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by 
Updated by 
Updated by 
Updated by