Defect #633

Redmine crashes randomly

Added by Liang Jin over 12 years ago. Updated over 7 years ago.

Status:ClosedStart date:2008-02-12
Priority:NormalDue date:
Assignee:-% Done:

50%

Category:-
Target version:-
Resolution:Invalid Affected version:

Description

I am running Redmine trunk version (r1108) with Mongrel_cluster.

Randomly the redmine process will hang up and stop responding. Restarting the mongrel_cluster gets back the site. The following error message was found in the mongrel log:

Reaping 1 threads for slow workers because of 'shutdown'
Waiting for 1 requests to finish, could take 60 seconds.

Any suggestions on how to eliminate the issue other than restarting the mongrel cluster with a cron job?

Operation System: Debian Etch


Related issues

Related to Redmine - Defect #7699: Subversion: 500 Internal Server Error when browsing a rep... Reopened 2011-02-23
Duplicated by Redmine - Defect #2684: Redmine Crashing on Ubuntu System Closed 2009-02-05
Duplicated by Redmine - Defect #2514: Memory usage grow up and cpu 100% Closed 2009-01-15

History

#1 Updated by Thomas Lecavelier over 12 years ago

Hi, Liang.

This is Mongrel issue, but maybe we could help you a little by locating the problem more precisely. I suppose you have an apache or whatever httpd server in front of your cluster: could you have a look into your httpd server logs: you should be able to discover which access failed. If it's a particular page, maybe it could be an indication of a too demanding content.

Thank you.

#2 Updated by Liang Jin over 12 years ago

Thanks for the tip, Thomas.
I found out that the problem comes from the PDF generation for specific issues. It works on some issues, but not others. Once the error occurs, the Mongrel will consume 100% of the CPU.

Any further ideas?

-Liang

@Processing IssuesController#show (for 66.249.72.198 at 2008-02-12 06:00:27) [GET]
Session ID: d5e4cc350eecdb3ef689a4c84949f079
Parameters: {"format"=>"pdf", "action"=>"show", "id"=>"16", "controller"=>"issues"}
Rendering issues/show.rfpdf

ActionView::TemplateError (Mongrel timed out this thread: shutdown) in app/views/issues/_pdf.rfpdf:

/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `MultiCell'
(erb):63:in `render'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:69:in `render'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `instance_eval'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `render'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:419:in `delegate_render'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:299:in `render_template'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:260:in `render_file'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:275:in `render'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/partials.rb:59:in `render_partial'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:30:in `benchmark'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/partials.rb:58:in `render_partial'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:287:in `render'
(erb):7:in `render'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:69:in `render'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `instance_eval'
/var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `render'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:419:in `delegate_render'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:299:in `render_template'
/var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:260:in `render_file'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:812:in `render_file'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:747:in `render_with_no_layout'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/layout.rb:256:in `render_without_benchmark'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:50:in `render'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:50:in `render'
/var/rails/redmine/app/controllers/issues_controller.rb:94:in `show'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:135:in `call'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:135:in `custom'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:167:in `call'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:167:in `respond'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:161:in `each'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:161:in `respond'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:105:in `respond_to'
/var/rails/redmine/app/controllers/issues_controller.rb:92:in `show'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:1101:in `send'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:1101:in `perform_action_without_filters'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:696:in `call_filters'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:688:in `perform_action_without_benchmark'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/rescue.rb:83:in `perform_action'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:435:in `send'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:435:in `process_without_filters'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:684:in `process_without_session_management_support'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/session_management.rb:114:in `process'
/var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:334:in `process'
/var/rails/redmine/vendor/rails/railties/lib/dispatcher.rb:41:in `dispatch'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in `process'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `process'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in `process_client'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `each'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `process_client'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `initialize'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `new'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `initialize'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `new'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:282:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `each'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb:212:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load'
/usr/bin/mongrel_rails:19

#3 Updated by Thomas Lecavelier over 12 years ago

rfpdf is a nightmare... :(

The view crashed will rendering the content of a cell. According to your name, are you using redmine with a east charset? UTF-8? Could you try to reproduce it on a issue using only ASCII chars, just to be sure that's the charset that provoke the crash.

Thank you.

#4 Updated by Liang Jin over 12 years ago

I have tried with English only chars and it works fine. The problem only occurs with issues with Chinese characters.
However, for some issues with Chinese characters, the PDF is rendered successfully. Although the characters are not correct in the PDF (font?).

It is a strange problem.

#5 Updated by Liang Jin over 12 years ago

Since I know that it is the PDF export that is causing the crash. Would it make sense to add in an option in Redmine to allow or disallow PDF export globally, before the issue is solved? I don't see why many people would like the PDF function (it is nice to have, but not a show stopper).

#6 Updated by Liang Jin over 12 years ago

I have disabled PDF export in the views. However, redmine crashed again with a different issue. It reported out of memory! One process of Mongrel was stuck and consumed 100% of CPU and 800MB of RAM!

This out of memory error only occurs in the repository viewing section:

Errno::ENOMEM (Cannot allocate memory - svn list --xml ...

#7 Updated by Thomas Lecavelier over 12 years ago

Hum... This kind of ENOMEM could be the result of a bad process ending: how do you serve your mongrel_cluster? Is it thought lighttpd or other?

#8 Updated by Liang Jin over 12 years ago

I have Nginx in front of mongrel_cluster.

#9 Updated by Thomas Lecavelier over 12 years ago

Damn, nginx is not known as a great memory eater :-/

Could you run the svn list --xml and say how many lines it output?

#10 Updated by Liang Jin over 12 years ago

First of all, this is running on a VPS with 1GB of total RAM (that is why I get the out of memory when Mongrel is using around 800MB). the "svn list --xml" command only outputed less than 40 lines, so I don't see this as a problem. However, I cannot find in the log what really caused the memory issue in the redmine instance. Is there any suggested way of debugging such problems? Thanks.

#11 Updated by Thomas Lecavelier over 12 years ago

At that point, the best thing to do is to search on the mongrel side. But I find suspicious that your mongrels eat so much RAM: a mongrel eat a lot of it, but yours seem just over-weight! How many user do you have at once and how many mongrels instance have you setup in your cluster?

#12 Updated by Liang Jin over 12 years ago

I have only two instances of mongrel, and this morning, one is stuck at 100%, but the other one is totally fine.

Well, I think this might be still related to the PDF bug. In the production.log file, I did find out that I haven't completely removed the PDF export feature. I am sure that there is no PDF export link on the pages anymore, but I guess some spiders or robots are accessing the links according to their caches. Again, is there a way to completely disable PDF export in RedMine? Thanks.

Here is the error in the log:
ActionView::TemplateError (Mongrel timed out this thread: shutdown) in app/views/issues/_pdf.rfpdf:

#13 Updated by Jean-Philippe Lang over 12 years ago

I can not help much since I don't use Mongrel.
This site in running Apache2 + fcgid without problems for now :-)

If you want to disable PDF exports, you can comment the 2 lines in app/controllers/issues_controller.rb that start with (lines 66 and 97 in latest revision):

format.pdf { send_data...

Spiders asking for the pdf will get the html version.

#14 Updated by Liang Jin over 12 years ago

  • % Done changed from 0 to 50

Thanks JP. I have disabled PDF exports as instructed, and will wait to see if there is any other issues.

I will also try to trouble shoot the PDF problem (it only crashes on certain pages with Chinese characters).

#15 Updated by Yuri Leikind over 12 years ago

Hello,

I think we are experiencing the same problem.

  • We are using a mongrel cluster behind apache
  • We only use the Issues functionality
  • We do not use PDF export (is there any PDF work taking place behind the scenes? I do not think so)
  • There are just ~4-5 users

After some time after a restart the application becomes extremely irresponsive, sometimes leading to Apache not being able to wait till the output from mongrels. At this point a mongrel process is eating almost all CPU resources. This is definitely not an SMTP issue, I checked it, and it happens regularly after some time after a restart.

I installed a ruby-prof Rails plugin to see where this is happening, but so far none of these long requests has produced a ruby-prof report - request processing just does not end.

#16 Updated by Liang Jin over 12 years ago

Hi, Yuri

Have you tried the JP's solution to disable PDF exports entirely by modifying app/controllers/issues_controller.rb ? Also, check your Ruby-MySQL version (discussion on the forum: http://rubyforge.org/forum/message.php?msg_id=25951)

-Liang

#17 Updated by Jean-Philippe Lang about 12 years ago

  • Status changed from New to Closed
  • Resolution set to Invalid

Reopen if needed.

#18 Updated by Heinz Gies almost 12 years ago

  • Status changed from Closed to Reopened

I'd like to reopen this, as I believe the underlaying problem sitl exists (trunk r1715). It is not related to PDF at all from what I can tell but tom big projects and SVN. I put a big (20+MB) project in a svn repo in my redmine installation and after visiting that repo a few times or looking at big files the repository view dies with the exact same error

This as well is running in a Mongrel Web Server 1.1.5 with a Apache/2.2.3 as fronted proxy, on a Debian etch system. The memory usage was high but not as high as reported before about 250MB when the server first went down. A simple shutdown and restart solves this temporarily.

The SVN info are hardly 20 lines so I doubt that is too much too handle.

Errno::ENOMEM (Cannot allocate memory - svn info --xml ....
    /lib/redmine/scm/adapters/abstract_adapter.rb:175:in `popen'
    /lib/redmine/scm/adapters/abstract_adapter.rb:175:in `shellout'
    /lib/redmine/scm/adapters/abstract_adapter.rb:165:in `shellout'
    /lib/redmine/scm/adapters/subversion_adapter.rb:53:in `info'
    /app/models/repository/subversion.rb:44:in `fetch_changesets'
    /vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:177:in `send'
    /vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:177:in `method_missing'
    /app/controllers/repositories_controller.rb:55:in `show'
    /vendor/rails/actionpack/lib/action_controller/base.rb:1162:in `send'
    /vendor/rails/actionpack/lib/action_controller/base.rb:1162:in `perform_action_without_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:580:in `call_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:573:in `perform_action_without_benchmark'
    /vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure'
    /vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /vendor/rails/actionpack/lib/action_controller/rescue.rb:201:in `perform_action_without_caching'
    /vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:13:in `perform_action'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache'
    /vendor/rails/activerecord/lib/active_record/query_cache.rb:8:in `cache'
    /vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:12:in `perform_action'
    /vendor/rails/actionpack/lib/action_controller/base.rb:529:in `send'
    /vendor/rails/actionpack/lib/action_controller/base.rb:529:in `process_without_filters'
....

#19 Updated by James Turnbull almost 12 years ago

+1 - I am also seeing this problem on Mongrel Web Server 1.1.5 with a Apache as proxy on Debian etch. I am currently at r1725.

#20 Updated by Heinz Gies almost 12 years ago

Okay I did a little testing and I think I am getting down to the root of the problem.
What I noticed is, the problem arose when I set up a project with redmine that has a few 'very' large files 1MB + of source code, so taking this as a start I looked at the memory footprint of redmine while I browsed the page one page at a time.
A few things I noticed:
  1. The memory footprint hardly ever droops once it raised and if it does very slowls
  2. Highlighting a 1MB file will add about 100M for the first time you call it, it slowly goes down 10 or 20 MB. When you call it a second time adds another 50 MB - given the call was a few moments ago so Mongel does not provide it straight ahead form he cache.
  3. It seems that re-highlighting a file causes the footprint to go down drastically before it raises again to it's old level.
  4. Opening a diferent file does not cause the footprint to lower again.

So what I conclude of this:
The memory leak seems to be somewhere around the part where the code gets highlighted.

#21 Updated by Sebastian Vassiliou almost 12 years ago

i have a similar problem, actually much more severe, which forced me to completely shut down redmine undefinetly.
i'm running a debian etch amd64 system btw.
so redmine has always had problems with memory leaking, killed my server twice until I discovered redmine/mongrel was to blame, so i made a cron to restart mongrel each night.
but at some point things worsened, and latelly memory was growing much faster, and mongrel processes would often hang and eat cpu power. very, very bad.
so i thought, mongrel is to blame, and i switched to passenger. not a good idea after all. after a few hours it eventually took down my whole apache and filled almost all memory but i was lucky enough to kill the stuff before losing the server again.
so redmine is really a no go for now :( am pondering to migrate the data back to mantis... somehow...
anyway, what i noticed with the process hanging was that it hangs when looking at the repositories. i have svn, btw. and search engines were looking at them, as the logs say.
so i modified robots.txt but engines were too slow to update. so i disabled the repository in redmine. but guess what: google's cache feature managed to fuck up redmine anyway, requesting things which should have been disabled.
btw sometime when trying to fix problems, i updated from 0.7.3 to svn, but i fear this has done more harm than good...
i'm desperate. what can i do?

#22 Updated by Sebastian Vassiliou almost 12 years ago

oh, with filling memory i meant: my machine has 2 gb ram + 2 gb swap, and redmine takes about 500 MB when in "normal" operation, then grows, and grows, and grows...

#23 Updated by Thomas Lecavelier over 11 years ago

The interesting point should be to find out what activity redmine was accomplishing: repository browsering? Fetching changeset? You should have a look into your logs to determine that, eventually turn log level to DEBUG.

#24 Updated by Eric Davis over 11 years ago

When I ran Redmine with mongrel I used monit to watch my mongrels and restart them if they started to take too much RAM or CPU. It ran great and my two mongrels never went over 90MB of RAM each on my 1GB VPS server. If you want my monit config for Redmine, it's posted to gist.

(I've since switched to Passenger and it restarts processes as they are not used so monit isn't needed)

#25 Updated by Thomas P. over 11 years ago

We get hit by this bug too very frequently. Commenting the format.pdf lines out has helped but still someone should fix this. Please ;)

#26 Updated by Ismail SEZEN over 11 years ago

I wanted to add another experiences about this issue and perhaps it might help you to solve. I've experienced same issue with both of passengers and mongrel. I had got an shared server and everything was fine. Later, i upgraded my server to VPS and it had 150 MB of RAM. And that was the point. The memory issue arose when i tried to access my svn repository on redmine. It wasn't matter which one of passenger or mongrel. Both of them had the same insufficient memory issue.

Anyway, later, i bumped up my RAM to 256 MB and now everything was fine. I didn't get an error about it. I saw that mongrel or passengers has been using the all of the memory when i had checked the my servers memory status. Perhasp, I'm not sure this issue is about redmine or passenger or mongrel. But i think, redmine may need a RAM usage improvements.

#27 Updated by Kiall Mac Innes over 10 years ago

I've noticed this problem as-well - it seems to be repository related alright, but I haven't been able to consistently reproduce the problem.

What I can say is - these htaccess rules have significantly increased the time between issues:

RewriteCond %{HTTP_USER_AGENT} ^(msnbot|sogou|baiduspider) [NC]
RewriteRule .* - [F,L]

All three of those spiders had been indexing every revision of every file (and with a few k files in trunk .. 10 or so tags ... 4.5k revisions = 150 million or so total repository requests...)

All three of those spiders also seemed to have no idea what rate limiting is...

Also - we're using apache proxying to a thin cluster (so no passenger memory leaks) and we've kept fairly up to date with trunk since the issue began a few months back...

Hope this helps!

#28 Updated by Jan Niggemann (redmine.org team member) over 7 years ago

  • Status changed from Reopened to Closed

Closing this as there's no feedback since 3 years and it concerns a very old version of redmine.

Also available in: Atom PDF