massive startup delays
Added by Gareth Evans over 13 years ago
Hi all
We are using redmine under apache2.
Whenever I go to use it and it hasn't been used for a while (like 30 min) it pushes the CPU load to 100% and sits there for a good few mins.
Our issues database is pretty small (maybe 300-400 issues) and all the CPU load seems to be on the ruby process.
I know very little about ruby as I'm a .net dev [sorry :)] but redmine is awesome when it works.
I've tried debugging - reading an article about inspecting ruby with gdb, and once I caught it in a select() but I couldn't trace it any more.
I've turned logging on and the best information I can find is this:
Processing IssuesController#index (for 10.0.0.58 at 2011-04-13 16:26:13) [GET]
Parameters: {"project_id"=>"aamobjob", "action"=>"index", "controller"=>"issues"}
Rendering template within layouts/base
Rendering issues/index.rhtml
Completed in 132043ms (View: 4368, DB: 5187) | 200 OK [http://ssaosvn/projects/aamobjob/issues]
We had SVN integration turned on, but I turned it off thinking this is what was slowing things down.
I've just updated the redmine to the latest version, as well as done a full update on the server, so everything is the latest ver, but the problem persists.
Immediately after I had upgraded redmine to the latest version, it loaded super fast, but a few mins later the issue had returned.
I'm not sure how to debug this issue.
The server is a ubuntu 10.4 LTS, happy to provide more debugging info.
The ssaosvn server is a VM with plenty (i think 1gb) of ram allocated to it, but the host machine is a quad core 3ghz xeon that's not doing much.
Locking the CPU at 100% for 2 mins seems like a LOT of work.
I notice the memory usage by ruby creep up (maybe 1mb every 3-4 secs) while this is happening
Once things start up, redmine is snappy and great, but that first load is really really bad.
Any help you can provide would be appreciated
n.b I didn't set up the redmine install originally
Thanks
Replies (12)
RE: massive startup delays - Added by Gareth Evans over 13 years ago
Rails 2.3.5
Apache/2.2.14 (Ubuntu)
redmine-1.1.2
mysql Ver 14.14 Distrib 5.1.41, for debian-linux-gnu (x86_64) using readline 6.1
root@ssaosvn:/opt/redmine-1.1.2# RAILS_ENV=production script/about
: No such file or directory
I read the FAQ and it looks like it could be the mysql ruby adapter, but I don't know how to update that - shouldn't an aptitude update && aptitude upgrade do it?
RE: massive startup delays - Added by Felix Schäfer over 13 years ago
The svn integration could at most slow down the repository views, and that even not by that much, so you can turn the on without worry.
Gareth Evans wrote:
I read the FAQ and it looks like it could be the mysql ruby adapter, but I don't know how to update that - shouldn't an aptitude update && aptitude upgrade do it?
It depends if it's been installed from apt or from rubygems (the "ruby" dependency and package manager). Have a look at the installed package list in apt and see if and which mysql-ruby or mysql-gem or something like that package has been installed.
While you're at it, have a look if passenger or mod_passenger or however it's called in ubuntu is installed. I guess the problem you are seeing is from the the default process management from the "rails apache"-module. Passenger spins up processes as needed (up to a certain limit), i.e. one process if you only have sequential requests, if two requests come in at the same time it spins up a second one, and so on, and those processes have a time to live of 30 minutes. Those processes make some heavy caching, i.e. once they're spun up, they run quite well, but it takes time to get them up. I think that's this phenomenon you're seeing after more that 30 minutes.
(oh, and getting the heavy and extremely narrow-width artillery like gdb or debugging to try to get behind performance problems is imho or at least in my experience tackling the problem from the wrong end, at least for ruby and especially rails)
RE: massive startup delays - Added by Gareth Evans over 13 years ago
I would say that it's been installed from rubygems, as this:
root@ssaosvn:~# dpkg --get-selections | grep ruby
libopenssl-ruby1.8 install
librack-ruby install
librack-ruby1.8 install
libreadline-ruby1.8 install
libruby1.8 install
ruby install
ruby1.8 install
rubygems install
rubygems1.8 install
shows nothing to do with mysql
yes, passenger is installed: libapache2-mod-passenger
I don't mind a spin-up time but 2-3 mins seems quite extreme
RE: massive startup delays - Added by Gareth Evans over 13 years ago
root@ssaosvn:~# gem list
- LOCAL GEMS ***
actionmailer (2.3.5)
actionpack (2.3.5)
activerecord (2.3.5)
activeresource (2.3.5)
activesupport (2.3.5)
fastthread (1.0.7)
i18n (0.4.2)
mysql (2.8.1)
rack (1.0.1)
rails (2.3.5)
rake (0.8.7)
RE: massive startup delays - Added by Anonymous over 13 years ago
I'm experiencing a similar problem. The first two requests after a about 3-5 hours of inactivity take about 30-60sec each. After these, redmine runs smooth.
The mysql-adapter can not be the reason, because I am PostgreSQL. In the beginning I used mod_passenger, after a while I switched to mongrel, thinking it might be passenger-specific. The problem remained, so this might also not be passenger-related.
I'm using ruby1.8 from Debian unstable, a manually installed rubygems 1.3.7 and all gems installed using gem install.
ii libruby1.8 1.8.7.334-1 Libraries necessary to run Ruby 1.8 ii ruby 4.5 An interpreter of object-oriented scripting language Ruby ii ruby-dev 4.5 Header files for compiling extension modules for Ruby ii ruby1.8 1.8.7.334-1 Interpreter of object-oriented scripting language Ruby 1.8 ii ruby1.8-dev 1.8.7.334-1 Header files for compiling extension modules for the Ruby 1.8
*** LOCAL GEMS *** actionmailer (2.3.5) actionpack (2.3.5) activerecord (2.3.5) activeresource (2.3.5) activesupport (2.3.5) cgi_multipart_eof_fix (2.5.0) daemons (1.1.0) fastthread (1.0.7) gem_plugin (0.2.3) i18n (0.5.0, 0.4.2) mongrel (1.1.5) pg (0.10.1) rack (1.0.1) rails (2.3.5) rake (0.8.7)
RE: massive startup delays - Added by Anonymous over 13 years ago
- Redmine 1.0.5.stable.5168 (PostgreSQL)
and - Redmine 1.1.2.stable.5168 (PostgreSQL)
RE: massive startup delays - Added by Felix Schäfer over 13 years ago
Gareth Evans wrote:
I don't mind a spin-up time but 2-3 mins seems quite extreme
That is much too much indeed. I can't say much about fixing this problem, but passenger can be configured to kill/restart processes after a certain number of requests instead of after an idle period, that way the startup penalty would happen only every so often, and you could probably reload the page if you it takes a little while and you would land with another process instead of the one spinning up. I'll try to post an example passenger config, feel free to up the thread if I forget :-)
RE: massive startup delays - Added by Gareth Evans over 13 years ago
Tried that :(
Passenger is up to date:
root@ssaosvn:~# dpkg -l | grep passenger
ii libapache2-mod-passenger 2.2.7debian-1 Rails and Rack support for Apache2
This is my apache site:
RailsEnv production RailsBaseURI /redmine PassengerAppRoot /opt/redmine-1.1.2 Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all
RE: massive startup delays - Added by Gareth Evans over 13 years ago
and my passenger.conf:
PassengerRoot /usr PassengerRuby /usr/bin/ruby PassengerDefaultUser redmine PassengerMaxPoolSize 6 PassengerMaxRequests 0 PassengerMinInstances 1 PassengerPreStart url_to_redmine PassengerPoolIdleTime 0
Restarting apache2 gives:
Syntax error on line 7 of /etc/apache2/mods-enabled/passenger.conf: Invalid command 'PassengerMinInstances', perhaps misspelled or defined by a module not included in the server configuration.
RE: massive startup delays - Added by Gareth Evans over 13 years ago
I removed PassengerMinInstances and PassengerPreStart and apache starts, but the problem is persisting - though, I think the startup time is not as bad now after I haven't used it for a while
RE: massive startup delays - Added by Ronie Henrich over 13 years ago
I have an issue that seems to be the same as yours. Take a look on my post:
http://www.redmine.org/boards/2/topics/21122
- Kill both Redmine/rails processes (kill -9 <pid #> );
- Open Redmine but not click on the Repository tab;
- Restart Redmine (touch /var/www/redmine/tmp/restart.txt);
- Open again Redmine but not click on the Repository tab;
- Now I can open the Repository tab;
RE: massive startup delays - Added by WMI Admin over 13 years ago
This is my issue:
http://www.redmine.org/boards/2/topics/24510
SVN is turned off.