Redmine

Hi,

We are running a production Redmine instance which has experienced a problem where number of Ruby processes grows out of control and causes 99% IO wait on the system (due to swapping). Redmine (v2.3.1) is deployed via Apache (v2.2.22) with mod_passenger (v3.0.13) on Ubuntu 12.04. It seems like the number of processes grows steadily over about an hour before bringing the server to it's knees as we get a warning and then critical message from our nagios monitoring.

So far this has only happened twice, about 6 weeks apart. The machine is not running much else apart from Redmine and the load is pretty low since we only have a few users. Each time this happened shutting down apache and then running 'killall ruby' before restarting apache resolved the problem. We also see nothing to indicate a problem in the apache logs.

I'm afraid I can't realy post any more information since our first priority was to get the system running again. It's also pretty difficult to debug when the system is under such a load.

Does anyone know what might be going on? Additionally is there any information that we should collect for debugging purposes if this happens again?

Thanks in advance.