Redmine

Apache error log says,

[Wed Apr 02 22:12:56 2008] [error] [client 10.5.115.45] (70007)The timeout specified has expired: proxy: error reading status line from remote server 10.1.81.106
[Wed Apr 02 22:12:56 2008] [error] [client 10.5.115.45] proxy: Error reading from remote server returned by /

No other logs (apache Access.log or anything in the applications logs) have any other entries.

I'll try this and report back.

http://www.workingwithrails.com/forums/4/topics/77-how-to-deal-with-proxy-error-s
-----------------

If you are running mongrel_cluster in background with apache+mod_proxy to serve rails requests then this might help you.

mod_proxy sometimes reject requests to Mongrel. This comes from two known Apache bugs #39499 and #37770

To resolve the problem and as workaround for it add the lines below to your Apache VirtualHost section. #Fix for Apache bug 39499

<VirtualHost *>
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
</VirtualHost>

Update: Initial testing seems to indicate that this did not solve the problem.

I made the changes last night, Redmine started fine and worked happily, then I went to bed.

Now, the symptom seems to still exist.

Your instructions worked for me, had the same problem. You said you restarted Redmine, but did you restart Apache?

Yep, I restarted Apache. But, I'll try it again since it worked for you and I don't have anything else to try.

Well, it worked a few days, but was hung again over the weekend.

I ran a test with two WEBrick instances running, one where apache is proxying to the WEBrick port, and one where it's listening directly. They were BOTH hung this morning. Running gdb against the Ruby process gives me this: I know it's not too helpful, but it at least seems to indicate some sort of threading problem?? Perhaps deadlock? How is the redmine.org site staying up?

0x00a977a2 in dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) where
#0 0x00a977a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b75a41 in __newselect_nocancel () from /lib/tls/libc.so.6
#2 0x00c32ed0 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#3 0x00c42849 in rb_thread_select () from /usr/lib/libruby.so.1.8
#4 0x00c57428 in rb_gets () from /usr/lib/libruby.so.1.8
#5 0x00c38683 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#6 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#7 0x00c33f2f in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#8 0x00c3558b in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#9 0x00c34487 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#10 0x00c36cbe in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#11 0x00c34623 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#12 0x00c3b355 in rb_need_block () from /usr/lib/libruby.so.1.8
#13 0x00c34256 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#14 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#15 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#16 0x00c33f2f in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#17 0x00c372e4 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#18 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#19 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#20 0x00c33f2f in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#21 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#22 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#23 0x00c33f2f in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#24 0x00c414cf in rb_load () from /usr/lib/libruby.so.1.8
#25 0x00c41abe in rb_require_safe () from /usr/lib/libruby.so.1.8
#26 0x00c41ce9 in rb_f_require () from /usr/lib/libruby.so.1.8
#27 0x00c38655 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#28 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#29 0x00c3408e in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#30 0x00c36cbe in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#31 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#32 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#33 0x00c3d561 in rb_call_super () from /usr/lib/libruby.so.1.8
#34 0x00c341b2 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#35 0x00c3b355 in rb_need_block () from /usr/lib/libruby.so.1.8
#36 0x00c34256 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#37 0x00c36ea2 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#38 0x00c36ea2 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#39 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#40 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#41 0x00c33f2f in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#42 0x00c372e4 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#43 0x00c36cbe in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#44 0x00c381ca in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#45 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#46 0x00c3408e in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#47 0x00c414cf in rb_load () from /usr/lib/libruby.so.1.8
#48 0x00c41abe in rb_require_safe () from /usr/lib/libruby.so.1.8
#49 0x00c41ce9 in rb_f_require () from /usr/lib/libruby.so.1.8
#50 0x00c38655 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#51 0x00c39169 in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#52 0x00c3408e in rb_thread_schedule () from /usr/lib/libruby.so.1.8
#53 0x00c40e96 in ruby_exec () from /usr/lib/libruby.so.1.8
#54 0x00c41f9e in ruby_run () from /usr/lib/libruby.so.1.8
#55 0x0804862c in main ()

I did a bunch of searching on the web and found many people having this problem with Rails apps, but no answers.

We increased the memory on the VMware slice (where Redmine is the only thing running) to 1GB from 512MB and while it improved performance I came in this morning and the issue is still there.

So, I tried to reach Redmine directly from localhost on one of the ports it was supposed to be listening on. Silence. Someone was home but they weren't answering. I stopped my mongrels and got standard 404 errors.

I restarted my mongrels and everything worked fine. I didn't even have to restart Apache. So it's something about Rails/Mongrel stopping listening that seems to be the root of the problem.

----------------------------
Also, what's going on with port 3001 here?

redmine@ubuntu710jeos:/var/www/html/redmine# mongrel_rails cluster::stop
stopping port 3000
stopping port 3001
stopping port 3002
stopping port 3003
stopping port 3004
stopping port 3005
redmine@ubuntu710jeos:/var/www/html/redmine# mongrel_rails cluster::start
starting port 3000
already started port 3001
starting port 3002
starting port 3003
starting port 3004
starting port 3005
redmine@ubuntu710jeos:/var/www/html/redmine# mongrel_rails cluster::stop
stopping port 3000
already stopped port 3001
stopping port 3002
stopping port 3003
stopping port 3004
stopping port 3005

I'm going to guess if you look at the log for 3001 you'll see a message telling you it's waiting on a thread and shutdown may take a while (I think it mentioned 60s). Probably your mongrel-stop only partially succeeded, and the start detects the ports in use.

BTW, I don't think the memory is the issue, since ruby/redmine seems to take only about 50M virtual size.

interesting thread that seems to relate to the problem.

http://www.ruby-forum.com/topic/107498

Oh man, I'm not going to go through all of this...
http://rubyforge.org/pipermail/mongrel-users/2007-August/003831.html

Here's the rules for resolving a "RAILS in Mongrel dies" problem (notice

the RAILS part?):

1) 1000's of sites run mongrel without this problem so look to what you
have installed first.
2) Make sure that every single bit of software you are running is the most
recent version and everything is installed and used correctly. Common
culprits are:
a) MySQL -- install the gem manually and make sure that the most recent
is the only one. DO NOT USE THE RAILS DEFAULT.
b) Memcached -- Use the very latest from Eric Hodel's project and do NOT
put any keys in that have spaces or null (\0) chars. Yes, your keys cannot
have spaces. YES YOUR KEYS CANNOT HAVE SPACES.
c) net::http to some web site. This can cycle forever.
3) Next, once you've made sure all the above is isolated then you can
proceed to stage 2.

STAGE 2

1) You must let your application run in the best configuration and be
ready to pounce on it the second a mongrel dies.
2) Log in to your server and find out the PID file of the mongrel that
isn't responding. You do this by hitting the process on it's actual port
(like 8000, 8001, 8002, NOT apache/nginx's port of 80).
3) Once you know what port it is, then use: sudo lsof -i -P | grep PORT to
find what process PID is on that port.
4) Next, attach to this process with: sudo strace -p PID. What you'll see
in a healthy mongrel is lots of variation. What you'll see in a dead
mongrel is probably either a bunch of calls to select/poll for the exact
same filedescriptors, or nothing.
5) If you see it doing a select on the file descriptors, then you need to
find out what is on that FD that is causing it to wait. Again, use: lsof |
grep FILEDESCR
a) This will potentially tell you where it's connected, etc. Once you
do this you know what is being read/written and can go find out what in your
rails app is using it, and apply even more debugging tools.
b) You can also just force it closed externally. There's a few mentions
of this but I don't remember the exact procedure.
c) The reason this typically happens is that you have a socket that ruby
has written to and not read from yet, but that socket is closed. Another
cause is that you are simply waiting for data (which is what happens with
memcached and putting a space in your keys).
6) If your mongrel is completely dead then move on to stage 3 and also
you're kind of fucked.

STAGE 3

1) Get your good shoes on because you're now in GDB land and C.
2) http://eigenclass.org/hiki.rb?ruby+live+process+introspection Explains
attaching to your mongrel process (you've identified above) using GDB,
loading some GDB scripts, and then stopping it, inspecting it, and forcing
an exception.
3) Do those thigns. Attach. Pause it. Inspect variables. Get stack
traces. Force an exception. See where it's stopped. Look for where OUTSIDE
of mongrel it's coming from.
4) That's all I can say right now. Let's hope other people can expand on
this.

--
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

I had a similar problem when my mobile phone is connected through USB to Windows XP to synchronize with Outlook (problem with ActiveSync)
May not be the same issue at your end, but maybe you can check if something similar is going on at your end.

regards
sriram