Project

General

Profile

Forums » Open discussion »

enabling x_sendfile to efficiently send large files

Added by Barrett Lawson about 13 years ago

Hi folks,

I had some problems with downloading large (>500MB) files using the "files" module. I've worked it out and thought I should share my solution.

My server is a win32 box running the BitNami stack: apache proxied to two mongrel instances.

Problem

  • I would try to download the file and eventually get a 500 server error.
  • redmine log showed that it handled the page fine
  • apache log showed 200 OK for the GET of the files page.
  • but one of the mongrel logs has lines like:
    Thu Sep 22 16:24:35 -0400 2011: Error calling Dispatcher.dispatch #<NoMemoryError: failed to allocate memory>

Right, duh, it's a big file... so lets go see what the issue is.

Blamed on the ruby garbage collector:
http://www.redmine.org/boards/2/topics/13815?r=18407

Good information here, too:
http://www.ruby-forum.com/topic/134897

Or perhaps that rails is caching the file when passing it along to mongrel:
http://www.therailsway.com/2009/2/22/file-downloads-done-right#comments

the latter suggests using an X-sendfile header. Apache and lighttpd (can) support this header, which tells the server to ignore all output and just send the file. This eliminates unnecessary caching of the file to be downloaded.

worth noting: older search results on this issue suggest a ruby plugin that is deprecated by the built in support. Don't waste your time.

Solution

Configure Apache to support X_Sendfile

*install mod_xsendfile: I'm lazy and used a compiled binary from http://www.apachelounge.com/download/
*add it to httpd.conf
*configure it to run. I had to add:

<IfModule xsendfile_module>
XSendFile on
XSendFilePath "C:/Program Files/BitNami Redmine Stack/apps/redmine/files"
</IfModule>

Edit the attachments controller to use x_sendfile:

  • add ":x_sendfile=>true" to the end of "def download" in attachments_controller.rb

Patch?

It seems like it would be easy enough to patch RM to create an option to enable x_sendfile for attachments. I'm new here, so should I do this or create an issue, or what?


Replies (8)

RE: enabling x_sendfile to efficiently send large files - Added by Aidin Abedi over 12 years ago

How did you upload large files from the beginning? Did you import them into redmine from the local filesystem?

RE: enabling x_sendfile to efficiently send large files - Added by Hank Hill about 12 years ago

Good stuff! However, there is a change in Rails 3.

Now, instead of ":x_sendfile=>true", do the following:

In your production.rb, add the following lines:

  # Specifies the header that your server uses for sending files
  config.action_dispatch.x_sendfile_header = "X-Sendfile"

And add to the apache directives (along with XSendFile on):

RequestHeader Set X-Sendfile-Type X-Sendfile

In CentOS/Fedora/RH, you can get mod_xsendfile from the epel repo.

RE: enabling x_sendfile to efficiently send large files - Added by @ go2null almost 12 years ago

I did the following:

  1. I added the following section to sites-available/mysite
            RequestHeader Set X-Sendfile-Type X-Sendfile
<IfModule xsendfile_module>
        XSendFile on
        XSendFilePath " /home/redmine/www/current/files" 
</IfModule>
  2. and the following to /config/environments/production.rb
     # Specifies the header that your server uses for sending files
  config.action_dispatch.x_sendfile_header = "X-Sendfile"
  3. I also installed libapache2-mod-xsendfile from Debian Sid (squeeze has 0.92, sid has 0.12 - which supports XSendFilePath)
  4. and enabled mod_headers (for RequestHeader).

The error I'm getting is:

File Not Found
The file https://mysite/attachments/download/54/myfile.zip cannot be found. Please check the location and try again.

production.log has this entry:

Started GET "/attachments/download/54/myfile.zip" for 10.10.9.99 at 2012-12-12 10:20:16 -0500
Processing by AttachmentsController#download as HTML
  Parameters: {"id"=>"54", "filename"=>"myfile.zip"}
  Current user: me (id=3)
Sent file /home/redmine/www/releases/20121209221527/files/121212102012_myfile.zip (0.1ms)
Completed 200 OK in 7ms (ActiveRecord: 1.9ms)

Any help appreciated.

I'm running Redmine on Debian 6 Squeeze

Environment:
  Redmine version                          2.1.0.stable
  Ruby version                             1.9.2 (x86_64-linux)
  Rails version                            3.2.8
  Environment                              production
  Database adapter                         Mysql2

RE: enabling x_sendfile to efficiently send large files - Added by @ go2null almost 12 years ago

Solved:
  1. Typo (spaces before path) XSendFilePath " /home/redmine/www/current/files"
  2. current is a symlink, and web links point to actual file location.

RE: enabling x_sendfile to efficiently send large files - Added by @ go2null about 11 years ago

If I had read this, it would have prevented error 2.

2. current is a symlink, and web links point to actual file location.

RE: enabling x_sendfile to efficiently send large files - Added by Karel Pičman almost 8 years ago

I've been using this feature for many years. In fact downloading of large files from Redmine is impossible without that option.
A few days ago I found that it's working no more. Although the setting remains without any change.

I've set in my additional_environment.rb:

# Specifies the header that your server uses for sending files
config.action_dispatch.x_sendfile_header = "X-Sendfile"

But, this option is simply ignored and requests for files download doesn't contain the header X-Sendfile and are consequently processed by Redmine instead of the webserver.

Tested in Redmine 3.3.1. I've also tested it with a clear Redmine installation (without any plugins) with the same result.

Any ideas?

    (1-8/8)