Using Amazon S3 for storage
Added by Christopher Dell over 15 years ago
Hello,
I'm trying to find some documentation about adding S3 support to Redmine be it by hacking the codebase or via a plugin. My current install is hosted on Heroku (which has a read only filesystem) hence the need for file uploads to be handled somewhere else in the cloud.
Has anyone been confronted with this issue before? Is there plugin in the works that I might be able to contribute to?
If not, do any of the devs know of any pitfalls that might be encountered before developing such a plugin? (ie, in principle should it simply be a matter of overloading the Attachment model?).
Any information would be much appreciated,
Chris
Replies (33)
RE: Using Amazon S3 for storage - Added by Christopher Dell over 15 years ago
After looking at the source code I'd be tempted to add in paperclip to the core - that would pave the way for a dead easy S3 implementation - any thoughts?
RE: Using Amazon S3 for storage - Added by Eric Davis over 15 years ago
Go right ahead, as long as you submit a patch for it ;)
I've been taking a look at paperclip for Redmine myself. I have one Redmine on Heroku (demo.redmine.org) but I'd like to be able to put my main Redmine's files on S3 to have more flexibility in hosting. I think the easiest path would be to create a Redmine plugin that overrode the core's Attachment model and saved to S3. That would make it easy to keep it compatible with the core Redmine until it's ready to be merged. If you could post it to Github, I might be able to help out once I finish up a few other projects.
Eric Davis
RE: Using Amazon S3 for storage - Added by Christopher Dell over 15 years ago
OK I'll have a go - not entirely sure about the best way of doing this but I guess it will become obvious as I plug away. Will post updates here.
RE: Using Amazon S3 for storage - Added by Shereef Bishay over 15 years ago
+1 for getting file uploads to work with S3. I too am hosted on heroku and would love to see this working. Christopher let me know how it's going, and if you need a hand. I'm no expert but will do what I can.
RE: Using Amazon S3 for storage - Added by Christopher Dell over 15 years ago
OK, so a first stab at an S3 plugin can be found here : http://github.com/tigrish/redmine_s3
I decided not implement paperclip in the end since it meant changing a whole bunch of the Attachment code.
Haven't had time yet far to tidy it up, add tests and sort out all of the plugin pages but it basically works.
Have tested on Heroku and it seems to be working fine. Yay!
More updates to come over the course of the next couple of days....
RE: Using Amazon S3 for storage - Added by Eric Davis about 15 years ago
Great, I'll try it on http://demo.redmine.org.
Eric Davis
RE: Using Amazon S3 for storage - Added by Andrzej Giniewicz about 15 years ago
That plugin looks great! I wasn't reading its code yet or testing it, I have question before I try - Is it designed to work with one S3 configuration per redmine instance or project? Also is it possible to configure S3 access data trough redmine administration options or it needs file manipulations?
RE: Using Amazon S3 for storage - Added by Christopher Dell about 15 years ago
Andrzej Kamiński - The plugin is made for 1 s3 bucket per redmine instance ie. all projects use the same S3 bucket.
All of the configuration is done with a yml file, so no it is not yet possible to configure this via the redmine administration area.
RE: Using Amazon S3 for storage - Added by Jim Mulholland about 15 years ago
Nice job on the S3 plugin! Are there any other gotcha's to hosting Redmine on Heroku?
We are thinking about migrating over from Slicehost.
RE: Using Amazon S3 for storage - Added by Christopher Dell about 15 years ago
Jim Orton - the only thing that is problematic now is source control. I seem to remember seeing a github plugin but haven't really investigated it yet...
RE: Using Amazon S3 for storage - Added by Jim Mulholland about 15 years ago
Thanks for the heads up on source control. I'll have to look into that before making the switch.
RE: Using Amazon S3 for storage - Added by Eric Davis about 15 years ago
Jim Mulholland:
I'm running http://demo.redmine.org on Heroku. It's code is public on Github so feel free to steal what you need. A few gotchas:
- Use heroku's config for storing your session store.
- You will need to start Redmine locally and add your Redmine plugin's assets to git (public/plugin_assets). When Rails Engines starts, it copies those files but Heroku has a read-only filesystem so it will fail.
- If you add SCM repositories, your database will grow really fast. I had demo.redmine.org up to about a 500MB database, well over Heroku's free plan.
Let me know if you need any help, I've deployed a few Redmine to Heroku and am planning to deploy a lot more of them there.
Eric Davis
RE: Using Amazon S3 for storage - Added by Shereef Bishay about 15 years ago
Eric,
First off thanks for all the great dev work! Truly inspiring.
I've got redmine hosted on Heroku, and can't figure out how to get the repositories up and running. I'm esp. interesting in getting GIT working. I don't mind paying a little for the DB space.
Can you point me in the write direction? What am I not seeing?
thx in advance
RE: Using Amazon S3 for storage - Added by Alain D. about 15 years ago
Thank you so much Christopher, it's working very well and I can at last store files on my Heroku hosted Redmine !
RE: Using Amazon S3 for storage - Added by Obiwan Chinobi about 15 years ago
Eric Davis wrote:
- You will need to start Redmine locally and add your Redmine plugin's assets to git (public/plugin_assets). When Rails Engines starts, it copies those files but Heroku has a read-only filesystem so it will fail.
Could you please elaborate on this? I setup rm to run locally, but how "add your Redmine plugin's assets to git (public/plugin_assets)"?
I am encountering the error when running rake db:migrate on heroku:
Read-only file system - /disk1/home/slugs/93246_e56aa89_a702/mnt/public/plugin_assets
RE: Using Amazon S3 for storage - Added by Christopher Dell about 15 years ago
I seem to recall it being along the lines of :
# edit .gitignore - remove the /public/plugin_assets line git add public/plugin_assets git commit -a -m "Adding in the plugin_assets dir" git push heroku master
The idea is to make sure the directory is already created so that Redmine's initialization script doesn't try to create it on the read-only file system.
RE: Using Amazon S3 for storage - Added by Eric Davis almost 15 years ago
Christopher Dell:
Great job on the plugin. Installed it, configured my S3 account, and it worked perfectly the first time.
I've forked it and will try to add some more features. I saw you wanted a gemspec eh, I just happen to have a script for that. ;)
Eric Davis
RE: Using Amazon S3 for storage - Added by Michael Bilderback almost 15 years ago
Eric, Chris,
Let me join the others in saying fabulous work on this code.
I was wondering if I could get some help though :) So, I would like to host my redmine on Heroku. I have the redmine_s3 plugin installed and running on my localhost just fine. It writes to the correct s3 bucket and all from the localhost instance. However, when I deploy to Heroku and try to upload files, it fails. Here is a snippet from the heroku logs for the app.
Processing ProjectsController#add_file (for 70.166.89.18 at 2010-01-29 11:37:30) [POST]
Parameters: {"commit"=>"Add", "action"=>"add_file", "authenticity_token"=>"CpduZCIaIZo4boOKr8RM4aF4yEQk21COR4ut9Z6LUzA=", "id"=>"big-bluff", "controller"=>"projects", "attachments"=>{"1"=>{"description"=>"trying again", "file"=>#<File:/tmp/RackMultipart.6075.1>}}}
Errno::ENOENT (No such file or directory - /disk1/home/slugs/122687_624f041_85c8/mnt/files/100129113730_S3-Browser-1.1.zip):
app/models/attachment.rb:79:in `initialize'
app/models/attachment.rb:79:in `open'
app/models/attachment.rb:79:in `before_save'
app/controllers/application_controller.rb:261:in `attach_files'
app/controllers/application_controller.rb:258:in `each_value'
app/controllers/application_controller.rb:258:in `attach_files'
app/controllers/projects_controller.rb:306:in `add_file'
/home/heroku_rack/lib/static_assets.rb:9:in `call'
/home/heroku_rack/lib/last_access.rb:25:in `call'
/home/heroku_rack/lib/date_header.rb:14:in `call'
thin (1.0.1) lib/thin/connection.rb:80:in `pre_process'
thin (1.0.1) lib/thin/connection.rb:78:in `catch'
thin (1.0.1) lib/thin/connection.rb:78:in `pre_process'
thin (1.0.1) lib/thin/connection.rb:57:in `process'
thin (1.0.1) lib/thin/connection.rb:42:in `receive_data'
eventmachine (0.12.6) lib/eventmachine.rb:240:in `run_machine'
eventmachine (0.12.6) lib/eventmachine.rb:240:in `run'
thin (1.0.1) lib/thin/backends/base.rb:57:in `start'
thin (1.0.1) lib/thin/server.rb:150:in `start'
thin (1.0.1) lib/thin/controllers/controller.rb:80:in `start'
thin (1.0.1) lib/thin/runner.rb:173:in `send'
thin (1.0.1) lib/thin/runner.rb:173:in `run_command'
thin (1.0.1) lib/thin/runner.rb:139:in `run!'
thin (1.0.1) bin/thin:6
/usr/local/bin/thin:20:in `load'
/usr/local/bin/thin:20
Rendering /disk1/home/slugs/122687_624f041_85c8/mnt/public/500.html (500 Internal Server Error)
Any idea of what I might have missed in the install or setup to cause this?
Thanks in advance, Mike
RE: Using Amazon S3 for storage - Added by Christopher Dell almost 15 years ago
Hi Mike,
From what I can remember from installing Redmine on Heroku, you need to :
- Open up .gitignore and remove the line that says
/files/*
git add files
(you'll probably want to make sure that there aren't any files in your local directory first)- commit, and push to heroku
Let me know if that helps,
Chris
RE: Using Amazon S3 for storage - Added by Michael Bilderback almost 15 years ago
Chris,
Thanks for the prompt reply. Your advice led me down the right path because once I push /files into heroku...I realized that the heroku install was still trying to write to the file system rather than s3. So I went to reconfiguring the gem bundle and got it working. Its now working just as designed.
Mike
RE: Using Amazon S3 for storage - Added by Barry Skidmore almost 15 years ago
I have installed both the S3 plugin and the Google IFrame plugin.
They work locally just fine but when I push my git and db to heroku the plugins do not show up.??
I am making sure that I am running production on my local machine and pushing my production db url to heroku.
Anyone have a clue why this might be?
RE: Using Amazon S3 for storage - Added by Christopher Dell over 14 years ago
Hi Barry,
You need to make sure you're pushing your git repo to Heroku, not just the db.
Hope that helps,
Chris
RE: Using Amazon S3 for storage - Added by Igor Balk over 14 years ago
I'm trying to install S3 plugin to work with Redmine 9.2 and getting following error:
/var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/S3.rb:251:in `make_request': undefined method `empty?' for nil:NilClass (NoMethodError)
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/S3.rb:157:in `create_bucket'
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/redmine_s3/connection.rb:43:in `create_bucket'
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/rails/../init.rb:17
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:182:in `call'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:182:in `evaluate_method'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:166:in `call'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `run'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `each'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `send'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `run'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:276:in `run_callbacks'
from /var/spool/redmine/rm_0196/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:51:in `send'
from /var/spool/redmine/rm_0196/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:51:in `run_prepare_callbacks'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:631:in `prepare_dispatcher'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:185:in `process'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:113:in `send'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:113:in `run'
from /var/spool/redmine/rm_0196/config/environment.rb:23
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:147:in `rails'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:113:in `cloaker_'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:149:in `call'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:149:in `listener'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:99:in `cloaker_'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:50:in `call'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:50:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:84:in `new'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:84:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
from /usr/local/bin/mongrel_rails:19:in `load'
from /usr/local/bin/mongrel_rails:19
Here is my s3.yml configuration
production:
access_key_id:XXXXXXXXXXXXXXXXXX
secret_access_key:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bucket:bucket_v1
cname_bucket: false
development:
access_key_id:
secret_access_key:
bucket:
cname_bucket: false
RE: Using Amazon S3 for storage - Added by Daudi Amani about 14 years ago
Hi,
Am trying to get the redmine_s3 plugin working on heroku. It appears to be working on local dev machine and appears in admin/settings/plugins listing. It does not appear in the plugins listing when I'm on heroku. Mike said that he "went to reconfiguring the gem bundle and got it working." I'm new to redmine and do not see the documentation on how to reconfigure the gem bundle for a plugin like this. Do I need an entry in heroku's required .gems? Do I need a config.gem "redmine_s3" in my environment.rb file?
thanks
David