No import function = giving up on Redmine
Added by James R over 11 years ago
Hello,
I am not a developer and have been looking at the possibility of using redmine to manage non IT projects.
I need to import about 100 existing projects with associated custom fields and issues. I have looked for importer plugins but none of them seem to function with v2.3. I have experimented with the API but there seems to be bugs when working with custom fields. I have scoured the forum and stack exchange looking for solutions and have posted a few questions looking for advice.
http://www.redmine.org/boards/2/topics/37645, http://www.redmine.org/boards/2/topics/37509, http://www.redmine.org/boards/2/topics/37595.
Now it seems that maybe python might be the best way of importing things: http://stackoverflow.com/questions/16200812/post-multiple-issues-to-redmine-via-api but this is yet another thing to learn.
It has been fun learning about curl, RESTful APIs and python but honestly at this point I just want to import my data and get on with my day job.
Surely the easy import/export of data to redmine must be a major issue for others people too and would be a major feature to prioritize.
Am I missing something?
Replies (30)
RE: No import function = giving up on Redmine
-
Added by Jean-Baptiste Barth over 11 years ago
You're not missing anything. There are already import tools from Mantis and Trac, two other popular management projects, but nothing official for a CSV file. Is your data somehow "private" ? I'd love to see something in core that would do exactly that, so if you can send me test data I'd be able to tell you if the task is difficult, and if not I could propose a script for that (we'd open an issue at this point, the forum is not the place where we should treat that kind of things). From what you said you should have two CSV files, one for projects, the other one for issues, am I right ? If your data is not too private, send me a version by email at " jeanbaptiste.barth@gmail.com ".
RE: No import function = giving up on Redmine
-
Added by James R over 11 years ago
My data is messy but I am looking for a generic solution that would hopefully be useful for others too. I propose initially extracting this data to three csv files. In the examples below I do not include all the custom fields I will probably need in the end.
Project.csv
id,name,identifier,description,customfield1 1,BFA0241,bfa0241,a first test,AAA001 2,KEN0228,ken0228,a second test,ABB002 3,UGA0222,uga0222,a third test,ABC003
issues.csv
project_id,project_name,tracker_id,status_id,subject,description,category_id,start_date,due_date,customfield2 1,BFA0241,4,2,problem1,bigmess,4,2013-05-08,2013-06-12,124552 1,BFA0241,3,1,problem2,small mess,2,2013-05-07,2013-05-08,125667 2,KEN0228,4,3,problem3,mediummess,3,2013-06-08,2013-06-11,130001
users.csv
login,password,firstname,lastname,mail,auth_source_id bob,pwd1,Bob,Boggins,bob@gmail.com,2 fred,pwd2,Fred,Fredkins,fred@gmail.com,3 sue,pwd3,Sue,Sasson,sue@gmail.com,4
I have already tried converting this to json/xml (using a big excel concatenate function) and importing through the api but it seems to fall down on custom fields, dates, tracker ids etc. Ideally it would be possible to reference projects by name rather than id when posting issues as ids would be generated automatically when I POST new projects meaning I would have to go back and edit my issues.csv to include project ids before importing. I hope that all makes sense?!?
Definitely in terms of user friendliness a script that can deal directly with csv would be the easiest thing for people like me to deal with!
Many thanks for any help you can offer with this!
RE: No import function = giving up on Redmine
-
Added by Jean-Baptiste Barth over 11 years ago
Preliminary version: https://gist.github.com/jbbarth/5551408
Copy this file in your "lib/tasks/" directory, then run it with:
RAILS_ENV=production rake redmine:import_from_csv users=tmp/users.csv projects=tmp/projects.csv issues=tmp/issues.csvSome notes:
- BE SURE TO MAKE A DB BACKUP BEFORE LAUNCHING THE SCRIPT. :)
- this is not fully functional for now, but I'd like some feedback before continuing (or not). Custom fields import doesn't work especially.
- those CSV files should not include "id" fields. In the issues.csv file, you should have a "project_identifier" column to reference the project name or identifier to which the issue belongs.
- user passwords should be 8 chars min. Globally, this script doesn't violate redmine constraints, e.g. the line will display an error if a validation doesn't pass, and the data won't be imported.
- for now I recommend you do a backup of your database and restore it between each try.
Any feedback is welcome, and please be indulgent it's only a work in progress...
RE: No import function = giving up on Redmine
-
Added by James R over 11 years ago
thanks for all your work on this. I have not been able to try it out yet but will do so over the next couple of evenings and let you know how things go. Ruby is all a bit of a mystery to me...
RE: No import function = giving up on Redmine
-
Added by Ryan Hsu over 11 years ago
Hi, Jean-Baptiste Barth,
Thanks for your effort on this. It did help me!
I just post what I have tried and modification to make the import task work.
The command is:
RAILS_ENV=production rake redmine:import_from_csv issues=/tmp/test2.csv
The fields I use:
project_id,author_id,tracker_id,status_id,subject,description,category_id,start_date,due_date,customfield8
customfield is fixed name, "8" is found in database.
attachment:
csv file: test2.csv
database: issue_sql.png
redmine task issue status: import_issue.png
Environment:
Redmine version 2.2.0.stable
Ruby version 1.8.7 (x86_64-linux)
Rails version 3.2.9
Environment production
Database adapter MySQL
Redmine plugins:
redmine_importer 1.0
redmine_tab 0.4.0
issue_sql.png (17.3 KB) issue_sql.png | |||
import_issue.png (59.4 KB) import_issue.png | |||
test2.csv (183 Bytes) test2.csv |
RE: No import function = giving up on Redmine
-
Added by James R about 11 years ago
Hi Jean Baptiste!
Sorry for the big delay... family priorities have taken precedence for the last few weeks...
Wow this is excellent! Works like a charm with not problems at all once I figured out how to run a ruby command! Now I need to experiment with the custom fields more...
Thank you so much for this. Definitely should be integrated into a future release as it opens up a whole world of import possibilities!
import issue by project_name
-
Added by James R about 11 years ago
A useful feature would be the ability to import the issues by project name as often it is difficult to identify the project ID.
I have no idea whether this would possibe?
Here is a modified example of the issues.csv:
project_name,author_id,tracker_id,status_id,subject,assigned_to_id,description,category_id,start_date,due_date UGA0222,1,2,2,Batty,5,bigmess,4,2013-05-05,2013-06-07 UGA0222,1,3,1,Batty,5,small mess,2,2013-05-05,2013-06-07 UGA0222,1,2,3,Batty,5,mediummess,3,2013-05-05,2013-06-07
At a later stage something similar for stuff like assigned_to_name, author_name would also be great for ease of import from csv files.
grateful for any insights...Or just tell me I am talking the impossible here!
RE: No import function = giving up on Redmine
-
Added by James R about 11 years ago
Experimenting with custom fields for the project.csv file. As you said this is not working yet
Is there any suggested approach as to how I could even start to get this working?
thanks for any help you can offer here.
RE: No import function = giving up on Redmine
-
Added by Vincent Olivieri about 11 years ago
This is a great util!
I've installed and tested it in my Redmine eval instance. I'm trying to import projects and issues from an in-house workflow system.
Any progress on custom fields? That would finish this for me and allow us to proceed with the conversion. I'm able to specify the custom fields in my csv and see rows created in custom_values, but with NULL values. I then noticed the "todo" comment in the script.
Thanks so much for your time and effort!
RE: No import function = giving up on Redmine
-
Added by James R about 11 years ago
Same results for me regarding custom fields. I do not have the skills to develop this further and am trying to find a good work around... maybe using the api?
no luck yet
RE: No import function = giving up on Redmine
-
Added by Saeed Richardson about 11 years ago
Just thought I'd chime in as an extra contributor. I've used the import for about 70 issues from Pivotal Tracker and it worked well with the exception of not liking the "fixed_version_id" field. I opted to just remove the column and it worked without a hitch.
Thanks for taking the time to create the rake task for this.
RE: No import function = giving up on Redmine
-
Added by Andrey Anufrienko almost 11 years ago
I've written some changes that allow to import issues (only ISSUES!) with custom fields.
Please use attached script or replace 'def run' from your script with the following:
- Runs the migration
def run
errors = []
puts
puts "Starting data import."
puts
w(users projects issues).each do |type|
next unless content[type]
klass = type.classify.constantize
print "#{klass}: "
my_custom_fields = Array.new
content[type].each do |attributes|
object = klass.new
object.tracker = Tracker.first if klass == Issue
attributes.each_with_index do |value, index|
field = mappings[type][index]
if type "issues" &x%x field "project_identifier"
object.project_id = Project.where("name = ? or name = ?", value, value).first
elsif field.is_a?(String)
object.send("#{field}=", value)
else
#customfield
v = CustomValue.new :custom_field_id => field.id,
:value => value
v.customized_type = "Issue"
my_custom_fields.push(v)
end
end
if object.valid?
print "."
object.save
#we have to create custom fields after creating issue because we need issue's ID
my_custom_fields.each do |mcf|
mcf.customized_id = object.id
#we have to replace existing default (zero) values with ones from CSV
cv = CustomValue.where(:customized_type => mcf.customized_type, :custom_field_id => mcf.custom_field_id, :customized_id => mcf.customized_id).first
if cv.present?
cv.value = mcf.value
cv.save
else
mcf.save
end
end
else
print "E"
errors << "Cannot save following line in #{type}: #{attributes.join(",")}\n => errors: #{object.errors.messages.inspect}\n => object: #{object.inspect}"
end
end
puts
end
puts
if errors.any?
puts "Errors:"
errors.each{|e| puts e}
end
end
import_from_csv.rake (6.75 KB) import_from_csv.rake | script with some fixes |
RE: No import function = giving up on Redmine
-
Added by Juan Fernandez almost 11 years ago
My attempt, a Comand line interface Configurable (through yml config file) PHP csv importer, with custom fields.
https://github.com/juanmf/RedmineIssueImporter
I imported 868 Issues from hand made Excel Sheets into redmine Issues, in <10 minutes (once all wrong value mappings wehe memorized).
RE: No import function = giving up on Redmine
-
Added by Petr Czech about 10 years ago
Is there a easier solution ?
I want to import my exel data, wich looks like(File: "Example issue").
I dont understand that one form Juan...
It would be perfect, then i only click on the one "Import" button in Redmine, chose my file and it would do everything.
Hope someone has a solution and now what i mean
see ja
Example Issue.xlsx (9.05 KB) Example Issue.xlsx | The Example |
RE: No import function = giving up on Redmine
-
Added by Jan Niggemann (redmine.org team member) about 10 years ago
Hi Petr,
no, that's not possible I'm afraid.
RE: No import function = giving up on Redmine
-
Added by Ivan Cenov about 10 years ago
- Read csv file line by line
- use Redmine REST API to import issues.
RE: No import function = giving up on Redmine
-
Added by Juan Fernandez about 10 years ago
Hi Petr,
I took 10 minutes to make an example config file for you, that follows your xslx data format (except for tue estimated time, you should add it if you plan to use the importer).
check it out here:
https://gist.github.com/juanmf/8f75706af3ec7ed74918
Regards,
Juan
Petr Czech wrote:
Is there a easier solution ?
I want to import my exel data, wich looks like(File: "Example issue").
I dont understand that one form Juan...
It would be perfect, then i only click on the one "Import" button in Redmine, chose my file and it would do everything.
Hope someone has a solution and now what i mean
see ja
RE: No import function = giving up on Redmine
-
Added by @ go2null about 10 years ago
RE: No import function = giving up on Redmine
-
Added by @ go2null over 9 years ago
RE: No import function = giving up on Redmine
-
Added by Olivier Houdas over 9 years ago
@ go2null: I'm not sure how to suggest fixes for your fork in github, so I forked mine (gc-olivierho, but I don't intend to maintain it...). Please check my changes: 2 fixes (including one on your last fix 2 days ago, which missed an "s") + 1 change of behavior to be able to both import new lines and update existing ones.
And I want to thank all those who have contributed to get this feature into Redmine for this great work! This plugin is just going to make our sprints twice faster if not more, with task definition & estimation to be made live in a spreadsheet instead of modifying Redmine issues in the interface (we have a "Task" type) over and over again !
RE: No import function = giving up on Redmine
-
Added by Korneliusz Moc over 9 years ago
Hey guys,
Since some time I'm struggling with issue importer. I've already tried various plugins and nothing help.
Now I've tried go2null fork, but again no luck (site is returning "internal error"). I've checked production.log the check out the reason and I found "_NoMethodErrror (undefinied method `find_or_create_by_user_id' for #<Class:0x007fbe77d481230>):_"
I assume this is problem with my redmine, not the plugins.
I'm running:
Environment: Redmine version 3.0.0.stable Ruby version 2.0.0-p598 (2014-11-13) [x86_64-linux] Rails version 4.2.0 Environment production Database adapter Mysql2 SCM: Subversion 1.8.10 Git 1.9.5
Do you have anything that run on Redmine v3.0. I will be greatefull for any advice.
RE: No import function = giving up on Redmine
-
Added by Juan Fernandez over 9 years ago
have you tried my cli tool? I posted it above.
many have succeded with it.
RE: No import function = giving up on Redmine
-
Added by Korneliusz Moc over 9 years ago
Yeah, I gave it a try. Enabled API access key on Redmine, and set up everything including config. Seems that everything should be working, parser is properly reading input file, but for unknown reasons I'm getting an error. For test purposes I'm trying to load input file only with one record, so process is returning one error. As I've checked in errors.dump, there is info
type => No response from server, not even an error code, check config data, project name, etc.
This is wierd as server is responding as it should. I've even checked the connection via cURL with the same data I use in config.yml
And it worked. I've received xml as expected. Don't know where else to look for.
RE: No import function = giving up on Redmine
-
Added by Juan Fernandez about 9 years ago
korneliusz w Moc
could you update and retry? I added the stringified response to the exception message. So you can see what was actually returned.
Custom Column Import csv into Redmine
-
Added by santhosh babu about 9 years ago
Dear All,
Is it Possible to Import Customize Column into Redmine through Import csv.