Feature #3647
openData Import/Export System
0%
Description
Hi All,
I think it will be very useful to have some kind of data import/export system for Redmine with the following features:
1. Entire projects could exported or imported to different instances of Redmine. I know this is dependent on user information being included and restored with the project in some capacity.
2. Selectively export/import administrative settings including user information. This one probably needs to be smart enough to allow a controlled data merge to take place during an import.
3. This could serve also as the basis of a backup system for various areas of Redmine
I got the idea as I was deleting a test project. If you can delete a project in its entirety, you should be able to archive all of that information before deletion in one simple step as the delete vs copy logic is pretty much the same. And if you can do that and then delete your project, you should be able to take that exact same archive and restore it.
I haven't investigated this at the code level but wanted to post this because it seems a good idea at this moment and a search of data export did not provide any interesting hits
Related issues
Updated by Kirill Ponomarev about 15 years ago
+1
Good idea.
Also it can be a simple way to make backup.
Updated by Hans Fangohr about 15 years ago
I agree this would be great, and just what we need for a number of cases.
This relates to http://www.redmine.org/boards/2/topics/5754#message-5761, where just this functionality is required.
Updated by Mr Embedded about 15 years ago
I would love to help out if I can. I am not a coder. My skill set is more geared to project management and testing/debugging. Let me know if I can assist.
Updated by Nanda P over 14 years ago
+1
Recently I received a request to Import only a specific project from our test server to PROD. (MySQL Server)
I did it manually exported records from table by table & imported them in PROD. Took few hours for about 100 Issues.
It will be nice to have this feature to export a single or multiple project data from one server & import to another RedMine server.
Updated by Andrea Denzler over 14 years ago
+1
Yes, it would be very nice to export/import projects and administration/user settings. Good for backup, moving between servers :)
Updated by Balazs Nadasdi over 14 years ago
+1
We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]
Pls make a migration tool...
/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */
thx
Updated by saturntechweb saturntechweb about 14 years ago
Raito Yitsushi wrote:
+1
We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]Pls make a migration tool...
/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */
thx
dfdf
Updated by Thomas Zlika almost 14 years ago
+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !
Updated by Andrea Denzler almost 14 years ago
Thomas Zlika wrote:
+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !
mmm....
1. If you have two different SVN repositories you must first of all merge them, maintaining all the history and creating a report with OldCommitReference->NewCommitReference. This is not a easy task, in any case it is related only to SVN. With such a report you have to update the RedMine servers renaming oldReference with newReference wherever it was used (wiki, notes, etc).
2. Once you merge two RedMine Servers you need a report with OldIssueID->NewIssueId. With that report you have to update all the Commit Comments in your SVN repository, replacing old issue id with new issue id as described in the Subversion FAQ (the Import process could do that automatically). You need to replace the issue IDs not only in SVN but also in all the notes, wiki of RedMine.
Sure, with a global issue ID (like with GIT commits) it would be easier to merge two servers, but the simple incremental Issue ID number is more human friendly....
Updated by Anthony Cartmell over 13 years ago
+1 This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.
Updated by Jean-Baptiste Barth over 13 years ago
It's nearly impossible to have a clean answer to this question. Something could be built on top of yaml_db
gem (https://github.com/ludicast/yaml_db ; it generates a database-independent export in YAML). But the latest posts are right on issue IDs, and I can't imagine such a change in a near future sorry.
Updated by Hans Fangohr over 13 years ago
Dear Jean-Baptiste,
thank you for taking the time to comment.
Okay, it is technically hard (nearly impossible) to provide this feature, so we'll have to leave without it (unless somebody volunteers to pay [a lot] for the development work).
Suppose several projects grow on one Redmine instance, and one of them needs to be transferred to a new server. Could one clone the whole system, and then on the cloned system delete all projects apart from the desired one? This is not pretty, but at least a workable solution available now.
Would we expect this to work?
Thank you,
Hans
Updated by Jean-Baptiste Barth over 13 years ago
Hans: indeed ! It can be a solution if your projects/users are strongly separated from each other. It could even be partially scripted.
Anyway, if you experience load problems :- you might be able to solve it outside of Redmine, with a caching reverse-proxy, multiple passenger/mongrel instances (or even a different hosting solution), etc.
- you can report any specific problem here, I think we would be happy to help Redmine be better with high traffic loads
Updated by Ryan Cross over 13 years ago
A couple things:
I also saw some code for a tentative way to do this over here: http://www.redmine.org/boards/1/topics/8077
Also, the unique id thing is the main problem definitely. I think there are a couple ways to possibly handle this:
1) start down the path of UUID's for objects. This would also have the benefit of allowing people to choose between a system wide unique issue number or a project wide unique issue number. This could possibly be done with the help of a plugin, but would more likely be best implemented at the core of redmine. See http://en.wikipedia.org/wiki/Universally_unique_identifier for some background. Drupal is a project I know that has some implementations of this http://drupal.org/project/uuid but almost all programming languages have at least one library for using them (ruby has 3-4 i think)
2) use a plugin to manage the mapping of old id's to new id's and then supersede the use of the old id's when linking between version control, issues, wikis, etc.
For example you import issue #253 from site1.com and it becomes #12 on site2.com... when rendering the a link to issue #253 on site2.com, based on per-project settings, it knows to link to issue #12 instead. (The plugin would probably have to create a custom field on the relevant object to store the old id's)
This method could work relatively easily but would probably be a bit hacky. There might need to be a way to resolve/choose which reference you're talking about at times.
Updated by Snehal p almost 13 years ago
Could you please provide me updates about this feature?
I want to transfer redmine data from one machine to another machine.Is there any solution to perform this?
Updated by Jonathan M over 12 years ago
+1
Just built a new server for production and would like to import a project from my old test server. Over 100 issues
Updated by Max Albrecht over 12 years ago
+1
(it seems there is no easy way to switch from sqlite3 to mysql, so a data import/export feature could take care of that.)
Updated by Marcelo Soares Souza over 12 years ago
+1
I need merge several redmine servers (with different databases) in one
Updated by Max Albrecht over 12 years ago
Marcelo Soares Souza wrote:
I need merge several redmine servers (with different databases) in one
I haven't thought of that, but now that you mention it:
Being able to import/export single projects would mean way better data portability, thus also more peace of mind hosting-wise. You could start of a little redmine on your dev box with sqlite3 knowing that if demand gets out of hand you can switch to a 'managed' hosted (like plan.io).
It is already the case with wordpress(.org/.com) and their implementation works really well.
Updated by Dieter Egert over 12 years ago
Can't it be done in this way:
All issue IDs will be extended internally by the according project ID. The projectID can be written in links, but is not needed in all cases.
The syntax for issue links (e.g. in wiki) will be extended by someting like #projectID:issueID
and alternatively with previous syntax #issueID referencing to local project (if #issueID occurs) else to other project (if #issueID occurs there). This means that the issueID will be specific only for 1 project in future (not for complete redmine).
So when a project is imported to a different redmine installation all issues will keep their numbers and will stay linked to the actually imported project. Just links to different projects need to be broken, if the according project is not imported in the same time.
Attachments need to be stored in a own folder per project.
Users will be created as new users, or (if name is same in old and new redmine) will be mapped accordingly.
All needed trackers will be transfered as well as workflows, news, wiki and forums.
Also if not all information is transferred it will be better than nothing.
This feature is really needed, if you use redmine for many projects (and therefore on some installations).
Updated by Jason Palmer over 12 years ago
I don't like the idea of non-unique issue ID's.
We have two instances of Redmine currently, and are looking to merge these. Already staff use the issue ID to point people to the relevant issue, and with two instances this can be very confusing. If the issue ID was only relevant when mentioned in the context of the project there would be chaos! Different business units use different projects, with their work going into subprojects of that B.U. Project space. There is enough cross-functional/cross project team work going on that noone would know what issue 123 meant anymore (we are lazy enough to leave off the project identifier).
Updated by Terence Mill over 12 years ago
Holger Kluge wrote:
Hi,
i also thought about this and stumbled about http://www.rubyrep.org. They have a >screencast for bidirectionally synchronizing two redmine instances at >http://www.rubyrep.org/screencast.html.
I did not test this myself, but it may be worth a try...
Why don't they sync the two tables 'schema_migrations' and plugin_shema_info'?
Does that mean i can have different plugins setups in both redmine instances?
If redmine standard tables are never touched by plugin migration, can this work?
Updated by John NWW about 12 years ago
+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500
Updated by Brad Rushworth almost 12 years ago
John NWW wrote:
+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500
I manage HostedRedmine.com. I would be willing to release this code under GNU Affero General Public License. It is PHP code, but it is simple enough to use and modify.
This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.
This is the main use case. It allows you to extract out particular users from a Redmine installation and all their associated data (issues, wikis, etc). It must be imported into a fresh installation of Redmine, due to unique issue numbering etc.
Updated by Fredrik Svensson almost 12 years ago
Terence Mill wrote:
Holger Kluge wrote:
Hi,
i also thought about this and stumbled about http://www.rubyrep.org. They have a >screencast for bidirectionally synchronizing two redmine instances at >http://www.rubyrep.org/screencast.html.
I did not test this myself, but it may be worth a try...
Why don't they sync the two tables 'schema_migrations' and plugin_shema_info'?
Does that mean i can have different plugins setups in both redmine instances?
If redmine standard tables are never touched by plugin migration, can this work?
have no idea about the tables but the synchronization as i see could have been done with a normal mysql cluster. the fun thing would be to do asynchronous bi-directional sync but for that one need to change stepping of how sequence indexes are generated in redmine.
Updated by Sudhanshu Agrawal over 11 years ago
Brad Rushworth wrote:
John NWW wrote:
+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500
I manage HostedRedmine.com. I would be willing to release this code under GNU Affero General Public License. It is PHP code, but it is simple enough to use and modify.
This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.
This is the main use case. It allows you to extract out particular users from a Redmine installation and all their associated data (issues, wikis, etc). It must be imported into a fresh installation of Redmine, due to unique issue numbering etc.
Hey Brad did you get a chance to release it under GNU? I am in a similar need. Please point me to the location if you published it.
Updated by Brad Rushworth over 11 years ago
Sudhanshu Agrawal wrote:
Hey Brad did you get a chance to release it under GNU? I am in a similar need. Please point me to the location if you published it.
Yes, it is now attached here: http://www.hostedredmine.com/news/500
Updated by Thomas Kohler about 11 years ago
+1
Hello,
there you can find an importer for "RequisitePro" baselines: https://github.com/gen2thomas/redmine_import_reqpro
"RequistePro" is using UID's. This ID's will be imported as custom field "RPUID".
Baselines are human readable xml files with special names and format (schemes). Relations between issues inside a project and also to other projects are supported currently.
The "DevelopmentV05"-Branch is working quite good. I have now imported more than 20000 issues with that plugin into my redmine database.
If anyone can write the same for export in such a baseline format 3 guys will be happy ;o}:
1.) The one who want to use his redmine datas in "RequistePro"
2.) That guy like you, who want to merge some redmine databases together
3.) Me, because I want to clean up my database for all the stuff I have added while testing the importer
Wiki and other associated data are not present in "RequisitePro". For that to export an separate file is possible which will be ignored for importing in "RequisitePro".
Updated by Viktor Berke over 10 years ago
+1
In the meanwhile, those who need a guide, I've compiled a bunch of instructions about data migration and upgrade:
Installing, Migrating & Upgrading Redmine with LDAP on Ubuntu
I hope this may be of help to some of you. For me it took about a week to figure all this out lol.
Updated by Kamil . over 10 years ago
I've found an importer tool (https://github.com/leovitch/redmine_importer/wiki). Maybe it could be used with Brad's exporter (http://www.redmine.org/issues/3647#note-44) somehow?
Updated by Anonymous over 10 years ago
Kamil Kaproń, looks like that importer tool is about 2 years old.. If you don't have any luck getting it to work with the Redmine 2.x range, you might want to try this one.
Updated by Toshi MARUYAMA about 10 years ago
- Related to Feature #17259: Import_Button added
Updated by Toshi MARUYAMA about 10 years ago
- Related to Feature #7884: Import issues from Excel or CSV file added
Updated by name zero almost 10 years ago
Fredrik Svensson wrote:
have no idea about the tables but the synchronization as i see could have been done with a normal mysql cluster. the fun thing would be to do asynchronous bi-directional sync but for that one need to change stepping of how sequence indexes are generated in redmine.
The database system will take care of that. E.g. MySQL will issue even ID's on member1 and odd on member2. This works for >2 members, too.
See MySQL Multimaster replication and auto_increment_increment.
Updated by Roger Hunwicks about 9 years ago
This is also related to #950 which although it only covers imports, has just been committed and will be released in 3.2.0.
Updated by Anonymous over 8 years ago
What ever happened with this?
It'd be really nice to have a way to export the data in a 'clean' form that didn't just dump the entire database.
For example, I have an old, heavily upgraded Redmine instance and I want to start fresh, but import all the data without taking across the schema and any weird tables that various plugins over the years may have left lying around.
Updated by Gerhard Gonter over 8 years ago
It'd be really nice to have a way to export the data in a 'clean' form that didn't just dump the entire database.
FWIW: A few months ago, I had a similar problem: One project on an instance with about 100 projects needed to be extracted and moved to a new server. At that time, I only needed to deal with this project's Wiki (no issues and forum), so I decided to copy the elements via MySQL. The Perl scripts are on GitHub [1].
This may not be what you are looking for or you may need to synchronize issues too. Unless something better comes along - that's why I track this issue here - I will use it again for the next migration task.
Cheers!
GG
Updated by Aleksandar Pavic about 8 years ago
+1, IMHO very important feature... I've tested plugins but it's not the same, now I'll test @Gerhard Gonter's link... Thanks!
Updated by Del Elson over 7 years ago
I created this which might help some people: https://github.com/delatbabel/redmine-migrator
Updated by Toshi MARUYAMA over 7 years ago
- Related to Feature #25625: Turn issue numbers into UUIDs added
Updated by Maximilian Eschenbacher over 3 years ago
I've created https://gitlab.com/meschenbacher/redmine-migration and successfully migrated two projects with a few hundret and a few thousand issues on redmine version 4.1.2.