Redmine

HI.
OK, this forum is dead :-(

Hi Rainer,

Much of my daily work is conding Redmine plugins. Moreover, we use Redmine DMSF. Therefore, it could be that I can help you.

What exactly is the problem? Do you have some error messages or the like?

Best Regards from Germany,

Liane

Hi Liane.

Thank you very much for your answer. At least, you're the only one.
.-)

Our problem regarding the DMSF-Plugin for Redmine is:

We are currently updating our Redmine-System from an old Linux Debian 9 and
Redmine 3.3.1.stable to Linux Debian 10 (buster) and Redmine 4.0.7.stable.

We can not use the most current RM release 4.1.1 because it's not integrated
in den Debian 10 repositories so far.

Unfortunately we get in much trouble because of the DMSF-Plugin which is
heavenly used by our students and employees.

Old version: redmine_dmsf: 1.5.9
New version we try to add: 2.4.5

We did a lot of tries to get it running and

convert the user data

to the new version so far.

https://www.redmine.org/plugins/redmine_dmsf

We followed the instructions step by step including all patches.
But without any success until now.

My question: Is there is somebody out there who can help to fix this issue?

This PI is very important for us.

-

OK, by using the rake option "VERSION=0" we found a way to update
DMSFm but:

We don't want to upgrade to DMSF with Redmine 4 WITHOUT LOOSING
THE USER-DATA!

All trials we found in the internet did not work.

Any idea to upgrade without loosing the user data?

THANK YOU.

--
Rainer Rutka
rainer.rutka@uni-konstanz.de

Hi Rainer,

I understand that you are trying to upgrade Redmine Dmsf plugin.

The migration seems not to work, right? Did you get an error message during migration?

In general there are plenty of reasons why something fails. Therefore, it would be helpful for me when you can send some more details. Sometimes temporary code changes are necessary to finish an interrupted migration.

I agree with you, "VERSION=0" is absolutely no option!

Best Regards,
Liane

One way (often it helps me):
Have you tried to upgrade the DSMF on the old system to 1.6.x before migrating to the new redmine system?

Hi.
Again THANK YOU.
I'll send all the required informations ASAP. Still collecting logs and more.
Your help is welcome.
CU.
Rainer

Hi Liane.
We are currently more than one person who are working on this problem.
Is it possible and allowed to contact you directly per E-Mail?
The logs are full of personal data (ip-adresses) we don't want to
share here without modifications.

rainer.rutka@uni-konstanz

C S wrote:

One way (often it helps me):
Have you tried to upgrade the DSMF on the old system to 1.6.x before migrating to the new redmine system?

NO WE DID NOT!!! :-)

Hi Rainer,

I understand.

I will contact you at uni-konstanz very soon.

Best regards,
Liane

Hi Liane.

In case of lack of direct communication, I'll forward the logs made by my colleague Markus
Grandpre here in this forum:

<snap>

Dear Liane,

I'd like to provide you with some information about our attempts to migrate the DMSF plugin

from

• DMSF 1.5.9,
• Redmine 3.3.1.stable,
• PostGreSQL DB 10.16,
• Debian GNU/Linux 9.13 (stretch)

to

• DMSF 2.4.5,
• Redmine 4.0.7.stable,
• PostGreSQL DB 10.16,
• Debian GNU/Linux 10 (buster).

whereby the PostGreSQL DB is not localhost. This is our migration approach:

1. remove all DMSF 1.5.9 related data from DB,
2. remove DMSF 1.5.9 plugin,
3. update OS to Debian 10,
4. update to Redmine 4.0.7 with packages from a Debian 10 backport,
5. reinstall DMSF 2.4.5 plugin

After several attempts we made the experience that we can only install DMSF 2.4.5 properly if we had removed all related DMSF data from the Redmine DB before. Because the attempt to remove all DMSF 1.5.9 related data with VERSIN=0 from DB resulted an error:

# cd /usr/share/redmine
# bundle exec rake redmine:plugins:migrate NAME=redmine_dmsf VERSION=0 RAILS_ENV=production
...
StandardError: An error has occurred, this and all later migrations canceled:

undefined method `container_type' for #<DmsfFile:0x0056194c645378>
Did you mean?  container
...

we followed the instructions from https://github.com/danmunn/redmine_dmsf/issues/975) and removed all DMSF 1.5.9 related data from DB by hand:

DROP TABLE dmsf_file_revision_accesses;
DROP TABLE dmsf_files;
DROP TABLE dmsf_folders;
DROP TABLE dmsf_links;
DROP TABLE dmsf_locks;
DROP TABLE dmsf_file_revisions;
DROP TABLE dmsf_workflow_step_actions;
DROP TABLE dmsf_workflow_step_assignments;
DROP TABLE dmsf_workflow_steps;
DROP TABLE dmsf_workflows;

-- Remove the following columns: projects.dmsf_%
ALTER TABLE projects
DROP COLUMN dmsf_description;

ALTER TABLE projects
DROP COLUMN dmsf_notification;

-- Remove the following columns: members.dmsf_%
ALTER TABLE members
DROP COLUMN dmsf_mail_notification;

-- Remove related schema migrations
DELETE FROM schema_migrations 
WHERE version LIKE '%redmine_dmsf%';

As you can see, we did not backup all DMSF 1.5.9 related data before we had removed the data, because we have no idea how to backup and to restore this data properly for the use with DMSF 2.4.5. After the OS has successfully been upgraded to Debian 10 and Redmine 3.3.1 has successfully been upgraded to version 4.0.7, we installed the DMSF 2.4.5 (from https://www.redmine.org/plugins/redmine_dmsf) plugin successfully without any errors by:

# cd /usr/share/redmine
# cp -r /somewhere/redmine_dmsf plugins/
# bundle install
# bundle exec rake redmine:plugins:migrate NAME=redmine_dmsf RAILS_ENV=production
# systemctl restart apache2

Due to the loss of DSMF data Redmine 4.0.7 started properly with DMSF 2.4.5 but empty. So I'd like to ask you:

1. Is there a save way to backup all DMSF 1.5.9 related data in order to restore and use the data with DMSF 2.4.5?
2. Is there a way to re-access all DMSF 1.5.9 managed files and folders in /var/redmine/files/dmsf with DMSF 2.4.5?

Your help is highly appreciated.

With kind regards,
Markus
</snap>

Hope this is the required information.

All the Best,

Rainer

Hi Rainer, hi Markus,

What a terrible story!

There are generally several ways possible to restore the data before someone really needs to delete them all by dropping the tables.

Backup and restore data¶

The first what comes in my mind is a db export of everything or only the respective tables in a database independent format such that yaml or csv format. A yaml file would be perfect since there are build-in rake tasks in Rails.

rake db:fixtures:load
    Loads fixtures into the current environment's database. Load specific fixtures using FIXTURES=x,y. Load from subdirectory in test/fixtures using FIXTURES_DIR=z. Specify an alternative path (eg. spec/fixtures) using FIXTURES_PATH=spec/fixtures.

When using this method each table should have its own yaml file. The data structure in such a file might be slighty adapted to the structure as typically used for test fixtures. You can read about it here: https://api.rubyonrails.org/v5.2.5/classes/ActiveRecord/FixtureSet.html.

When importing via the rake task from yaml files, then the import works only table wise. If this is not what you want in any case then you need a tiny script to import the data record wise, i.e. the data will be imported via ActiveRecord, the Rails model layer. In such a script you could also update records where only some columns refer to dmsf data.

The second approach for backing up the data is even more database independent as the first but similar. There is a redmine rake task for exporting all data as yaml.

rake extract_fixtures
    Create YAML test fixtures from data in an existing database.
    Defaults to development database. Set RAILS_ENV to override.

However, the code is not mainained and therefore, not working anymore. You can fix it, by changing in lib/tasks/extract_fixtures.rake as follows:

task :extract_fixtures => :environment do
  sql = "SELECT * FROM %s" 
  skip_tables = ["schema_info"]
  ActiveRecord::Base.establish_connection
  connection = ActiveRecord::Base.connection
  (connection.tables - skip_tables).each do |table_name|
    i = "000" 
    File.open("#{Rails.root}/#{table_name}.yml", 'w' ) do |file|
      data = connection.select_all(sql % table_name)
      file.write data.each_with_object({}) { |record, hash|
        # cast extracted values
        connection.columns(table_name).each { |col|
        next unless record[col.name]

        record[col.name] = connection.type_cast(record[col.name], col) 
       }
        hash["#{table_name}_#{i.succ!}"] = record
        hash
      }.to_yaml
    end
  end
end

I hope there is no copy-paste error or similar in the code above, because I implemented it slightly different but wanted to give yo a fix what is closer on the original method. So let me know if it is not working.

EDIT: What I forgott to mention is, that you should check if there are columns in an old table that does not exist in the corresponding new table after upgrading. If so, the columns needs to be renamed or sorted out, respectively. Otherwise you will get an import error.

Restore files¶

When you did a backup and restored the data then it should be no problem to access the files as you did before any update.

If you don't have the respective data then the context for each file is missing. The only thing you can derive from the file name is the record id of the :dmsf_files table, since each file gets a prefix consisting of timestamp and id like so:

(timestamp)_(id)_(filename and ending)

Therefore, you won't know to which project the files actually belong. Sorry, there is no chance for an assigment.

If you will have a hard time doing similar tasks in Redmine sometime, please contact me. I can give you professional support with my company and help you before you waste your time and lost important data.

Best Regards,
Liane

Hi Liane.

If you will have a hard time doing similar tasks in Redmine sometime, please contact me. I can give you
professional support with my company and help you before you waste your time and lost important data.

And yes. Ja, das ist genau was wir suchen. What's what we're looking for.
A professional support company (located in Germany). Including support contract.
For the rest: Let's see what's up.

Again: Please contact me here rainer.rutka@uni-konstanz.de.

.-)