Project

General

Profile

Actions

Patch #41954

open

Update extract_fixtures task with configurable output directory

Added by Ko Nagase 4 months ago. Updated about 23 hours ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Code cleanup/refactoring
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

I noticed that running extract_fixtures rake task failed with the following error.

% bundle exec rake extract_fixtures
rake aborted!
NoMethodError: undefined method `type_cast' for an instance of ActiveRecord::ConnectionAdapters::PostgreSQL::Column (NoMethodError)

          record[col.name] = col.type_cast(record[col.name]) if record[col.name]
                                ^^^^^^^^^^

Attached 1st patch (0001-Update-extract_fixtures-task-with-configurable-dir.patch) solves above issue with configurable output directory by DIR argument (envirionment variable, default ./tmp/fixtures), so reviewing it is helpful.

--- a/lib/tasks/extract_fixtures.rake
+++ b/lib/tasks/extract_fixtures.rake
@@ -19,17 +19,20 @@ desc 'Create YAML test fixtures from data in an existing database.
 Defaults to development database. Set RAILS_ENV to override.'

 task :extract_fixtures => :environment do
+  dir = ENV['DIR'] || './tmp/fixtures'
+  FileUtils.mkdir_p(dir)
+
   sql = "SELECT * FROM %s" 
-  skip_tables = ["schema_info"]
+  skip_tables = ["schema_migrations", "ar_internal_metadata"]
   ActiveRecord::Base.establish_connection
   (ActiveRecord::Base.connection.tables - skip_tables).each do |table_name|
     i = "000" 
-    File.open("#{Rails.root}/#{table_name}.yml", 'w' ) do |file|
+    File.open(File.join(dir, "#{table_name}.yml"), 'w') do |file|
       data = ActiveRecord::Base.connection.select_all(sql % table_name)
       file.write data.inject({}) { |hash, record|
         # cast extracted values
         ActiveRecord::Base.connection.columns(table_name).each { |col|
-          record[col.name] = col.type_cast(record[col.name]) if record[col.name]
+          record[col.name] = ActiveRecord::Type.lookup(col.type).deserialize(record[col.name]) if record[col.name]
         }
         hash["#{table_name}_#{i.succ!}"] = record
         hash

I tested mysql2, postgresql and sqlite3 adapters with attached test_extract_fixture.sh file on my local M1 Mac + Ruby 3.3.6 environment.

bash test_extract_fixture.sh mysql2
bash test_extract_fixture.sh postgresql
bash test_extract_fixture.sh sqlite3


Attached 2nd to 5th patches are for sort stability (2nd) and time offset/format things (3rd to 5th) and these are optional.


Files

0001-Update-extract_fixtures-task-with-configurable-dir.patch (1.64 KB) 0001-Update-extract_fixtures-task-with-configurable-dir.patch Ko Nagase, 2024-12-09 09:35
test_extract_fixture.sh (620 Bytes) test_extract_fixture.sh Ko Nagase, 2024-12-09 09:35
0002-Sort-records-for-stability.patch (1.64 KB) 0002-Sort-records-for-stability.patch Ko Nagase, 2024-12-09 09:39
0003-Use-UTC-time-for-datetime-column-value.patch (1.26 KB) 0003-Use-UTC-time-for-datetime-column-value.patch Ko Nagase, 2024-12-09 09:39
0004-Support-time-offset-for-datetime-column-value.patch (1.58 KB) 0004-Support-time-offset-for-datetime-column-value.patch Ko Nagase, 2024-12-09 09:39
0005-Override-Psych-yaml-format_time-method.patch (1.17 KB) 0005-Override-Psych-yaml-format_time-method.patch Ko Nagase, 2024-12-09 09:39
test_extract_fixture_with_time_offset.sh (729 Bytes) test_extract_fixture_with_time_offset.sh Ko Nagase, 2024-12-09 09:42
0006-Support-configurable-skip-tables.patch (1.14 KB) 0006-Support-configurable-skip-tables.patch Ko Nagase, 2024-12-13 09:24
0007-Support-table-filters.patch (1.51 KB) 0007-Support-table-filters.patch Ko Nagase, 2024-12-13 09:24
0008-Use-record-id-for-fixture-key-id-if-exists.patch (1.49 KB) 0008-Use-record-id-for-fixture-key-id-if-exists.patch Ko Nagase, 2024-12-13 09:24
0009-Support-configurable-tables.patch (1.33 KB) 0009-Support-configurable-tables.patch Ko Nagase, 2025-03-30 12:05
0010-Support-omit-default-or-nil-value.patch (1.88 KB) 0010-Support-omit-default-or-nil-value.patch Ko Nagase, 2025-03-30 12:05
0011-Skip-empty-table.patch (1.91 KB) 0011-Skip-empty-table.patch Ko Nagase, 2025-03-30 12:05
0012-Convert-wiki_content_versions-data-binary-to-utf-8.patch (931 Bytes) 0012-Convert-wiki_content_versions-data-binary-to-utf-8.patch Ko Nagase, 2025-03-30 12:05
Actions #1

Updated by Ko Nagase 4 months ago

Attached patches (6th to 8th) are also optional.
Full option example:

bundle exec rake extract_fixtures \
  DIR=./tmp/fixtures \
  TIME_OFFSET=+09:00 \
  SKIP_TABLES=import_items,imports \
  TABLE_FILTERS="issues:id IN (1,2);projects:id IN (1,2)" 

Restore example:
bundle exec rake db:drop && bundle exec rake db:create && bundle exec rake db:migrate
bundle exec rake db:fixtures:load FIXTURES_PATH=./tmp/fixtures

Actions #2

Updated by Go MAEDA 3 months ago

  • Target version set to Candidate for next major release
Actions #3

Updated by Go MAEDA about 2 months ago

  • Target version changed from Candidate for next major release to 6.0.4

Setting the target version to 6.0.4.

Actions #4

Updated by Ko Nagase about 1 month ago

Sorry, I noticed that the 1st main fix patch seems to have a bug in binary data serialization (especially in sqlite3+docker environment).
https://github.com/sanak/redmine-storybook/pull/3/files#diff-e4ce7c298a84bf5e310db65d0682beb599851b37ba5995799ad4522713851834

I will take a look at it in this weekend.

Actions #5

Updated by Go MAEDA 29 days ago

  • Target version changed from 6.0.4 to 6.0.5
Actions #6

Updated by Ko Nagase about 23 hours ago

Ko Nagase wrote in #note-4:

Sorry, I noticed that the 1st main fix patch seems to have a bug in binary data serialization (especially in sqlite3+docker environment).
https://github.com/sanak/redmine-storybook/pull/3/files#diff-e4ce7c298a84bf5e310db65d0682beb599851b37ba5995799ad4522713851834

I will take a look at it in this weekend.

The cause of binary data serialization issue seems to depend on whether the data includes multibyte string like Japanese,
and additional Base64 (ASCII-8BIT) to UTF-8 encode conversion solved the issue, so 1st core patch itself should not be problem, I think.
https://github.com/sanak/redmine-storybook/pull/3/commits/72e55372e5ef49f5d36077cd85b5d4c8104eaa08

I upload the encode conversion diff as 12th patch with additional 9-11th patches.
(The total diff can be seen at here (https://github.com/sanak/redmine/pull/28) .)

Actions

Also available in: Atom PDF