Redmine

Background¶

In Redmine’s test suite, each test file specifies the required test data using fixtures. For example, in test/unit/user_test.rb:

class UserTest < ActiveSupport::TestCase
  fixtures :users, :email_addresses, :members, :projects, :roles, :member_roles, :auth_sources, (snip)

This approach causes test failures to occur randomly, as reported in the following issues, among others:

• Patch #41934: Fix random test failure in ProjectsControllerTest::test_post_copy_should_copy_requested_items due to missing :issue_categories fixture
• Patch #41623: Fix tests that randomly failed due to required fixtures not being loaded

For instance, consider the following two test files:

# test/unit/a_test.rb
class ATest < ActiveSupport::TestCase
  fixtures :users

  test "A test" do
    puts "-- A test" 
    assert User.exists?(1)
  end
end

# test/unit/b_test.rb
class BTest < ActiveSupport::TestCase
  test "B test" do
    puts "-- B test" 
    assert User.exists?(1)
  end
end

The result of BTest depends on the execution order:

• A test -> B test: BTest passes.
• B test -> A test: BTest fails.

Execution results:

A test -> B test

$ rm db/test.sqlite3 && bin/rails test test/unit/a_test.rb test/unit/b_test.rb
Run options: --seed 59321

# Running:

-- A test
.-- B test
.

B test -> A test

$ rm db/test.sqlite3 && bin/rails test test/unit/b_test.rb test/unit/a_test.rb
Run options: --seed 22170

# Running:

-- B test
F

Failure:
BTest#test_B_test [test/unit/b_test.rb:6]:
Expected false to be truthy.

bin/rails test test/unit/b_test.rb:4

-- A test
.

The state of the database during tests depends on the execution order and the specified fixtures. This makes it challenging to stabilize tests.

Proposal¶

To address this issue, I propose setting fixtures :all for all tests. This change is expected to offer the following benefits:

Improved Test Stability and Reliability¶

Tests will always run with a consistent set of data, eliminating failures caused by missing fixtures. This ensures reliable and consistent test results.

Easier Test Implementation¶

When writing tests, it is often difficult to identify all necessary fixtures. With this approach, specifying fixtures individually is no longer required.

Performance Impact¶

Applying this patch is expected to have little to no impact on test execution time, as shown below:

Before applying the patch
https://github.com/redmine/redmine/actions/runs/12102413935

After applying the patch
https://github.com/hidakatsuya/redmine/actions/runs/12218226574