Project

General

Profile

Actions

Patch #41961

closed

Use `fixtures :all` to ensure consistent test data and improve test reliability

Added by Katsuya HIDAKA 10 days ago. Updated 8 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Code cleanup/refactoring
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Background

In Redmine’s test suite, each test file specifies the required test data using fixtures. For example, in test/unit/user_test.rb:

class UserTest < ActiveSupport::TestCase
  fixtures :users, :email_addresses, :members, :projects, :roles, :member_roles, :auth_sources, (snip)
This approach causes test failures to occur randomly, as reported in the following issues, among others:

For instance, consider the following two test files:

# test/unit/a_test.rb
class ATest < ActiveSupport::TestCase
  fixtures :users

  test "A test" do
    puts "-- A test" 
    assert User.exists?(1)
  end
end

# test/unit/b_test.rb
class BTest < ActiveSupport::TestCase
  test "B test" do
    puts "-- B test" 
    assert User.exists?(1)
  end
end

The result of BTest depends on the execution order:

  • A test -> B test: BTest passes.
  • B test -> A test: BTest fails.

Execution results:

A test -> B test

$ rm db/test.sqlite3 && bin/rails test test/unit/a_test.rb test/unit/b_test.rb
Run options: --seed 59321

# Running:

-- A test
.-- B test
.

B test -> A test

$ rm db/test.sqlite3 && bin/rails test test/unit/b_test.rb test/unit/a_test.rb
Run options: --seed 22170

# Running:

-- B test
F

Failure:
BTest#test_B_test [test/unit/b_test.rb:6]:
Expected false to be truthy.

bin/rails test test/unit/b_test.rb:4

-- A test
.

The state of the database during tests depends on the execution order and the specified fixtures. This makes it challenging to stabilize tests.

Proposal

To address this issue, I propose setting fixtures :all for all tests. This change is expected to offer the following benefits:

Improved Test Stability and Reliability

Tests will always run with a consistent set of data, eliminating failures caused by missing fixtures. This ensures reliable and consistent test results.

Easier Test Implementation

When writing tests, it is often difficult to identify all necessary fixtures. With this approach, specifying fixtures individually is no longer required.

Performance Impact

Applying this patch is expected to have little to no impact on test execution time, as shown below:

Before applying the patch
https://github.com/redmine/redmine/actions/runs/12102413935

After applying the patch
https://github.com/hidakatsuya/redmine/actions/runs/12218226574


Files

Actions #1

Updated by Go MAEDA 10 days ago

  • Target version set to 5.1.6

Setting the target version to 5.1.6.

Actions #2

Updated by Go MAEDA 8 days ago

  • Status changed from New to Resolved
  • Assignee set to Go MAEDA

Committed the patch in r23391.

With this update, random test failures caused by missing fixtures will no longer occur, and there will be no need to spend time identifying the required fixtures. Thank you for your contribution.

Actions #3

Updated by Go MAEDA 8 days ago

  • Subject changed from Use fixtures :all to ensure consistent test data and improve test reliability to Use `fixtures :all` to ensure consistent test data and improve test reliability
Actions #4

Updated by Go MAEDA 8 days ago

  • Status changed from Resolved to Closed

Merged the change into the stable branches in r23404 and r23405.

Actions

Also available in: Atom PDF