Project

General

Profile

Actions

Feature #32938

closed

Rails 6: Zeitwerk support

Added by Pavel Rosický about 4 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Category:
Rails support
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Resolution:
Fixed

Description

https://github.com/fxn/zeitwerk

DEPRECATION WARNING: Initialization autoloaded the constants ActiveRecord::Acts,
 ActiveRecord::Acts::Tree, Redmine::I18n, Redmine::Helpers, Redmine::Helpers::URL
, Redmine::SafeAttributes, Redmine::SubclassFactory, CustomField, Redmine::Utils,
 Redmine::Configuration, Redmine::Scm::Adapters::CommandFailed, IssueRelation,
 Redmine::WikiFormatting::Macros, Redmine::Pagination, Redmine::SudoMode, ApplicationHelper
, Redmine::WikiFormatting::Textile, Redmine::WikiFormatting::Textile::Formatter,
 Redmine::WikiFormatting::Textile::Helper, Redmine::WikiFormatting::HtmlParser
, Redmine::WikiFormatting::Textile::HtmlParser, Redmine::WikiFormatting::Markdown,
 Redmine::WikiFormatting::Markdown::Formatter, Redmine::WikiFormatting::Markdown::HTML,
 Redmine::WikiFormatting::Markdown::Helper, Redmine::WikiFormatting::Markdown::HtmlParser,
 Redmine::Views::ApiTemplateHandler, and Setting.

Being able to do this is deprecated. Autoloading during initialization is going
to be an error condition in future versions of Rails.

Reloading does not reboot the application, and therefore code executed during
initialization does not run again. So, if you reload ActiveRecord::Acts, for example,
the expected changes won't be reflected in that stale Module object.

`config.autoloader` is set to `classic`. These autoloaded constants would have been unloaded if `config.autoloader` had been set to `:zeitwerk`.

Please, check the "Autoloading and Reloading Constants" guide for solutions.

Files

0001-Explicitly-require-a-class-called-in-initializers-th.patch (4.01 KB) 0001-Explicitly-require-a-class-called-in-initializers-th.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0002-Split-multiple-classes-and-modules-that-existed-in-t.patch (18.7 KB) 0002-Split-multiple-classes-and-modules-that-existed-in-t.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0003-Move-lib-redmine-core_ext-to-lib-core_ext.patch (2.29 KB) 0003-Move-lib-redmine-core_ext-to-lib-core_ext.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0004-Remove-require_dependency.patch (2.06 KB) 0004-Remove-require_dependency.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0006-Change-camelcase-rules-to-keep-special-module-or-cla.patch (1.14 KB) 0006-Change-camelcase-rules-to-keep-special-module-or-cla.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0005-Fix-modules-where-folder-structure-does-not-meet-Zei.patch (82.8 KB) 0005-Fix-modules-where-folder-structure-does-not-meet-Zei.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0008-Change-autoload-mode-from-classic-to-zeitwerk.patch (1.01 KB) 0008-Change-autoload-mode-from-classic-to-zeitwerk.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0007-Move-only-modules-and-classes-that-match-Zeitwerk-ru.patch (109 KB) 0007-Move-only-modules-and-classes-that-match-Zeitwerk-ru.patch Mizuki ISHIKAWA, 2021-10-19 08:50
0001-Move-DateValidator-to-app-validators.patch (1.14 KB) 0001-Move-DateValidator-to-app-validators.patch Marius BĂLTEANU, 2021-10-23 00:50
split-lib-diff.patch (11.6 KB) split-lib-diff.patch Mizuki ISHIKAWA, 2021-10-25 09:44
0001-Refactor-lib-diff.rb.patch (19 KB) 0001-Refactor-lib-diff.rb.patch Marius BĂLTEANU, 2021-10-26 09:32
alt-0001-Extract-Redmine-PluginLoader-from-Redmine-Plugin.patch (11 KB) alt-0001-Extract-Redmine-PluginLoader-from-Redmine-Plugin.patch Takashi Kato, 2021-10-27 11:57
alt-0002-Move-preparation-code-to-redmine-preparation.rb.patch (46.4 KB) alt-0002-Move-preparation-code-to-redmine-preparation.rb.patch Takashi Kato, 2021-10-27 11:57
alt-0003-Add-core-extensions-in-prepare-method.patch (2.4 KB) alt-0003-Add-core-extensions-in-prepare-method.patch Takashi Kato, 2021-10-27 11:57
alt-0004-Load-core-plugins-with-require.patch (2.67 KB) alt-0004-Load-core-plugins-with-require.patch Takashi Kato, 2021-10-27 11:57
alt-0005-Use-zeitwerk.patch (5.21 KB) alt-0005-Use-zeitwerk.patch Takashi Kato, 2021-10-27 11:58

Related issues

Related to Redmine - Feature #29914: Migrate to Rails 6.1 with Zeitwerk autoloadingClosedGo MAEDA

Actions
Related to Redmine - Defect #36218: Plugin assets are not copied correctly in trunk r21289ClosedMarius BĂLTEANU

Actions
Related to Redmine - Defect #36245: ActiveSupport::Reloader.to_prepare not working in trunk 21287Resolved

Actions
Related to Redmine - Defect #36287: rake redmine:plugins:assets failsClosedGo MAEDA

Actions
Related to Redmine - Defect #15958: AnonymousUser class not found by pluginClosed

Actions
Related to Redmine - Defect #36835: Redmine::Plugin.assets_directory is not working after r21283ClosedMarius BĂLTEANU

Actions
Has duplicate Redmine - Defect #35004: DEPRECATION WARNING during startupClosed

Actions
Actions #1

Updated by Marius BĂLTEANU about 4 years ago

  • Related to Feature #29914: Migrate to Rails 6.1 with Zeitwerk autoloading added
Actions #2

Updated by Go MAEDA about 4 years ago

  • Category set to Rails support
Actions #3

Updated by Enziin System about 4 years ago

+1 Thank you!

Actions #4

Updated by Marius BĂLTEANU about 3 years ago

Pavel, do you have any working patch for this?

Actions #5

Updated by Pavel Rosický about 3 years ago

sry, I don't. This is a pretty heavy change and Redmine has a lot of technical debt in this area. But I have a few hints based on my findings:

Zeitwerk depends on Rails conventions, especially the folder structure.
1/ initializers
/master/config/initializers/10-patches.rb
depends on

include Redmine::I18n

before lib/redmine.rb is loaded, so it triggers autoload. Split them into modules, require explicitly.

2/ there're multiple module definitions in a single file (many places), that don't match the filename and the expected folder structure.
3/ Ruby / Rails patches are global from the pre Ruby 2.0 era. They're patched non-deterministically in some random file

module ActiveRecord
  class Base
    def self.human_attribute_name(attr, options = {})
      ...
    end
  end
end

or a require chain like this
/master/lib/redmine/core_ext/string.rb

this issue can be avoided by using manual require at startup time, but you may consider using refinements instead
https://www.cloudbees.com/blog/ruby-refinements/

4/ require_dependency should be removed
/master/app/models/principal.rb#L221

here's a Rails guide and there's also an example of how to deal with STI
https://edgeguides.rubyonrails.org/autoloading_and_reloading_constants.html

5/ autoloads based on #constantize
/master/lib/redmine/wiki_formatting.rb#L42

6/ a folder structure violations, a few examples
/master/lib/diff.rb#L3

module RedmineDiff
  class Diff
  end
end

should be
class Diff
end

or
/master/lib/redmine/diff.rb
module Redmine
  class Diff
  end
end

7/ module names should use camel-case, upper-case is reserved for constants
/master/lib/redmine/helpers/url.rb#L24
/master/lib/redmine/version.rb#L7

I think now it's pretty clear this is an extremely breaking change. Redmine is a part 1, but even if we fix all these issues, it's impossible to keep compatibility for plugins, but it's well, unavoidable...

Zeitwerk mode can be enabled by putting

config.load_defaults 6.1

into config/application.rb

and tested by

rails zeitwerk:check

Zeitwerk mode will become mandatory in Rails 7.

I think that decisions like module names and folder structure should be discussed with your core team. It's hard to accept a small PR without a confirmation it fixes anything and on the other hand, a big PR might include changes you won't be eventually willing to accept. I would recommend you to investigate these problems yourself, split them into smaller tasks and comment them with your expectations. The community can help you solve it this way, but this task is too big with many open questions.

Actions #6

Updated by Pavel Rosický about 3 years ago

I had to remove some references due to your spam protection :)

Actions #7

Updated by Go MAEDA about 3 years ago

  • Has duplicate Defect #35004: DEPRECATION WARNING during startup added
Actions #8

Updated by Go MAEDA about 3 years ago

  • Target version set to 5.0.0
Actions #9

Updated by Mizuki ISHIKAWA almost 3 years ago

I haven't tried it, it seems that problem 7 can be avoided by writing config/initializers/inflations.rb.
https://github.com/fxn/zeitwerk#inflection

Actions #10

Updated by Mizuki ISHIKAWA over 2 years ago

I've created a draft patches to discuss Zeitwerk support.
Rails 7.0 is going to remove the old loading methods, so we need to think about how to support the new loading methods. https://weblog.rubyonrails.org/2021/9/15/Rails-7-0-alpha-1-released/ (Zeitwerk Exclusively section)


Description of changes. These patches have been split up as much as possible based on the description in #32938#note-5.

1/ initializers
/master/config/initializers/10-patches.rb
depends on

include Redmine::I18n

before lib/redmine.rb is loaded, so it triggers autoload. Split them into modules, require explicitly.

0001-Explicitly-require-a-class-called-in-initializers-th.patch

There are two ways to work around this, either by enclosing it in Rails.application.reloader.to_prepare or by explicitly writing "require" before using it.
The simpler way is to enclose it in Rails.application.reloader.to_prepare, but the impact on the plugin is so great that we've decided to use the "require" method in 0001-Explicitly-require-a-class-called-in-initializers-th.patch.

Rails.application.reloader.to_prepare do
  # Autoload classes and modules needed at boot time here.
end

2/ there're multiple module definitions in a single file (many places), that don't match the filename and the expected folder structure.

0002-Split-multiple-classes-and-modules-that-existed-in-t.patch

Previously, AnonymousUser was defined in app/models/user.rb, but since it is not correct according to Zeitwerk rules that the class name and file name do not match, I created a new app/models/anonymous_user.rb and split it.
We have also made similar changes to several other classes and modules.

3/ Ruby / Rails patches are global from the pre Ruby 2.0 era. They're patched non-deterministically in some random file

module ActiveRecord
class Base
def self.human_attribute_name(attr, options = {})
...
end
end
end
or a require chain like this
/master/lib/redmine/core_ext/string.rb

this issue can be avoided by using manual require at startup time, but you may consider using refinements instead
https://www.cloudbees.com/blog/ruby-refinements/

0003-Move-lib-redmine-core_ext-to-lib-core_ext.patch
It seemed difficult to fit this into the Zeitwerk rules, so I moved the directory.

4/ require_dependency should be removed
/master/app/models/principal.rb#L221

here's a Rails guide and there's also an example of how to deal with STI
https://edgeguides.rubyonrails.org/autoloading_and_reloading_constants.html

0004-Remove-require_dependency.patch

5/ autoloads based on #constantize

/master/lib/redmine/wiki_formatting.rb#L42

This seems to have been resolved by other changes without a direct fix.

6/ a folder structure violations, a few examples
/master/lib/diff.rb#L3
...

0005-Fix-modules-where-folder-structure-does-not-meet-Zei.patch

7/ module names should use camel-case, upper-case is reserved for constants
/master/lib/redmine/helpers/url.rb#L24
/master/lib/redmine/version.rb#L7

0006-Change-camelcase-rules-to-keep-special-module-or-cla.patch
The effect of changing the class name and module name seemed to be significant, so I overwrote the movement of the camelize method without changing the name.

8 / other

0007-Move-only-modules-and-classes-that-match-Zeitwerk-ru.patch
0008-Change-autoload-mode-from-classic-to-zeitwerk.patch

When I tried to pass everything in the lib directory to autoload_paths, I found it very difficult to make the structure of the plugins and generators directories conform to Zeitwerk's rules. (Relation: 6/ a folder structure violations)
So I changed only the lib/redmine directory to lib/autoloads/redmine directory where I could apply autoloading, and set only the files under lib/autoloads directory as `config.autoload_paths += %W(#{config.root}/lib/autoloads)`.


I already know that applying these patches will cause multiple problems, mainly related to development.

  • probrem-1. The directory change from lib/redmine to lib/autoloads/redmine will cause many patches to fail to apply, and the changelog will be hard to read.
  • probrem-2. I changed redmine.rb to "require" many files directly, but autoload doesn't work for files loaded here. Even if you rewrite ApplicationHelper during development, the changes will not be reflected until you restart `rails s`.

I'm sure there are many other problems with it, so feedback and new patches would be greatly appreciated.

Actions #11

Updated by Marius BĂLTEANU over 2 years ago

Thanks Mizuki for working on this. I have as weel some code, I will share it with you as soon I can.

Actions #12

Updated by Marius BĂLTEANU over 2 years ago

I've committed for now the second patch which is safe regardless the type of autoloader. Also, I've fixed some Rubocop offenses and I've updated the .rubocop_todo.yml to match the new files.

I will continue with more fixes in order to decrease the number of changes required for zeitwerk.

Actions #13

Updated by Marius BĂLTEANU over 2 years ago

For DateValidator, I propose to move it from lib/redmine/core_ext/active_record.rb to app/validators/date_validator.rb. It will work on both classic and zeitwerk and I think it make more sense to stay in the app. Also, other Rails projects do this. Unfortunately, I didn't find a recommended by Rails team.

Actions #14

Updated by Mizuki ISHIKAWA over 2 years ago

Marius BALTEANU wrote:

For DateValidator, I propose to move it from lib/redmine/core_ext/active_record.rb to app/validators/date_validator.rb. It will work on both classic and zeitwerk and I think it make more sense to stay in the app. Also, other Rails projects do this. Unfortunately, I didn't find a recommended by Rails team.

I also think app/validators/date_validator.rb would be better.
Thank you for your work on this issue.

Actions #15

Updated by Marius BĂLTEANU over 2 years ago

Thanks Mizuki! I've already committed the change.

In this moment, we have only one issue left that makes bin/rails zeitwerk:check to fail, is lib/diff.rb. This class is very confusing because we already have the Diff class extracted from the unified_diff.rb. I think it's better to rename the class or even better, to move it to lib/redmine/string_array_diff/ so in the end we will have:
  • lib/redmine/string_array_diff/diff.rb
  • lib/redmine/string_array_diff/diffable.rb

What do you think about this? Once we fix this, we can discuss the remaining deprecation warnings and hopefully we can close this issue.

Any feedback/help is appreciated.

Actions #16

Updated by Mizuki ISHIKAWA over 2 years ago

Marius BALTEANU wrote:

What do you think about this?

I think the directory structure and naming is good.
In which file do you plan to include the Diffable for Array and String? Am I correct in interpreting the changes as in the attached patch?

1. Change the module structure to match the change in directory structure. (RedmineDiff::Diff => Redmine::StringArrayDiff::Diff, RedmineDiff::Diffable => Redmine::StringArrayDiff::Diffable)
2. Keep lib/diff.rb to maintain the behavior of expanding Arrays and Strings when require 'diff' is executed in app/models/wiki_page.rb, etc.

Actions #17

Updated by Marius BĂLTEANU over 2 years ago

Mizuki ISHIKAWA wrote:

Marius BALTEANU wrote:

What do you think about this?

I think the directory structure and naming is good.
In which file do you plan to include the Diffable for Array and String? Am I correct in interpreting the changes as in the attached patch?

1. Change the module structure to match the change in directory structure. (RedmineDiff::Diff => Redmine::StringArrayDiff::Diff, RedmineDiff::Diffable => Redmine::StringArrayDiff::Diffable)
2. Keep lib/diff.rb to maintain the behavior of expanding Arrays and Strings when require 'diff' is executed in app/models/wiki_page.rb, etc.

I think we can move everything to lib/redmine/string_array_diff and update the require in app/modeles/wiki_page.rb and lib/redmine/helpers/diff.rb as I did in attached patch. It works for you? In my CI, all tests pass.

Actions #18

Updated by Mizuki ISHIKAWA over 2 years ago

Marius BALTEANU wrote:

I think we can move everything to lib/redmine/string_array_diff and update the require in app/modeles/wiki_page.rb and lib/redmine/helpers/diff.rb as I did in attached patch. It works for you? In my CI, all tests pass.

Thanks for attaching the patch for clarity.
It looks good and actually worked fine in my environment.

Actions #19

Updated by Takashi Kato over 2 years ago

I made alternative patches to reduce require at initialization time and to increase autoloading coverage.

  • I cut out Redmine::PluginLoader from Redmine::Plugin and use PluginLoader at initialization time. Now, Zeitwerk autoloads Redmine::Plugin and other classes and modules in it.
  • I extracted the preparation code(building menu, setting permissions, queries) in /lib/redmine.rb to Redmine::Preparation and it's now in Rails.application.config. to_prepare block. It reduces the use of require during initialization.
  • When setting the autoloading paths for Zeitwerk, I used the namespace option. It allows that Zeitwerk autoloads libraries under the lib/redmine directory without moving them to another directory.
  • To simplify patching, I've left the core plugins and RedCloth3 out of the autoload for now. I will fix them later.
Actions #20

Updated by Takashi Kato over 2 years ago

The above patches are appliable in current trunk(r21261).

Actions #21

Updated by Marius BĂLTEANU over 2 years ago

Takashi Kato wrote:

The above patches are appliable in current trunk(r21261).

Thanks Takashi for the patches, they look good to me. I've already committed some changes related to require from patch 0004, I've renamed CoreExtensions to CoreExt (r21267) and moved lib/diff.rb to lib/redmine/string_array_diff/diff (r21262).

Applying them on top of the changes committed today fix all the warnings related to zeitwerk:

➜  redmine git:(feature/32938_zeitwerk) ✗ bin/rails zeitwerk:check          
Hold on, I am eager loading the application.
W, [2021-10-28T01:34:34.662994 #61216]  WARN -- : Creating scope :system. Overwriting existing method Enumeration.system.
W, [2021-10-28T01:34:34.858254 #61216]  WARN -- : Creating scope :sorted. Overwriting existing method Group.sorted.
W, [2021-10-28T01:34:34.929581 #61216]  WARN -- : Creating scope :sorted. Overwriting existing method User.sorted.
All is good!
Actions #22

Updated by Mizuki ISHIKAWA over 2 years ago

Takashi Kato wrote:

The above patches are appliable in current trunk(r21261).

I've applied the patch #32938#note-19 and it looks great.
I think this patch supports Zeitwerk in a better way than the patch I posted.

I've tried to run some plugins, and they work well as long as the plugin side is adapted to Zeitwerk's rules.

Actions #23

Updated by Marius BĂLTEANU over 2 years ago

  • Status changed from New to Closed
  • Assignee set to Marius BĂLTEANU
  • Target version deleted (5.0.0)
  • Resolution set to Fixed

Patches committed, thanks everyone for working on this!

Actions #24

Updated by Mizuki ISHIKAWA over 2 years ago

Thank you for your hard work.

Actions #25

Updated by Marius BĂLTEANU over 2 years ago

  • Related to Defect #36218: Plugin assets are not copied correctly in trunk r21289 added
Actions #26

Updated by Go MAEDA over 2 years ago

  • Related to Defect #36245: ActiveSupport::Reloader.to_prepare not working in trunk 21287 added
Actions #27

Updated by Go MAEDA over 2 years ago

  • Related to Defect #36287: rake redmine:plugins:assets fails added
Actions #28

Updated by Go MAEDA over 2 years ago

  • Related to Defect #15958: AnonymousUser class not found by plugin added
Actions #29

Updated by Go MAEDA about 2 years ago

  • Related to Defect #36835: Redmine::Plugin.assets_directory is not working after r21283 added
Actions #30

Updated by Go MAEDA about 2 years ago

  • Status changed from Closed to Reopened

Reopening this issue to handle #36835.

Actions #31

Updated by Marius BĂLTEANU about 2 years ago

  • Status changed from Reopened to Closed
Actions

Also available in: Atom PDF