Feature #37807

Allow access to /robots.txt even if logins are required

Added by Holger Just about 1 month ago. Updated about 1 month ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:Permissions and roles
Target version:5.1.0
Resolution:Fixed

Description

Right now, if logins are globally required, the /robots.txt path is not accessible for search engines since the Welcome#robots path also observes this setting. Requests to /robots.txt will thus receive an empty HTTP 401 in this case today.

The attached patch series improves this behavior in multiple ways. The patches were extracted from Planio.

  • 0001-Render-all-visible-projects-in-robots.txt-including-.patch - While initially not strictly related to the issue, this patch extends the list of projects included in the /robots.txt to not only list active projects but also closed projects (which are still visible to anonymous). This ensures that the list is correct for cases where the Redmine does not enforce logins. Still, as the /robots.txt file is intended to be consumed by search engines rather than logged-in users, we only list projects which are visible to Anonymous now. The /robots.txt output thus does not distinguish between the project visibility of the current user but will only output projects visible to Anonymous.
  • 0002-Always-allow-access-to-robots.txt-for-Anonymous.patch - This patch allows Anonymous to always access /robots.txt, regardless of the Setting.login_required setting. Previously, this would have been denied if logins are required.
  • 0003-Disallow-all-in-robots.txt-if-login-is-required.patch - With a required login, Anonymous should not be able to view ANY project information. Even in case some routes are manually excluded from this restriction, we still don't want those to be index by search engines. As such, with required logins, we just instruct all robots to not index anything. This patch also makes sure that we are not leaking any information about public projects in case logins are required. Before this patch (but after the previous patches), we would include a list of all public projects there, even if Anonymous could not see them without a login.

0002-Always-allow-access-to-robots.txt-for-Anonymous.patch Magnifier (1.49 KB) Holger Just, 2022-10-20 17:27

0001-Render-all-visible-projects-in-robots.txt-including-.patch Magnifier (1.7 KB) Holger Just, 2022-10-20 17:27

0003-Disallow-all-in-robots.txt-if-login-is-required.patch Magnifier (2.1 KB) Holger Just, 2022-10-20 17:27

Associated revisions

Revision 21938
Added by Go MAEDA about 1 month ago

Render all visible projects in robots.txt (including closed projects) (#37807).

Patch by Holger Just.

Revision 21939
Added by Go MAEDA about 1 month ago

Always allow access to /robots.txt for Anonymous (#37807).

Patch by Holger Just.

Revision 21940
Added by Go MAEDA about 1 month ago

Disallow all in /robots.txt if login is required (#37807).

Patch by Holger Just.

History

#1 Updated by Go MAEDA about 1 month ago

  • Target version set to Candidate for next major release

#2 Updated by Go MAEDA about 1 month ago

  • Target version changed from Candidate for next major release to 5.1.0

Setting the target version to 5.1.0.

#3 Updated by Go MAEDA about 1 month ago

  • Tracker changed from Patch to Feature
  • Status changed from New to Closed
  • Assignee set to Go MAEDA
  • Resolution set to Fixed

Committed the patches. Thank you for your contribution.

Also available in: Atom PDF