Actions
Feature #37807
closedAllow access to /robots.txt even if logins are required
Start date:
Due date:
% Done:
0%
Estimated time:
Resolution:
Fixed
Description
Right now, if logins are globally required, the /robots.txt
path is not accessible for search engines since the Welcome#robots
path also observes this setting. Requests to /robots.txt
will thus receive an empty HTTP 401 in this case today.
The attached patch series improves this behavior in multiple ways. The patches were extracted from Planio.
0001-Render-all-visible-projects-in-robots.txt-including-.patch
- While initially not strictly related to the issue, this patch extends the list of projects included in the/robots.txt
to not only list active projects but also closed projects (which are still visible to anonymous). This ensures that the list is correct for cases where the Redmine does not enforce logins. Still, as the/robots.txt
file is intended to be consumed by search engines rather than logged-in users, we only list projects which are visible to Anonymous now. The/robots.txt
output thus does not distinguish between the project visibility of the current user but will only output projects visible to Anonymous.0002-Always-allow-access-to-robots.txt-for-Anonymous.patch
- This patch allows Anonymous to always access/robots.txt
, regardless of theSetting.login_required
setting. Previously, this would have been denied if logins are required.0003-Disallow-all-in-robots.txt-if-login-is-required.patch
- With a required login, Anonymous should not be able to view ANY project information. Even in case some routes are manually excluded from this restriction, we still don't want those to be index by search engines. As such, with required logins, we just instruct all robots to not index anything. This patch also makes sure that we are not leaking any information about public projects in case logins are required. Before this patch (but after the previous patches), we would include a list of all public projects there, even if Anonymous could not see them without a login.
Files
Updated by Go MAEDA almost 2 years ago
- Target version set to Candidate for next major release
Updated by Go MAEDA almost 2 years ago
- Target version changed from Candidate for next major release to 5.1.0
Setting the target version to 5.1.0.
Updated by Go MAEDA almost 2 years ago
- Tracker changed from Patch to Feature
- Status changed from New to Closed
- Assignee set to Go MAEDA
- Resolution set to Fixed
Committed the patches. Thank you for your contribution.
Actions