Redmine

Right now, if logins are globally required, the /robots.txt path is not accessible for search engines since the Welcome#robots path also observes this setting. Requests to /robots.txt will thus receive an empty HTTP 401 in this case today.

The attached patch series improves this behavior in multiple ways. The patches were extracted from Planio.

• 0001-Render-all-visible-projects-in-robots.txt-including-.patch - While initially not strictly related to the issue, this patch extends the list of projects included in the /robots.txt to not only list active projects but also closed projects (which are still visible to anonymous). This ensures that the list is correct for cases where the Redmine does not enforce logins. Still, as the /robots.txt file is intended to be consumed by search engines rather than logged-in users, we only list projects which are visible to Anonymous now. The /robots.txt output thus does not distinguish between the project visibility of the current user but will only output projects visible to Anonymous.
• 0002-Always-allow-access-to-robots.txt-for-Anonymous.patch - This patch allows Anonymous to always access /robots.txt, regardless of the Setting.login_required setting. Previously, this would have been denied if logins are required.
• 0003-Disallow-all-in-robots.txt-if-login-is-required.patch - With a required login, Anonymous should not be able to view ANY project information. Even in case some routes are manually excluded from this restriction, we still don't want those to be index by search engines. As such, with required logins, we just instruct all robots to not index anything. This patch also makes sure that we are not leaking any information about public projects in case logins are required. Before this patch (but after the previous patches), we would include a list of all public projects there, even if Anonymous could not see them without a login.