Feature #33658

robots.txt: disallow crawling login, register, and lost password form

Added by Go MAEDA over 1 year ago. Updated 10 months ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:SEO
Target version:4.2.0
Resolution:Fixed

Description

The attached patch adds robots exclusion tags (<meta name="robots" content="noindex,follow,noarchive" />) to login, register, and lost password form.

Currently, Redmine allows web crawlers such as Googlebot and Baiduspider to index those forms. I think it is undesirable for the following reasons:

  • There is no meaningful information there
  • If the Redmine instance requires login, probably the admin doesn't want to publish any information to the internet. It is so bad to be indexed by Google

That can be resolved by the attached patch by adding robots exclusion tags to the pages.

robots-exclusion-tags.patch Magnifier (1.43 KB) Go MAEDA, 2020-06-23 18:04

33658.patch Magnifier (2.92 KB) Go MAEDA, 2020-11-28 09:06

33658-use-robots_txt-instead-of-robots-meta-tag.patch Magnifier (4.17 KB) Go MAEDA, 2020-12-02 09:33

Associated revisions

Revision 20532
Added by Go MAEDA 10 months ago

Robots meta tag to prevent web crawlers from indexing login, register, and lost password form (#33658).

Patch by Go MAEDA.

Revision 20577
Added by Go MAEDA 10 months ago

Use robots.txt instead of robots meta tag to prevent web crawlers from indexing login, register, and lost password form (#33658).

Patch by Go MAEDA.

History

#1 Updated by Go MAEDA 10 months ago

  • File 33658.patchMagnifier added
  • Target version set to Candidate for next major release

Added tests to the patch.

#2 Updated by Go MAEDA 10 months ago

  • Target version changed from Candidate for next major release to 4.2.0

Setting the target version to 4.2.0.

#3 Updated by Go MAEDA 10 months ago

  • Subject changed from Prevent web crawlers from indexing login, register, and lost password form to Robots meta tag to prevent web crawlers from indexing login, register, and lost password form

#4 Updated by Go MAEDA 10 months ago

  • Status changed from New to Closed
  • Assignee set to Go MAEDA
  • Resolution set to Fixed

Committed the patch.

#5 Updated by Holger Just 10 months ago

  • Status changed from Closed to Reopened

Redmine already generates a robots.txt in welcome#robots. Wouldn't this be better suited there, specifically in source:trunk/app/views/welcome/robots.text.erb so that we have a single location where we define behavior for robots?

#6 Updated by Go MAEDA 10 months ago

Holger Just wrote:

Redmine already generates a robots.txt in welcome#robots. Wouldn't this be better suited there, specifically in source:trunk/app/views/welcome/robots.text.erb so that we have a single location where we define behavior for robots?

You are absolutely right. I forgot about the existence of robots.txt for some reason.

The attached new patch uses robots.txt instead of robots meta tag.

#7 Updated by Holger Just 10 months ago

Looks good to me, thanks!

#8 Updated by Go MAEDA 10 months ago

  • Subject changed from Robots meta tag to prevent web crawlers from indexing login, register, and lost password form to robots.txt: disallow crawling login, register, and lost password form
  • Status changed from Reopened to Closed

Committed patch #33658#note-6.

Also available in: Atom PDF