Feature #27876

Add project id to robots.txt

Added by Grischa Zengel over 4 years ago. Updated over 3 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:SEO
Target version:4.0.0
Resolution:Fixed

Description

Bots found activities by project number and traverse to the whole history:

redmine:443 66.249.76.119 - - [02/Jan/2018:05:42:49 +0000] "GET /redmine/projects/3/activity?from=1979-03-13&show_files=1&user_id=3 HTTP/1.1" 200 8090 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;)"

robots.txt looks like:

User-agent: *
Disallow: /redmine/projects/support/repository
Disallow: /redmine/projects/support/issues
Disallow: /redmine/projects/support/activity
Disallow: /redmine/issues/gantt
Disallow: /redmine/issues/calendar
Disallow: /redmine/activity
Disallow: /redmine/search

Expected:

User-agent: *
Disallow: /redmine/projects/support/repository
Disallow: /redmine/projects/support/issues
Disallow: /redmine/projects/support/activity
Disallow: /redmine/projects/3/repository
Disallow: /redmine/projects/3/issues
Disallow: /redmine/projects/3/activity
Disallow: /redmine/issues/gantt
Disallow: /redmine/issues/calendar
Disallow: /redmine/activity
Disallow: /redmine/search

robot.diff Magnifier (633 Bytes) Toshi MARUYAMA, 2018-01-02 11:02

Associated revisions

Revision 17189
Added by Toshi MARUYAMA over 4 years ago

add project id to robots.txt (#27876)

History

#1 Updated by Grischa Zengel over 4 years ago

I thinks it's better to block the whole numeric project:

User-agent: *
Disallow: /redmine/projects/support/repository
Disallow: /redmine/projects/support/issues
Disallow: /redmine/projects/support/activity
Disallow: /redmine/projects/3
Disallow: /redmine/issues/gantt
Disallow: /redmine/issues/calendar
Disallow: /redmine/activity
Disallow: /redmine/search

#2 Updated by Toshi MARUYAMA over 4 years ago

  • Tracker changed from Defect to Feature

#3 Updated by Toshi MARUYAMA over 4 years ago

Try this patch.

#4 Updated by Grischa Zengel over 4 years ago

It works. Thanks.

User-agent: *
Disallow: /redmine/projects/support/repository
Disallow: /redmine/projects/support/issues
Disallow: /redmine/projects/support/activity
Disallow: /redmine/projects/3/repository
Disallow: /redmine/projects/3/issues
Disallow: /redmine/projects/3/activity
Disallow: /redmine/issues/gantt
Disallow: /redmine/issues/calendar
Disallow: /redmine/activity
Disallow: /redmine/search

#5 Updated by Toshi MARUYAMA over 4 years ago

  • Subject changed from add numerical project to robots.txt to add project id to robots.txt

#6 Updated by Toshi MARUYAMA over 4 years ago

  • Status changed from New to Closed
  • Target version set to 4.0.0
  • Resolution set to Fixed

I think it is no problem because project id is disclosed in project.xml and robots.txt hides non public projects.
source:trunk/app/controllers/welcome_controller.rb@17189#L26

#7 Updated by Go MAEDA almost 4 years ago

  • Category set to SEO

#8 Updated by Marius BALTEANU over 3 years ago

  • Subject changed from add project id to robots.txt to Add project id to robots.txt

Also available in: Atom PDF