Redmine

When robots.txt is requested, it is returned with the content type text/html instead of text/plain. Is this the case in your installations? Could you please test?

According to my experience this results in the robots.txt file being ignored by the vast majority of indexing bots, even the most popular ones, and thus crawlers index sections that should normally be excluded. This also results in significant increase of the server load.