Redmine

I can confirm this also happens on my Windows box.
Database is in PostgreSQL 8.3 (UTF8 encoding).
Email messages are sent in ISO-8859-1.
Messages are fetched via IMAP.
The mail server is running Zimbra 4.5.13.

The problematic headers were :
Subject: test =?ISO-8859-1?Q?cr=E9ation_t=E2che_email?=
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Resulting task name was "test cr辿ation t但che email".
Same platform info as the original poster, appart from Win2003-R2 OS and PostgreSQL 8.3 database.

Arnaud Lesauvage wrote:

I can confirm this also happens on my Windows box.

Does your problem also temporarily disappear after restarting the Ruby application?

Fredrik Frodlund wrote:

Does your problem also temporarily disappear after restarting the Ruby application?

I always have the problem, but I don't understand what you mean by restarting the ruby application.
In my setup (IIS+fastcgi) I have one ruby.exe process that manages web requests and a scheduled task that launches a ruby.exe process to fetch emails.
This scheduled task reloads the environment every time it runs (very anoying since it takes ~20 seconds of processing time).
I could stop/restart the web service to recycle the fastcgi ruby process but that wouldn't change anything, right ?

Arnaud Lesauvage wrote:

Fredrik Frodlund wrote:

Does your problem also temporarily disappear after restarting the Ruby application?

I always have the problem, but I don't understand what you mean by restarting the ruby application.
In my setup (IIS+fastcgi) I have one ruby.exe process that manages web requests and a scheduled task that launches a ruby.exe process to fetch emails.
This scheduled task reloads the environment every time it runs (very anoying since it takes ~20 seconds of processing time).
I could stop/restart the web service to recycle the fastcgi ruby process but that wouldn't change anything, right ?

I guess since you have a differing platform, that does not apply here... Nevermind. :)

I just reinstalled everything on Archlinux.
Redmine is from trunk, revision 2847. Mongrel
Exact same problem.

I stopped/restarted mongrel to see whether it helped, but it doesn't.

Same problem here. German umlauts (Ä, Ö, Ü) are turned into kanji signs.
Redmine runs on a linux box here. Everything is UTF-8. Only the the mail's subject is inserted with wrong encoding/characters. Everything works fine with message body.

I just ran into a weird thing. I tried sending a mail with just "åäö ÅÄÖ" in the subject and as expected, the characters got converted to kanji. However, I tried adding spaces between each character, like this: "å ä ö Å Ä Ö" and the characters were left intact!

I sure hope this might be a clue as to why it happens.

I also tested something.
Try to put "Ä öl, häuser, für" as subject. No character will be converted into strange signs. It will suffice to put an ÄÖÅ (whatever) as first character of the subject and the rest of the subject will be fine.

Perhaps this is another hint.

I just noted that mails sent in other encodings than UTF8 will also get the "kanji"-treatment. I can also confirm André's test.

Fredrik Frodlund wrote:

I just noted that mails sent in other encodings than UTF8 will also get the "kanji"-treatment. I can also confirm André's test.

Indeed, see my comment , encoding was ISO-8859-1.

It got broken in #2059 (see patch)

The culprit: /app/models/mail_handler.rb:123

issue.subject = email.subject.chomp.toutf8

TMail#subject handles my case correctly (subject in iso-8859-1, with Spanish characters), and I suspect most other cases (as it should).

I don't know any Japanese, so all I can do is ask someone who does to verify this:

TMail::Unquoter.unquote_and_convert_to("=?ISO-8859-1?Q?Nueva_modificaci=F3n?=", "utf-8")
# => "Nueva modificación" (this is correct)
TMail::Unquoter.unquote_and_convert_to("=?iso-2022-jp?B?GyRCJCIkJCQmJCgkKhsoQg==?=", "utf-8")
# => "あいうえお" (this seems OK to me, but I'm no Japanese)

If the result is incorrect, it should be fixed (or monkey patched) in TMail::Unquoter, so as not to break other encodings.

Please notice you may need to require tmail or something.

But I believe that is indeed correct, so the problem may be elsewhere (don't know where though).

-- nachokb

Now that we know what the culprit is, shouldn't we be able to resolve this. I'm just getting a little tired of fixing issues via directly in the database, is all. :)

Fredrik Frodlund wrote:

Now that we know what the culprit is, shouldn't we be able to resolve this. I'm just getting a little tired of fixing issues via directly in the database, is all. :)

Jean Philippe fixed it in a branch: http://www.redmine.org/projects/redmine/repository/revisions/2620/diff/sandbox/rails-2.3/app/models/mail_handler.rb (though I don't know if it will get to a release soon)...

-- nachokb

The code is looking a bit different in r2924. I would like at least to make a local change to make the problem go away. Anyone that can help me with this?

/app/models/mail_handler.rb:124

issue.subject = issue.subject.toutf8 if issue.subject.respond_to?(:toutf8)

Fredrik Frodlund wrote:

The code is looking a bit different in r2924. I would like at least to make a local change to make the problem go away. Anyone that can help me with this?

/app/models/mail_handler.rb:124
[...]

Just comment that line out...

-- nachokb

Wow, it really was that simple. Thanks! :)

Jean-Philippe Lang wrote:

Fix committed in r3247.

Thanks!