Redmine

Here are some precisions about the incoming emails:

Looking at the source code of the emails, here is what i noticed.

• BAD INCOMING MAIL TURNED INTO EMPTY ISSUE
  Content-Type: text/plain; charset=utf-8; format=flowed
  Content-Transfer-Encoding: 8bit

• GOOD MAIL TURNED INTO FULLFILLED ISSUE
  Content-Type: text/plain; charset=UTF-8
  Content-Transfer-Encoding: quoted-printable

I really don't have any idea why 8bit encoding would cause such a behavior, but the fact is that all
bad emails are formated this way.

Ok, i can confirm this is the cause.

I'm using thunderbird to send emails, and it's configured to transfer mails in 8bits by default.
It has always been the case and worked until the upgrade in 3.2.8.
Now i had to setup thunedrbird to use MIME for outgoing email. With that email arrive with "Content-Transfer-Encoding: quoted-printable" and are well processed by the mail_handler.

I think it's clearly a bug, because it has always worked through the past. I won't tell my customers to change their MTA configuration to send me emails.

If it can help someone...

encountering same parsing bug with extended utf-8 characters. same discovery as comment 1.

When set to unknown_user=create we get

MailHandler: ignoring email from unknown user []
Completed 422 Unprocessable Entity in 4ms (ActiveRecord: 0.0ms)

example of source of email that causes problem below:

From: REDACTED
Subject: emdash test
To: REDACTED
Message-ID: <eba059c5-7557-9744-61e2-8c52c46cea86@palantetech.coop>
Date: Tue, 5 Dec 2017 13:33:28 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------F142713BE64E3D9019D87845" 
Content-Language: en-US

This is a multi-part message in MIME format.
--------------F142713BE64E3D9019D87845
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

— — — —

--------------F142713BE64E3D9019D87845
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64

PGh0bWw+DQogIDxoZWFkPg0KICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCiAgPC9oZWFkPg0KICA8Ym9k
eSB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4NCiAgICA8cD48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUPC9zcGFuPjwvcD4NCiAgICA8
cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3MiI+LS0gDQpKZXNzaWUgUm9zZSBM
ZWUNClBhbGFudGUgVGVjaG5vbG9neSBDb29wZXJhdGl2ZQ0KPGEgY2xhc3M9Im1vei10eHQt
bGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL3BhbGFudGV0ZWNoLmNvb3AiPmh0dHA6Ly9w
YWxhbnRldGVjaC5jb29wPC9hPg0KKDg4OCkgNDgxLTM2NTUgeC4gMTA2PC9wcmU+DQogIDwv
Ym9keT4NCjwvaHRtbD4NCg==
--------------F142713BE64E3D9019D87845--

current redmine version:

Environment:
  Redmine version                3.4.3.stable
  Ruby version                   2.2.5-p319 (2016-04-26) [x86_64-linux]
  Rails version                  4.2.8
  Environment                    production
  Database adapter               Mysql2
SCM:
  Bazaar                         2.7.0
  Git                            2.1.4
  Filesystem                     
  GitRemote                      2.1.4
Redmine plugins:
  clipboard_image_paste          1.12
  redmine_git_remote             0.0.1
  redmine_local_avatars          1.0.3
  redmine_mattermost             0.6
  redmine_stealth                0.6.0
  redmine_wiki_extensions        0.8.1

MTA has same version of ruby

ran irb rdm-mailhandler.rb

handler = RedmineMailHandler.new
#<RedmineMailHandler:0x000000010b9c48 @issue_attributes={"project"=>"catch", "tracker"=>"support"}, @verbose=true, @url="REDACTED", @key="REDACTED">
exit(handler.submit(STDIN.read))
Posting to https://REDACTED/mail_handler...
Response received: 422
Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.
                        

In can confirm the issue in 3.4.3.stable. The issue didn't occur in 3.4.1. After updating to 3.4.3 I'm experiencing the same issue as mentioned by Jessie Lee.
I ran a few quick test and for the exact same E-Mail I get an

MailHandler: ignoring email from unknown user []

if

Content-Transfer-Encoding: 8bit

while

Content-Transfer-Encoding: 7bit

or

Content-Transfer-Encoding: quoted-printable

works as expected an opens a ticket for the correct user. However I seems to depend on the content itself. Not all message with the 8bit enconding get bounced.

Environment:
  Redmine version                3.4.3.stable
  Ruby version                   2.3.1-p112 (2016-04-26) [x86_64-freebsd10.3]
  Rails version                  4.2.8
  Environment                    production
  Database adapter               Mysql2
SCM:
  Git                            2.14.1
  Filesystem                     
Redmine plugins:
  redmine_checklists             3.1.10
  redmine_default_custom_query   1.3.0
  redmine_ldap_sync              2.0.9.devel

I have to correct myself. It seems that also for 7bit encoding I can trigger the bug. If for instance sending a mail which contains a German Umlaut äüö...

Sorry, 3.2.8 Gemfile does not have mail gem entry.
Try this.

diff --git a/Gemfile b/Gemfile
--- a/Gemfile
+++ b/Gemfile
@@ -17,6 +17,7 @@ gem "actionpack-xml_parser" 
 gem "roadie-rails", "~> 1.1.1" 
 gem "roadie", "~> 3.2.1" 
 gem "mimemagic" 
+gem "mail", "= 2.6.4" 

 gem "nokogiri", (RUBY_VERSION >= "2.1" ? "~> 1.7.2" : "~> 1.6.8")
 gem "i18n", "~> 0.7.0" 

can confirm that downgrading to 2.6.4 mail gem worked for me.

cheers

I think you would better report to mail gem team.
https://github.com/mikel/mail/issues

binding mail gem to 2.6.4 (was 2.7.0. in my rvm env) solved it for 3.4.3 as well.

Julien TEHERY wrote:

Here are some precisions about the incoming emails:

Looking at the source code of the emails, here is what i noticed.

• BAD INCOMING MAIL TURNED INTO EMPTY ISSUE
  Content-Type: text/plain; charset=utf-8; format=flowed
  Content-Transfer-Encoding: 8bit

• GOOD MAIL TURNED INTO FULLFILLED ISSUE
  Content-Type: text/plain; charset=UTF-8
  Content-Transfer-Encoding: quoted-printable

I really don't have any idea why 8bit encoding would cause such a behavior, but the fact is that all
bad emails are formated this way.

• source:tags/3.4.3/test/fixtures/mail_handler
• https://github.com/mikel/mail/tree/2.6.6/spec/fixtures/emails