Redmine

I created a patch, so I will attach it.

Pavel Rosický wrote:

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

Thank you for the feedback.
I confirmed the situation. The cause was that when forwarding mail from Postfix to Redmine using rdm-mailhandler.rb, CRLF was replaced with LF. I think that you are configuring Postfix referring to Forwarding emails from your email server .

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

Yuichi HARADA wrote:

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

I didn't need encoding, so I rewrote the patch.

Using the same regular expression as Mail::Utilities.binary_unsafe_to_crlf is faster.

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index 51a0a5afb..16816689f 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email,
+    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

Benchmark script:

require 'benchmark/ips'

str = "0123456789\r0123456789\r\n0123456789\n" * 1000

Benchmark.ips do |x|
  x.report('/\r?\n|\r\n?/') do
    str.gsub(/\r?\n|\r\n?/, "\r\n")
  end

  x.report('/(?<!\r)\n|\r(?!\n)/') do
    str.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n")
  end

  x.compare!
end

Result:

Warming up --------------------------------------
       /\r?\n|\r\n?/    56.000  i/100ms
/(?<!\r)\n|\r(?!\n)/    73.000  i/100ms
Calculating -------------------------------------
       /\r?\n|\r\n?/    559.559  (± 3.6%) i/s -      2.800k in   5.011150s
/(?<!\r)\n|\r(?!\n)/    731.531  (± 3.0%) i/s -      3.723k in   5.094019s

Comparison:
/(?<!\r)\n|\r(?!\n)/:      731.5 i/s
       /\r?\n|\r\n?/:      559.6 i/s - 1.31x  slower

Committed the patch. Thank you for your contribution.

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

Status: 5.3.0
Diagnostic-Code: x-unix; /etc/postfix/rdm-mailhandler.rb:156:in `gsub': invalid
    byte sequence in US-ASCII (ArgumentError)  from
    /etc/postfix/rdm-mailhandler.rb:156:in `submit'  from
    /etc/postfix/rdm-mailhandler.rb:215:in `<main>'

The message piped by Postfix is UTF-8, with LF endings

If I manually convert the message from LF to CRLF before piping to rdm-mailhandler, it works.

cat mail_in_utf8_with_lf.eml | unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status

I've tried the following `/etc/aliases` config but it doesn't work (same error with gsub:156):

support: "|unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status" 

Sylvain Tissot wrote:

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

[...]

Thank you for reporting the error.

Maybe it is because the value of LANG environment variable on your server is set to the locale which does not support UTF-8. A possible value is "C". Ruby's Encoding.default_external respects LANG (see https://docs.ruby-lang.org/en/2.6.0/Encoding.html#class-Encoding-label-External+encoding). If the LANG is "C", Ruby considers the encoding of the content of files as US-ASCII, not UTF-8.

Could you try again after applying the following patch against the latest rdm-mailhandler.rb?

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index b7284841d..62c658e09 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
+    data = { 'key' => key, 'email' => email.b.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

Hi Go MAEDA,

The LANG environment on the server is en_US.utf8

root@mail:/etc/postfix# echo $LANG
en_US.utf8

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

permission denied. Command output: Request was denied by your Redmine
    server. Possible reasons: email is sent from an invalid email address or is
    missing some information.

Sylvain Tissot wrote:

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Thank you for your helpful feedback. I have just committed the fix in r18544.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

I think the error is irrelevant to this issue. Please open a new issue.