Project

General

Profile

Actions
Copy link

Defect #21742

closed

Received text attachments doesn't hold the original encoding on Ruby >= 2.1

Added by Pavel Rosický almost 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Email receiving
Target version:
3.3.0
Start date:
Due date:
% Done:

0%

Estimated time:
Resolution:
Affected version:

Description

Hello,
if I send an email with .txt file encoded in ISO 8859-2, it's received from the mailbox and saved to /files, but the encoding is different (some characters are lost). I think this is originaly caused by the utf-8 header added by email clients. In my opinion attachments shouln't be reencoded. Am I missed somethig?

Attached patch fixes the issue.

Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: base64

Environment:
  Redmine version                3.2.0.devel.15058
  Ruby version                   2.1.5-p273 (2014-11-13) [i386-mingw32]
  Rails version                  4.2.5
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.8.13
  Git                            1.9.5
  Filesystem
Redmine plugins:
  no plugin installed

Files

Download all files
mail_handler.rb.patch (639 Bytes) mail_handler.rb.patch Pavel Rosický, 2016-01-17 18:15
mail_handler_test.rb.patch (1.15 KB) mail_handler_test.rb.patch Pavel Rosický, 2016-01-18 11:55
ticket_with_text_attachment_iso-8859-2.eml.patch (1.2 KB) ticket_with_text_attachment_iso-8859-2.eml.patch Pavel Rosický, 2016-01-18 11:56
original.txt (19 Bytes) original.txt Pavel Rosický, 2016-02-08 10:46
encoded.txt (12 Bytes) encoded.txt Pavel Rosický, 2016-02-08 10:46
Actions
Copy link
 #4

Updated by Toshi MARUYAMA almost 9 years ago

I cannot reproduce on trunk r15148 with ruby-1.9.3-p551.
Test passes without mail_handler.rb.patch .
You can see latin2 encoded file in files/ directory with following change.

diff --git a/app/models/attachment.rb b/app/models/attachment.rb
--- a/app/models/attachment.rb
+++ b/app/models/attachment.rb
@@ -53,7 +53,7 @@ class Attachment < ActiveRecord::Base
   @@thumbnails_storage_path = File.join(Rails.root, "tmp", "thumbnails")

   before_create :files_to_final_location
-  after_rollback :delete_from_disk, :on => :create
+  # after_rollback :delete_from_disk, :on => :create
   after_commit :delete_from_disk, :on => :destroy

   # Returns an unsaved copy of the attachment



$ ruby test/unit/mail_handler_test.rb -n test_mail_with_attachment_latin2
Run options: -n test_mail_with_attachment_latin2 --seed 14783

# Running:

.

Finished in 2.025752s, 0.4936 runs/s, 3.4555 assertions/s.

1 runs, 7 assertions, 0 failures, 0 errors, 0 skips
$ cat files/2016/02/160208173558_yourfile.txt | iconv -f latin2 -t utf8
příliš žluoučký kůn
Actions
Copy link
 #5

Updated by Pavel Rosický almost 9 years ago

retested against revision r15148 without mail_handler.rb.patch

MailHandlerTest#test_mail_with_attachment_latin2 [C:/redmine svn/test/unit/mail_handler_test.rb:590]:
Expected: 19
Actual: 12

About your application's environment
Rails version 4.2.5.1
Ruby version 2.1.5-p273 (x64-mingw32)
RubyGems version 2.2.3
Rack version 1.6.4
Middleware Rack::Sendfile, ActionDispatch::Static, Rack::Lock, #<
ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x000000094378e0>, Rack::
Runtime, Rack::MethodOverride, ActionDispatch::RequestId, Rails::Rack::Logger, A
ctionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::
RemoteIp, ActionDispatch::Reloader, ActionDispatch::Callbacks, ActiveRecord::Con
nectionAdapters::ConnectionManagement, ActiveRecord::QueryCache, ActionDispatch:
:Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDis
patch::ParamsParser, ActionDispatch::XmlParamsParser, Rack::Head, Rack::Conditio
nalGet, Rack::ETag, Rack::ContentLength, RequestStore::Middleware, OpenIdAuthent
ication
Application root C:/redmine svn
Environment development
Database adapter mysql2
Database schema version 20151031095005

Environment:
Redmine version 3.2.0.devel
Ruby version 2.1.5-p273 (2014-11-13) [x64-mingw32]
Rails version 4.2.5.1
Environment production
Database adapter Mysql2
SCM:
Git 1.9.5
Filesystem
Redmine plugins:
no plugin installed

no other tests are failing (except IssueNestedSetConcurrencyTest which supposed to be a known problem on mysql) and with mail_handler.rb.patch applied this one passes as well.

IssueNestedSetConcurrencyTest#test_concurrency [C:/redmine svn/test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock; try restarting
transaction: SELECT `issues`.`id` FROM `issues` WHERE (root_id IN (SELECT root_i
d FROM issues WHERE id IN (800,794))) ORDER BY `issues`.`id` ASC FOR UPDATE" to
be nil.

I attached both attachments, the original and the one saved to files (encoded.txt). Maybe it depends on the environment, I'll try to find out.

Actions
Copy link
 #6

Updated by Pavel Rosický almost 9 years ago

toshio harita MARUYAMA - could you retest it on a more recent Ruby version?

passed - ruby 1.9.3p551 (2014-11-13) [i386-mingw32]
failed - ruby 2.1.5p273 (2014-11-13 revision 48405) [i386-mingw32]
failed - ruby 2.2.3p173 (2015-08-18 revision 51636) [x64-mingw32]
failed - jruby 9.0.5.0-SNAPSHOT (2.2.3) 2016-01-20 775b79d Java HotSpot(TM) 64-Bit Server

Actions
Copy link
 #7

Updated by Toshi MARUYAMA almost 9 years ago

  • Target version set to 3.3.0
Actions
Copy link
 #8

Updated by Toshi MARUYAMA almost 9 years ago

  • Tracker changed from Patch to Defect
  • Subject changed from Received text attachments doesn't hold the original encoding to Received text attachments doesn't hold the original encoding on Ruby >= 2.1
  • Status changed from Needs feedback to New
Actions
Copy link
 #9

Updated by Toshi MARUYAMA almost 9 years ago

  • Status changed from New to Closed

Committed in trunk, thanks.
I have reported to upstream.
https://github.com/mikel/mail/issues/972

Actions
Copy link

Also available in: Atom PDF