MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8)

Background¶

I've seen these errors off/on for a while now and the workaround is always to manually process the emails, either copy/paste into a new "ticket" or strip out all non-ASCII characters and reprocess. As we're increasing use of Redmine within our organization we're running into this a lot more often, especially with groups that process foreign language material.

A small sample of the messages I've seen:

MailHandler: an unexpected error occurred when receiving email: code converter not found (Windows-1258 to UTF-8)
MailHandler: an unexpected error occurred when receiving email: code converter not found (UTF-7 to UTF-8)

Thank you in advance for looking over this and for any suggestions you may have to help resolve it.

Config files¶

grep -ri utf /opt/redmine/config/ | grep -v locales

/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/application.rb:    config.encoding = "utf-8" 
/opt/redmine/config/settings.yml:# encodings used to convert repository files content to UTF-8
/opt/redmine/config/settings.yml:# encoding used to convert commit logs to UTF-8
/opt/redmine/config/settings.yml:  default: 'UTF-8'
/opt/redmine/config/database.yml:  encoding: utf8
/opt/redmine/config/database.yml:  encoding: utf8

`bin/about`¶

Show Hide

sh: 1: darcs: not found
sh: 1: hg: not found
sh: 1: cvs: not found
sh: 1: bzr: not found
sh: 1: git: not found
Environment:
  Redmine version                3.3.4.stable.16752
  Ruby version                   2.3.1-p112 (2016-04-26) [x86_64-linux-gnu]
  Rails version                  4.2.7.1
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.9.3
  Filesystem                     
Redmine plugins:
  no plugin installed

`bundle list`¶

Show Hide

Gems included by the bundle:
  * actionmailer (4.2.7.1)
  * actionpack (4.2.7.1)
  * actionpack-action_caching (1.2.0)
  * actionpack-xml_parser (1.0.2)
  * actionview (4.2.7.1)
  * activejob (4.2.7.1)
  * activemodel (4.2.7.1)
  * activerecord (4.2.7.1)
  * activesupport (4.2.7.1)
  * addressable (2.5.1)
  * arel (6.0.4)
  * builder (3.2.3)
  * bundler (1.11.2)
  * coderay (1.1.1)
  * concurrent-ruby (1.0.5)
  * css_parser (1.5.0)
  * erubis (2.7.0)
  * globalid (0.4.0)
  * htmlentities (4.3.4)
  * i18n (0.7.0)
  * jquery-rails (3.1.4)
  * json (1.8.6)
  * loofah (2.0.3)
  * mail (2.6.6)
  * mime-types (3.1)
  * mime-types-data (3.2016.0521)
  * mimemagic (0.3.2)
  * mini_portile2 (2.1.0)
  * minitest (5.10.2)
  * mysql2 (0.4.7)
  * net-ldap (0.12.1)
  * nokogiri (1.7.2)
  * protected_attributes (1.1.4)
  * public_suffix (2.0.5)
  * rack (1.6.8)
  * rack-openid (1.4.2)
  * rack-test (0.6.3)
  * rails (4.2.7.1)
  * rails-deprecated_sanitizer (1.0.3)
  * rails-dom-testing (1.0.8)
  * rails-html-sanitizer (1.0.3)
  * railties (4.2.7.1)
  * rake (12.0.0)
  * rbpdf (1.19.2)
  * rbpdf-font (1.19.1)
  * redcarpet (3.3.4)
  * request_store (1.0.5)
  * rmagick (2.16.0)
  * roadie (3.2.2)
  * roadie-rails (1.1.1)
  * ruby-openid (2.3.0)
  * sprockets (3.7.1)
  * sprockets-rails (3.2.0)
  * thor (0.19.4)
  * thread_safe (0.3.6)
  * tzinfo (1.2.3)

Scraping script¶

In the script we're using to scrape emails (basically a wrapper around the rake task) we're setting multiple environment variables in an attempt to workaround the issue. Here is a trimmed down version of that script:

Show Hide

# Explicitly set locale settings
#
# https://stackoverflow.com/a/9993718/903870
LANG=en_US.UTF-8
LANGUAGE=$LANG
LC_CTYPE=$LANG
LC_ALL=$LANG

# Export the settings in an attempt to provide the proper values for any called
# tools to work with. The hope is that this will help with character
# conversion issues encountered when scraping emails into Redmine.
export LANG LANGUAGE LC_CTYPE LC_ALL

# Explicitly setting PATH instead of relying on cron to prepare values for us
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

cd /opt/redmine

bundle exec rake -f /opt/redmine/Rakefile redmine:email:receive_imap \
    RAILS_ENV="development" \
    host=${mail_host} \
    username=${account_name}\
    password=${account_pass} \
    move_on_success=accepted_emails \
    move_on_failure=rejected_emails \
    project=unassigned \
    category=unassigned \
    tracker=support \
    no_permission_check=1 \
    unknown_user=accept \
    port=993 \
    ssl=1 \
    2>&1 >> ${email_scraping_script_log_file}

Replies (5)

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Matthew Paul over 7 years ago

Deoren - that info you posted on my forum thread was excellent -

Some related issues: #24992, #21398

#21398#note-10 in particular goes into a good level of detail.

and really it gives the 'solution' which is to run a conversion on your DB to handle utf8mb4. But, it all takes a long time and some technical expertise, neither of which I have. Even your script would seem to require some knowledge of running a wrapper script etc.

For many (like me) - what is needed is a simple plugin that accepts any inbound email, updates what it can in the description/notes and simply converts to '?' (or similar) anything that causes an error. That's not a great loss because we're only talking about emojii and the like in any case. My workflow plugin + script does that, but it's not a single plugin, and it may miss stuff. So I don't know if any such plugin would be available, basically to never 'error out' but always to log what it can as best it can. But if you find something please let me know I'd be interested :-)

I know you're looking for a more complete solution, so thanks again and sorry to bust into this thread...

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Toshi MARUYAMA over 7 years ago

Matthew Paul wrote:

Deoren - that info you posted on my forum thread was excellent -

Some related issues: #24992, #21398

#21398#note-10 in particular goes into a good level of detail.

and really it gives the 'solution' which is to run a conversion on your DB to handle utf8mb4.

No. It is because Ruby does not have code converter of Windows-1258 and UTF-7.

$ cat ja.yml | iconv -f utf8 -t EUC-JP | ruby -ne '$_.force_encoding("EUC-JP") ;puts $_.encode("UTF-8")' > /dev/null
$ cat vi.yml | iconv -f utf8 -t Windows-1258 | ruby -ne '$_.force_encoding("Windows-1258") ;puts $_.encode("UTF-8")' > /dev/null
-e:1:in `encode': code converter not found (Windows-1258 to UTF-8) (Encoding::ConverterNotFoundError)
    from -e:1:in `<main>'
$ cat vi.yml | iconv -f utf8 -t utf-7 | ruby -ne '$_.force_encoding("UTF-7") ;puts $_.encode("UTF-8")' > /dev/null
-e:1:in `encode': code converter not found (UTF-7 to UTF-8) (Encoding::ConverterNotFoundError)
    from -e:1:in `<main>'

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Toshi MARUYAMA over 7 years ago

FYI for UTF-7: #24910.

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Matthew Paul over 7 years ago

So, according to that ticket, it says it can be handled?

require "net/imap"
Net::IMAP.decode_utf7(mail_body)

Would this go into mailer.rb? Or somewhere else?

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Karel Pičman over 4 years ago

I haven't found a better solution than to patch the code.

converter_not_found.patch (811 Bytes) converter_not_found.patch

(1-5/5)

Project

General

Profile

Redmine