Problem with rdm-mailhandler.rb and Unicode

Mails, which are forwarded to Redmine with a postfix alias and rdm-mailhandler.rb will be rejected, if these mails contains this unicode
https://emojipedia.org/de/telefonh%C3%B6rer#technical (Can't post it directly here, got a internal server error if I had this char).

The alias:

"kd+tm: "|/usr/lib/postfix/sbin/rdm-mailhandler.rb --url https://support.redmine.local --key oV5VDJinRClSrtozLrQG --project kd-support --unknow-user accept --no-permission-check --assigned-to me"

The error message:

#< #5.7.0 x-unix; Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.> #SMTP#

There is no problem with mails to an IMAP account from which Redmine call the mails, this error happend only with rdm-mailhandler.rb. $LANG is de_DE.UTF-8 and if I use this Unicode character on console (or vi) it's correctly visible.

Environment:
  Redmine version                5.0.5.stable
  Ruby version                   3.1.2-p20 (2022-04-12) [x86_64-linux-gnu]
  Rails version                  6.1.7.3
  Environment                    production
  Database adapter               Mysql2
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                sendmail
Redmine settings:
  Redmine theme                  Purplemine (includes JavaScript)
SCM:
  Subversion                     1.14.2
  Git                            2.39.2
  Filesystem                     
Redmine plugins:
  additionals                    3.0.8
  drag_and_drop                  0.0.5
  redmine_agile                  1.6.5
  redmine_checklists             3.1.22
  redmine_contacts               4.3.7
  redmine_contacts_helpdesk      4.2.1
  redmine_dark                   1.0.2
  redmine_dashboard              2.14.0
  redmine_issues_tree            0.0.15
  redmine_more_previews          5.0.8
  redmine_tint_issues            1.3.2
  redmine_wysiwyg_editor         0.29.0
  redmine_zenedit                2.0.4
  redmineup_tags                 2.0.13

  OS is Debian 12.4

How could I resolve this problem?

Tom

Replies (5)

RE: Problem with rdm-mailhandler.rb and Unicode - Added by Thomas Malkus 5 months ago

Important additional notes:

on our local Redmine installation and also here, I got always a

Internal error

An error occurred on the page you were trying to access.
If you continue to experience problems please contact your Redmine administrator for assistance.

If you are the Redmine administrator, check your log files for details about the error.

if I try to insert thís unicode

U+1F4DE

If I send a mail to the IMAP folder from which Redmine creates a ticket, I got no error, but the view of the phoneset is defect, it's then a Item list with one item, instead the Unicode, i.e. "* -430"

RE: Problem with rdm-mailhandler.rb and Unicode - Added by Holger Just 5 months ago

U+1F4DE is encoded with 4 bytes in UTF-8. To be able to store 4-byte characters in your MySQL database, you have to use the utf8mb4 charset in all text/ varchar columns in your database. Some older MySQL versions (and possibly if you updated your database from a previous Redmine version), you might still use utf8mb3 which was previously also known as just utf8. Thus, you likely have to update your database schema.

Please see #31921 and HowTo_convert_a_database_from_utf8_to_utf8mb4 for some guidelines on how to achieve this.

RE: Problem with rdm-mailhandler.rb and Unicode - Added by Thomas Malkus 5 months ago

Hi,

neither Redmine nor mariadb are updated from old versions, it has been a completely fresh installation last year. Some users have a new mail sig so we have only seen the problem since this week.

+--------------+---------------------+----------------------------+------------------------+----------+----------------+
| CATALOG_NAME | SCHEMA_NAME         | DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME | SQL_PATH | SCHEMA_COMMENT |
+--------------+---------------------+----------------------------+------------------------+----------+----------------+
| def          | information_schema  | utf8mb3                    | utf8mb3_general_ci     | <null>   |                |
| def          | mysql               | utf8mb4                    | utf8mb4_general_ci     | <null>   |                |
| def          | performance_schema  | utf8mb3                    | utf8mb3_general_ci     | <null>   |                |
| def          | sys                 | utf8mb3                    | utf8mb3_general_ci     | <null>   |                |
| def          | redmine             | utf8mb4                    | utf8mb4_general_ci     | <null>   |                |
+--------------+---------------------+----------------------------+------------------------+----------+----------------+

Thanks, but this couldn't be the problem.

Try to insert the char https://emojipedia.org/de/telefonh%C3%B6rer#technical in the editor from https://www.redmine.org, you will get a server Oups too.

Tom

RE: Problem with rdm-mailhandler.rb and Unicode - Added by Holger Just 5 months ago

It may be worthwhile to check the character set of the tables and even the columns too, e.g. the description column of the issues table or the notes column of the journals table in your redmine database. I'm still pretty certain that your error is caused by the utf8mb4 / utf8mb3 issue...

(Also, I'm not sure how redmine.org is setup currently, specifically if it allows emojis at all. It appears that the redmine.org database is still using utf8mb3 too...)

RE: Problem with rdm-mailhandler.rb and Unicode - Added by Thomas Malkus 5 months ago

Thanks for your help!

All database tables and the related columns are utf4mb4.

But I think, I found the problem in database.yml. There was the encoding only utf8 and not utf8mb4.

production:
  adapter: mysql2
  database: redmine
  host: localhost
  encoding: utf8

development:
  adapter: mysql2
  database: redmine_development
  host: localhost
  encoding: utf8mb4

After I change it to utf8mb4 I have no problems with these Unicode characters anymore. But I'm not sure to change this on production db too, could I change it or does it have other effects?

Thanks!

(1-5/5)

Project

General

Profile

Redmine