Defect #20143
Mailhandler cannot handle 4-byte characters
| Status: | New | Start date: | ||
|---|---|---|---|---|
| Priority: | Low | Due date: | ||
| Assignee: | - | % Done: | 0% | |
| Category: | Database | |||
| Target version: | - | |||
| Resolution: | Affected version: | 3.0.2 |
Description
When receiving an e-mail containing a 4-byte character (an apple-emoji in the example), the mail-handler dies with
Mysql2::Error: Incorrect string value: '\xF0\x9F\x98\x89' for column 'description' at row 1: INSERT INTO `issues` (`project_id`, `author_id`, `priority_id`, `tracker_id`, `status_id`, `subject`, `description`, `start_date`, `updated_on`, `created_on`, `lock_version`) VALUES (15, 4, 2, 3, 1, 'Emoticon test', 'Here comes an emoji: ?~_~X~I', '2015-06-21', '2015-06-21 18:05:05', '2015-06-21 18:05:05', 0)
and no ticket is created.
Related issues
History
#1
Updated by Toshi MARUYAMA over 6 years ago
- Priority changed from High to Low
#2
Updated by Toshi MARUYAMA over 6 years ago
- Category changed from Email receiving to Database
#3
Updated by Toshi MARUYAMA over 6 years ago
Try utf8mb4 (#10772#note-7).
#4
Updated by Toshi MARUYAMA over 6 years ago
- Related to Patch #19742: RedmineInstall: MySQL: collation_database added
#5
Updated by Toshi MARUYAMA over 6 years ago
- Duplicated by Defect #10772: 4-byte utf-8 characters added
#6
Updated by Ragnar Nevries over 6 years ago
Toshi MARUYAMA wrote:
Try utf8mb4 (#10772#note-7).
In fact, that solves the problem, but it does not work for all tables:
MariaDB [redmine]> ALTER TABLE changesets CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
MariaDB [redmine]> ALTER TABLE schema_migrations CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
Wouldn't it be great to have this character encoding used per default or as an option for mysql? Four byte unicode characters are not that rare these days, especially in emails (where the emails client converts ASCII-things to utf8). Sorry if this is already possible, I'm using the great sameersbn Dockerfile for installing/running redmine.
#7
Updated by Toshi MARUYAMA over 6 years ago
Ragnar Nevries wrote:
Wouldn't it be great to have this character encoding used per default or as an option for mysql?
Redmine does not define creating database process.
I don't prefer defining MySQL encoding (#19742#note-3).
I'm using the great sameersbn Dockerfile for installing/running redmine.
Please ask its maintainer.
#8
Updated by Peter Pfläging almost 6 years ago
Ragnar Nevries wrote:
Toshi MARUYAMA wrote:
Try utf8mb4 (#10772#note-7).
In fact, that solves the problem, but it does not work for all tables:
MariaDB [redmine]> ALTER TABLE changesets CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytesMariaDB [redmine]> ALTER TABLE schema_migrations CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytesWouldn't it be great to have this character encoding used per default or as an option for mysql? Four byte unicode characters are not that rare these days, especially in emails (where the emails client converts ASCII-things to utf8). Sorry if this is already possible, I'm using the great sameersbn Dockerfile for installing/running redline.
It works for all tables, if you change the ROW_FORMAT in MySQL/MariaDB to DYNAMIC!
I've made a script for that http://www.pflaeging.net/blog/archives/938 (Sorry, it's in german, but the script ist in sh ;-))
#9
Updated by Marius BALTEANU about 2 years ago
- Related to Feature #31921: Changes to properly support 4 byte characters (emoji) when database is MySQL added