Redmine

utf8

I used the directions on the RedmineInstall page:

CREATE DATABASE redmine CHARACTER SET utf8;

Should it be something else instead?

https://www.google.com/search?q=utf8+4byte+mysql

Toshi,

I understand that you wish to draw my attention to MySQL, utf8 and utf8mb4. Reading over the links you've provided, I understand that MySQL's UTF-8 implementation is only a subset (1-3 bytes) of the complete UTF-8 character set (1-4 bytes). utf8mb4 appears to be MySQL's complete implementation of UTF-8 for versions 5.5.3 and later. I also understand that current versions of PostgreSQL and SQLite3 support the full UTF8 character set.

Based on the RedmineInstall page, I understand that Redmine still supports MySQL 5.0 which doesn't support utf8mb4. Even if you can't update the installation docs to recommend that all MySQL users go with utf8mb4, perhaps by adding a note that all users of MySQL 5.5.3 and newer should use utf8mb4 for full UTF-8 character support would help prevent the need for later conversion.

I found this page:

https://mathiasbynens.be/notes/mysql-utf8mb4

which seems to indicate that converting from MySQL's utf8 to utf8mb4 is doable, but quite a chore. Are those steps complete? Is there support already in the upgrade/maintenance Redmine scripts for converting an existing utf8 character set db to utf8mb4?

I've never had to convert a database's character set before, so any tips/tricks you have would be welcome.

Thanks.

EDIT

I found this post which seems a bit easier to follow:

https://pento.net/2014/04/07/wordpress-and-utf-8/

Deoren Moor wrote:

We have set RAILS_ENV to development for our email scraping, so we were fortunate enough to have this logged when we found that several emails had been rejected:

Hello Deoren, we seem to be running into a similar issue, but I don't know where to look for the relevant log file. There is no error in the redmine log and none in the rake output. In which log file did you find the error?
Thanks!

Rol Sch wrote:

Deoren Moor wrote:

We have set RAILS_ENV to development for our email scraping, so we were fortunate enough to have this logged when we found that several emails had been rejected:

Hello Deoren, we seem to be running into a similar issue, but I don't know where to look for the relevant log file. There is no error in the redmine log and none in the rake output. In which log file did you find the error?
Thanks!

In our case we have Redmine installed via a SVN working copy at /opt/redmine/. There the default directory layout applies. Since we use the development RAILS environment (RAILS_ENV) for our scraping script, the /opt/redmine/log/development.log file is created and holds details related to the scraping process.

Thanks! I believe now that we do have the same issue:
MailHandler: an unexpected error occurred when receiving email: Mysql2::Error: Incorrect string value: '\xC5\x82owy ...' for column 'description' at row 1: INSERT INTO `issues` (`project_id`, `author_id`, `priority_id`, `tracker_id`, `status_id`, `subject`, `description`, `start_date`, `updated_on`, `created_on`, `lock_version`) VALUES (2, 2, 4, 1, 1, 'Test', 'Dodaj szczegółowy opis błędu\r\n\r\n**************************\r\n\r\nVersion: 3.9.2.126\r\n\r\n\r\n\r\n**************************', '2017-03-23', '2017-03-23 10:59:41', '2017-03-23 10:59:41', 0)
Message 31 can not be processed

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

Rol Sch wrote:

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

Do you know if this is proven to be a safe option? For example, if/when Redmine updates are released in the future, do those database schema migrations expect the character set to be configured specifically as utf8 or are those changes separate from the character set?

Rol Sch wrote:

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

I believe, that URL address of article changed to: https://www.pflaeging.net/blog/redmine-8bit-sicher-f%C3%BCr-stickie~flow-installieren/ or https://www.pflaeging.net/blog/redmine-und-emojis-probleme-mit-utf8-und-sonderzeichen/

Related with more discussion: #18866 and #21398