https://www.redmine.org/https://www.redmine.org/favicon.ico?16793021292012-04-26T09:01:38ZRedmineRedmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=376992012-04-26T09:01:38ZEtienne Massip
<ul><li><strong>Category</strong> set to <i>Accounts / authentication</i></li></ul><p>Could you please give more details according to <a class="wiki-page" href="https://www.redmine.org/projects/redmine/wiki/SubmittingBugs">SubmittingBugs</a>?</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=377002012-04-26T09:12:23ZRichard Lee
<ul><li><strong>Assignee</strong> set to <i>Etienne Massip</i></li></ul><p>The character '</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=377012012-04-26T09:15:40ZEtienne Massip
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Assignee</strong> deleted (<del><i>Etienne Massip</i></del>)</li><li><strong>Resolution</strong> set to <i>Invalid</i></li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=377022012-04-26T09:19:16ZEtienne Massip
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>New</i></li><li><strong>Resolution</strong> deleted (<del><i>Invalid</i></del>)</li></ul><p>Sorry, there was a bug here, your post has been truncated.</p>
<p>Here is the content of the mail:</p>
<p>The character '<problematic character>' make redmine Internal Error</p>
<p>My redmine version is 1.2.0<br />When I update the first name of my account, the internal error appear.<br />The error description below:<br />ActiveRecord::StatementInvalid in MyController#account <br />Mysql::Error: Incorrect string value: '\xF0\xA0\xAE\x9F' for column 'firstname' at row 1: UPDATE `users` SET `firstname` = '李飞<problematic character>', `updated_on` = '2012-04-26 10:48:31' WHERE `id` = 1<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `log'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:324:in `execute'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `update_sql'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:339:in `update_sql'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:49:in `update_without_query_dirty'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:19:in `update'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/base.rb:2936:in `update_without_lock'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/locking/optimistic.rb:81:in `update_without_dirty'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/dirty.rb:146:in `update_without_timestamps'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/timestamp.rb:64:in `update_without_callbacks'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/callbacks.rb:282:in `update'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/base.rb:2927:in `create_or_update_without_callbacks'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/callbacks.rb:250:in `create_or_update'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/base.rb:2577:in `save_without_validation'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/validations.rb:1090:in `save_without_dirty'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/dirty.rb:79:in `save_without_transactions'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:229:in `send'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:229:in `with_transaction_returning_status'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `transaction'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:182:in `transaction'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:228:in `with_transaction_returning_status'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:196:in `save'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:208:in `rollback_active_record_state!'<br />C:/redmine/vendor/rails/activerecord/lib/active_record/transactions.rb:196:in `save'<br />C:/redmine/app/controllers/my_controller.rb:60:in `account'</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=377092012-04-26T12:04:39ZEtienne Massip
<ul><li><strong>Subject</strong> changed from <i>The character '</i> to <i>The character '</i></li></ul><p>Could you try with the mysql2 database adapter?</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=377122012-04-26T13:12:30ZEtienne Massip
<ul></ul><p>-</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=516032013-09-05T07:59:14ZNot Relevant
<ul></ul><p>I encountered the same problem and wanted to let you know my findings. <br />My test environment:<br /><pre>
Environment:
Redmine version 2.3.1.stable
Ruby version 1.9.3 (i686-linux)
Rails version 3.2.14
Environment production
Database adapter Mysql2
</pre></p>
<p>The underlying problem is the mysql database. Per default mysql databases and tables are utf8. But only up to 3 caracters:</p>
<p>Here is what the mysql manual states:</p>
<blockquote>
<p>The character set named utf8 uses a maximum of three bytes per character and contains only BMP characters. As of MySQL 5.5.3, the utf8mb4 character set uses a maximum of four bytes per character supports supplemental characters:</p>
</blockquote>
<p>To solve this issue you need to convert your database and tables to utf8mb4</p>
<pre>
ALTER DATABASE redmine_default CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
ALTER TABLE attachments CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE auth_sources CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE boards CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE changes CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE changeset_parents CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE changesets CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE changesets_issues CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE comments CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE custom_fields CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE custom_fields_projects CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE custom_fields_trackers CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
.
.
.
</pre>
<p>And change the database.yml accordingly</p>
<pre>
adapter: mysql2
database: redmine_default
host: localhost
port:
username: redmine_default
password: secret
encoding: utf8mb4
</pre>
<p>Worst part about this bug is: Everything after chars 4 bytes long will be truncated without warning. That's the reason for your "Sorry, there was a bug here, your post has been truncated."</p>
<p>Some sanity check should be done. Not sure if that's redmine or upstream.</p>
<p>And no, this has nothing to do with Account or i18n, it's a general problem.</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=519242013-09-21T10:30:25ZToshi MARUYAMA
<ul><li><strong>Category</strong> changed from <i>Accounts / authentication</i> to <i>Database</i></li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643442015-06-22T03:46:16ZToshi MARUYAMA
<ul><li><strong>Subject</strong> changed from <i>The character '</i> to <i>4-byte utf-8 characters</i></li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643462015-06-22T03:48:32ZToshi MARUYAMA
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-5 priority-4 priority-default closed" href="/issues/19742">Patch #19742</a>: RedmineInstall: MySQL: collation_database</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643492015-06-22T03:49:27ZToshi MARUYAMA
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/18866">Defect #18866</a>: MySQL: disappear after 4-Byte UTF-8</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643572015-06-22T03:54:56ZToshi MARUYAMA
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-3 status-5 priority-4 priority-default closed" href="/issues/19742">Patch #19742</a>: RedmineInstall: MySQL: collation_database</i>)</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643672015-06-22T04:03:44ZToshi MARUYAMA
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Resolution</strong> set to <i>Duplicate</i></li></ul><p>I close this issue because description is Ruby 1.8 (Mysql, not Mysql2).</p>
There are duplicate issues.
<ul>
<li>lose 4-byte utf8 (<a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Defect: MySQL: disappear after 4-Byte UTF-8 (Closed)" href="https://www.redmine.org/issues/18866">#18866</a>)</li>
<li>Mailhandler raises exception (<a class="issue tracker-1 status-1 priority-3 priority-lowest" title="Defect: Mailhandler cannot handle 4-byte characters (New)" href="https://www.redmine.org/issues/20143">#20143</a>)</li>
</ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643722015-06-22T04:06:47ZToshi MARUYAMA
<ul><li><strong>Has duplicate</strong> deleted (<i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/18866">Defect #18866</a>: MySQL: disappear after 4-Byte UTF-8</i>)</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643752015-06-22T04:08:36ZToshi MARUYAMA
<ul><li><strong>Is duplicate of</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/18866">Defect #18866</a>: MySQL: disappear after 4-Byte UTF-8</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=643772015-06-22T04:09:08ZToshi MARUYAMA
<ul><li><strong>Is duplicate of</strong> <i><a class="issue tracker-1 status-1 priority-3 priority-lowest" href="/issues/20143">Defect #20143</a>: Mailhandler cannot handle 4-byte characters</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=826972017-12-18T12:40:53ZJēkabs Jānis Kalniņš
<ul></ul><p>Not Relevant wrote:</p>
<blockquote>
<p>I encountered the same problem and wanted to let you know my findings. <br />My test environment:<br />[...]</p>
<p>The underlying problem is the mysql database. Per default mysql databases and tables are utf8. But only up to 3 caracters:</p>
<p>Here is what the mysql manual states:</p>
<blockquote>
<p>The character set named utf8 uses a maximum of three bytes per character and contains only BMP characters. As of MySQL 5.5.3, the utf8mb4 character set uses a maximum of four bytes per character supports supplemental characters:</p>
</blockquote>
<p>To solve this issue you need to convert your database and tables to utf8mb4</p>
<p>[...]</p>
<p>And change the database.yml accordingly</p>
<p>[...]</p>
<p>Worst part about this bug is: Everything after chars 4 bytes long will be truncated without warning. That's the reason for your "Sorry, there was a bug here, your post has been truncated."</p>
<p>Some sanity check should be done. Not sure if that's redmine or upstream.</p>
<p>And no, this has nothing to do with Account or i18n, it's a general problem.</p>
</blockquote>
<p>Thanks this worked for me <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Defect: Can't post a smiley face in issues description (Closed)" href="https://www.redmine.org/issues/27803">#27803</a></p>
<pre>
ALTER TABLE issues CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
ALTER TABLE journals CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
ALTER TABLE wiki_contents CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
</pre>
<p>In config/database.yml<br />encoding: utf8<br />to<br />encoding: utf8mb4</p> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=932062019-08-17T16:32:40ZMarius BĂLTEANU
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/31921">Feature #31921</a>: Changes to properly support 4 byte characters (emoji) when database is MySQL</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=1006642021-01-24T23:29:18ZMarius BĂLTEANU
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/34235">Defect #34235</a>: Losing data due to unsupported characters</i> added</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=1006682021-01-24T23:30:28ZMarius BĂLTEANU
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/34235">Defect #34235</a>: Losing data due to unsupported characters</i>)</li></ul> Redmine - Defect #10772: 4-byte utf-8 charactershttps://www.redmine.org/issues/10772?journal_id=1006702021-01-24T23:32:52ZMarius BĂLTEANU
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/34235">Defect #34235</a>: Losing data due to unsupported characters</i> added</li></ul>