Defect #24030

When SVN or Git repository has a commit comment include an emoji (4 bytes charactor), error occurs

Added by Jiamin Wang almost 4 years ago. Updated 4 months ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:SCM
Target version:-
Resolution:Invalid Affected version:3.1.2

Description

I already set the MySQL and all databases' charset to utb8mb4

the error is in fetch_changeset method, when insert the commit comment into database table

error.png (34.2 KB) Toshi MARUYAMA, 2020-04-15 18:13

env.png (49.1 KB) Toshi MARUYAMA, 2020-04-15 18:13

subversion_repository.dump.gz (113 KB) Toshi MARUYAMA, 2020-04-15 19:02

svn.diff Magnifier (4.8 KB) Toshi MARUYAMA, 2020-04-15 19:02

mercurial.hg (445 Bytes) Toshi MARUYAMA, 2020-04-15 19:08

mercurial.png (36.8 KB) Toshi MARUYAMA, 2020-04-15 19:08

same-with-r19706.diff Magnifier (1.95 KB) Toshi MARUYAMA, 2020-04-20 04:33


Related issues

Related to Redmine - Feature #31921: Changes to properly support 4 byte characters (emoji) whe... Closed

Associated revisions

Revision 19698
Added by Toshi MARUYAMA 4 months ago

scm: git: add new test repository which has UTF-8 Emoji committer and comments (#24030)

Revision 19699
Added by Toshi MARUYAMA 4 months ago

scm: git: add test of UTF-8 Emoji committer and comments (#24030)

Revision 19700
Added by Toshi MARUYAMA 4 months ago

fix source indent of previous commit (#24030)

Revision 19701
Added by Toshi MARUYAMA 4 months ago

scm: git: replace test repository text file content meaningfully (#24030)

Revision 19706
Added by Go MAEDA 4 months ago

Fix a RuboCop offense Style/ColonMethodCall due to r19699 (#24030).

History

#1 Updated by Jiamin Wang almost 4 years ago

error occurs in save_revision method in /app/models/git.rb or /app/models/subversion.rb

#2 Updated by Toshi MARUYAMA almost 4 years ago

  • Category changed from Database to SCM

#3 Updated by Tobias Fischer about 1 year ago

This is still an issue in Redmine 3.4.11

Environment:
  Redmine version                3.4.11.stable.18254
  Ruby version                   2.2.1-p85 (2015-02-26) [x86_64-linux]
  Rails version                  4.2.11.1
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.6.17
  Git                            1.7.9.5
  Filesystem                     

Error message in log:

Started GET "/projects/PROJECT/repository/REPO_ID" for 37.24.44.210 at 2019-07-31 09:43:30 +0200
Processing by RepositoriesController#show as HTML
  Parameters: {"id"=>"PROJECT", "repository_id"=>"REPO_ID"}
  Current user: USERNAME (id=5)
Completed 500 Internal Server Error in 145ms (ActiveRecord: 13.2ms)

ActiveRecord::StatementInvalid (Mysql2::Error: Incorrect string value: '\xF0\x9F\xA4\xA6' for column 'comments' at row 1: INSERT INTO `changesets` (`repository_id`, `revision`, `scmid`, `committer`, `commit_date`, `committed_on`, `comments`, `user_id`) VALUES (18, 'a5814b480017f7702d46d084da1aa1a7cb5d05b0', 'a5814b480017f7702d46d084da1aa1a7cb5d05b0', 'Tobias Fischer <tobias.fischer@domain.tld>', '2019-07-04', '2019-07-04 21:35:45', '[ci] gradle file bei vorherigem Commit vergessen

#4 Updated by Tobias Fischer about 1 year ago

Funny thing: Also the comment here above get's truncated at the position of the emoji in my log message. Emojis are a big issue for Redmine!

#5 Updated by Marius BALTEANU 12 months ago

  • Related to Feature #31921: Changes to properly support 4 byte characters (emoji) when database is MySQL added

#6 Updated by Toshi MARUYAMA 4 months ago

I have added new test repository and tests because existing test repository encoding is 'ISO-8859-1'.
source:trunk/test/unit/repository_git_test.rb@19705#L42

On my CentOS 7 MySQL 5.5.62, utf8mb4 environment passes test.

With this change,

diff --git a/test/unit/repository_git_test.rb b/test/unit/repository_git_test.rb
--- a/test/unit/repository_git_test.rb
+++ b/test/unit/repository_git_test.rb
@@ -592,7 +592,8 @@ class RepositoryGitTest < ActiveSupport:
   end

   if File.directory?(REPOSITORY_UTF8_PATH) &&
-      !(Redmine::Database::mysql? && !is_mysql_utf8mb4)
+      # !(Redmine::Database::mysql? && !is_mysql_utf8mb4)
+      true
     def test_utf8_emoji
       repo = Repository::Git.create(
                           :project      => @project,

NOT utf8mb4 environment causes error.

#7 Updated by Toshi MARUYAMA 4 months ago

This is Subversion test repository and test code.
Same results with git.

I give up to commit because it is very risky
if CI server is NOT utf8mb4, tests skip.

#8 Updated by Toshi MARUYAMA 4 months ago

This is Mercurial bundle file and its results.

#9 Updated by Marius BALTEANU 4 months ago

  • Priority changed from High to Normal

Toshi, what do you think if we assign this to 4.2.0? We already have a ticket there for utf8mb4 support on mysql (#32054).

#10 Updated by Toshi MARUYAMA 4 months ago

  • Status changed from New to Closed
  • Resolution set to Invalid

Conclusion:

No error with correct server configuration and "encoding: utf8mb4" of database.yml.
source:trunk/config/database.yml.example@19705#L12

#11 Updated by Toshi MARUYAMA 4 months ago

Marius BALTEANU wrote:

Toshi, what do you think if we assign this to 4.2.0? We already have a ticket there for utf8mb4 support on mysql, please see #32054.

Good idea.

#12 Updated by Toshi MARUYAMA 4 months ago

This is additional patch.
"use same "Redmine::Database.mysql?" as with r19706".

Also available in: Atom PDF