Redmine

This is caused by a change of MySQL 5.7.3-m13. Please see the following URL for details.

mysql - Creating tables and problems with primary key in Rails - Stack Overflow

The workaround is included in Rails 4.1. But current Redmine is based on Rails 3.2, so it seems that we have to rely on monkey patch for now.

Same problem here using redmine-2.6.0.

# mysqld --version
mysqld  Ver 5.7.5-m15 for linux-glibc2.5 on x86_64 (MySQL Community Server (GPL))
# bundle exec rails --version
Rails 3.2.19
# ruby --version
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux-gnu]

If your're gonna wait for Rails 4 in Redmine 3, maybe an update to the docs to inform it's not compatible with this specific version of MySQL.

Jean-Philippe Lang wrote:

1. it does not pass the issue concurrency test (dead locks), although the 5.7 changelog does not mention

http://dev.mysql.com/doc/refman/5.6/en/mysql-nutshell.html

InnoDB uses a new, faster algorithm to detect deadlocks.

Jean-Philippe Lang wrote:

1. it does not pass the issue concurrency test (dead locks)

#19344 says MySQL 5.6 too.

Toshi MARUYAMA wrote:

#19344 says MySQL 5.6 too.

Indeed, the CI server runs MySQL 5.1.

I had a deeper look at the deadlocks issue and it seems to work when doing SELECT * ... FROM UPDATE instead of SELECT id ... FOR UPDATE. Here is a patch for current trunk tested with mysql5.7, the concurrency test passes for me. Could you give it a try?

On my CentOS7 mariadb-5.5.41-2.el7_0.x86_64:

clean r14128:

$ ruby test/unit/issue_nested_set_concurrency_test.rb
Run options: --seed 12276

# Running:

F.

Finished in 19.053029s, 0.1050 runs/s, 0.4199 assertions/s.

  1) Failure:
IssueNestedSetConcurrencyTest#test_concurrency [test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock;
try restarting transaction:
SELECT `issues`.`id` FROM `issues` WHERE (root_id IN (SELECT root_id FROM issues WHERE id IN (319,316)))
ORDER BY `issues`.`id` ASC FOR UPDATE" to be nil.

2 runs, 8 assertions, 1 failures, 0 errors, 0 skips

r14128 with note-13 patch:

$ ruby test/unit/issue_nested_set_concurrency_test.rb
Run options: --seed 50424

# Running:

F.

Finished in 5.455071s, 0.3666 runs/s, 1.2832 assertions/s.

  1) Failure:
IssueNestedSetConcurrencyTest#test_concurrency [test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock;
try restarting transaction:
SELECT `issues`.* FROM `issues` WHERE `issues`.`root_id` = 432 
ORDER BY `issues`.`id` ASC FOR UPDATE" to be nil.

2 runs, 7 assertions, 1 failures, 0 errors, 0 skips

This change passes test half times, but fails half times on my MariaDB 5.5.

diff --git a/lib/redmine/nested_set/issue_nested_set.rb b/lib/redmine/nested_set/issue_nested_set.rb
--- a/lib/redmine/nested_set/issue_nested_set.rb
+++ b/lib/redmine/nested_set/issue_nested_set.rb
@@ -158,7 +158,8 @@ module Redmine
           self.class.reorder(:id).where(:root_id => sets_to_lock).lock(lock).ids
         else
           sets_to_lock = [id, parent_id].compact
-          self.class.reorder(:id).where("root_id IN (SELECT root_id FROM #{self.class.table_name} WHERE id IN (?))", sets_to_lock).lock.ids
+          root_ids = self.class.where(:id => sets_to_lock).select(:root_id).to_a
+          self.class.where(:root_id => root_ids).lock.ids
         end
       end

$ ruby test/unit/issue_nested_set_concurrency_test.rb
Run options: --seed 63128

# Running:

..

Finished in 25.875842s, 0.0773 runs/s, 0.3865 assertions/s.

2 runs, 10 assertions, 0 failures, 0 errors, 0 skips

$ ruby test/unit/issue_nested_set_concurrency_test.rb
Run options: --seed 40861

# Running:

FF

Finished in 6.222392s, 0.3214 runs/s, 0.6428 assertions/s.

  1) Failure:
IssueNestedSetConcurrencyTest#test_concurrency [test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction: 
UPDATE `issues` SET lft = CASE WHEN lft > 9 THEN lft - 2 ELSE lft END, rgt = CASE WHEN rgt > 9 THEN rgt - 2 ELSE rgt END WHERE `issues`.`root_id` = 4249 AND (lft > 9 OR rgt > 9)" to be nil.

  2) Failure:
IssueNestedSetConcurrencyTest#test_concurrent_subtasks_creation [test/unit/issue_nested_set_concurrency_test.rb:61]:
Expected "Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction: 
UPDATE `issues` SET lft = CASE WHEN lft >= 18 THEN lft + 2 ELSE lft END, rgt = CASE WHEN rgt >= 18 THEN rgt + 2 ELSE rgt END WHERE `issues`.`root_id` = 4263 AND (lft >= 18 OR rgt >= 18)" to be nil.

2 runs, 4 assertions, 2 failures, 0 errors, 0 skips

Toshi, your patch does not do what it's supposed to.

You may want to write:

root_ids = self.class.where(:id => sets_to_lock).pluck(:root_id)

instead of:

root_ids = self.class.where(:id => sets_to_lock).select(:root_id).to_a

which returns records without their ids. The lock after that does nothing:

SELECT `issues`.`id` FROM `issues` WHERE `issues`.`root_id` IS NULL FOR UPDATE

I've isolated the log for a thread that triggers a dead lock. It ends with:

[52206168] BEGIN
[52206168] SELECT  `issues`.* FROM `issues` WHERE `issues`.`id` = 781 LIMIT 1
[52206168] SELECT `issues`.`id` FROM `issues` WHERE `issues`.`root_id` = 778  ORDER BY `issues`.`id` ASC FOR UPDATE
[52206168] ROLLBACK
[52206168] ERROR: Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction: 
SELECT `issues`.`id` FROM `issues` WHERE `issues`.`root_id` = 778  ORDER BY `issues`.`id` ASC FOR UPDATE

As we can see, the thread starts a transaction, has no lock yet and gets a dead lock error on the first lock. Anyone knows what would explain that?

Jean-Philippe Lang wrote:

Toshi, your patch does not do what it's supposed to.

You may want to write:

root_ids = self.class.where(:id => sets_to_lock).pluck(:root_id)

instead of:

root_ids = self.class.where(:id => sets_to_lock).select(:root_id).to_a

which returns records without their ids. The lock after that does nothing:

SELECT `issues`.`id` FROM `issues` WHERE `issues`.`root_id` IS NULL FOR UPDATE

This changes fails 3/4 times.

diff --git a/lib/redmine/nested_set/issue_nested_set.rb b/lib/redmine/nested_set/issue_nested_set.rb
--- a/lib/redmine/nested_set/issue_nested_set.rb
+++ b/lib/redmine/nested_set/issue_nested_set.rb
@@ -158,7 +158,8 @@ module Redmine
           self.class.reorder(:id).where(:root_id => sets_to_lock).lock(lock).ids
         else
           sets_to_lock = [id, parent_id].compact
-          self.class.reorder(:id).where("root_id IN (SELECT root_id FROM #{self.class.table_name} WHERE id IN (?))", sets_to_lock).lock.ids
+          root_ids = self.class.where(:id => sets_to_lock).pluck(:root_id).compact.uniq
+          self.class.where(:root_id => root_ids).lock.ids
         end
       end

$ ruby test/unit/issue_nested_set_concurrency_test.rb 
Run options: --seed 1553

# Running:

F.

Finished in 20.314407s, 0.0985 runs/s, 0.4430 assertions/s.

  1) Failure:
IssueNestedSetConcurrencyTest#test_concurrency [test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction: SELECT `issues`.`id` FROM `issues` WHERE `issues`.`root_id` = 7049 FOR UPDATE" to be nil.

2 runs, 9 assertions, 1 failures, 0 errors, 0 skips

"SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE;" reduces failure times on my MariaDB 5.5.

This is code from source:tags/2.6.3/lib/plugins/awesome_nested_set/lib/awesome_nested_set/model/transactable.rb .

diff --git a/lib/redmine/nested_set/issue_nested_set.rb b/lib/redmine/nested_set/issue_nested_set.rb
--- a/lib/redmine/nested_set/issue_nested_set.rb
+++ b/lib/redmine/nested_set/issue_nested_set.rb
@@ -148,7 +148,29 @@ module Redmine
         new_record? || !is_or_is_ancestor_of?(issue)
       end

+      def in_tenacious_transaction(&block)
+        retry_count = 0
+        begin
+          transaction(&block)
+        rescue ActiveRecord::StatementInvalid => error
+          raise unless error.message =~ /Deadlock found when trying to get lock/
+          raise unless retry_count < 10
+          retry_count += 1
+          logger.info "Deadlock detected on retry #{retry_count}, restarting transaction" 
+          sleep(rand(retry_count)*0.1) # Aloha protocol
+          retry
+        end
+      end
+
       def lock_nested_set
+        if self.class.connection.adapter_name =~ /mysql/i
+          in_tenacious_transaction { lock_nested_set_in_tenacious_transaction }
+        else
+          lock_nested_set_in_tenacious_transaction
+        end
+      end
+
+      def lock_nested_set_in_tenacious_transaction
         if self.class.connection.adapter_name =~ /sqlserver/i
           lock = "WITH (ROWLOCK HOLDLOCK UPDLOCK)" 
           # Custom lock for SQLServer

Note-20 is wrong because it uses nested transaction and parent transaction does not use lock.
This is fix.

diff --git a/lib/redmine/nested_set/issue_nested_set.rb b/lib/redmine/nested_set/issue_nested_set.rb
--- a/lib/redmine/nested_set/issue_nested_set.rb
+++ b/lib/redmine/nested_set/issue_nested_set.rb
@@ -148,7 +148,29 @@ module Redmine
         new_record? || !is_or_is_ancestor_of?(issue)
       end

+      def get_lock_mysql(&block)
+        retry_count = 0
+        begin
+          yield
+        rescue ActiveRecord::StatementInvalid => error
+          raise unless error.message =~ /Deadlock found when trying to get lock/
+          raise unless retry_count < 10
+          retry_count += 1
+          logger.info "Deadlock detected on retry #{retry_count}, restarting transaction" 
+          sleep(rand(retry_count)*0.1) # Aloha protocol
+          retry
+        end
+      end
+
       def lock_nested_set
+        if self.class.connection.adapter_name =~ /mysql/i
+          get_lock_mysql { get_lock }
+        else
+          get_lock
+        end
+      end
+
+      def get_lock
         if self.class.connection.adapter_name =~ /sqlserver/i
           lock = "WITH (ROWLOCK HOLDLOCK UPDLOCK)" 
           # Custom lock for SQLServer

Toshi, the idea of the implementation of nested sets in 3.0.0 is to start the transaction by locking all the rows that might be updated or used to compute shifts in the transaction, in order by prevent dead locks and inconsistencies. I won't commit that workaround until I figure out why it doesn't work as I expect (note-17) in recent versions of MySQL.

I think MySQL uses Gap Locks, so we cannot avoid deadlock.
http://dev.mysql.com/doc/refman/5.6/en/innodb-record-level-locks.html#idm140169015854080

I tried READ COMMITTED on MariaDB 5.5, but deadlock raised. I don't know the reason.

Do current versions of Redmine still not support MySQL 5.7?

#23318#note-18 patch reduces test failure times from about 100% to 50% on my CentOS7 mariadb-5.5.52-1.el7.x86_64.

Latest versions of Ubuntu server only provide the 5.7 package, version 5.5 is really difficult to install on it: https://askubuntu.com/questions/763240/is-it-possible-to-install-mysql-5-5-or-5-6-on-ubuntu-16-04

Here are some logs from MySQL 5.7:

mysql> select * FROM INNODB_LOCKS \G;
*************************** 1. row ***************************
    lock_id: 56163:265:12:54
lock_trx_id: 56163
  lock_mode: X
  lock_type: RECORD
 lock_table: `redmine_test`.`issues`
 lock_index: index_issues_on_root_id_and_lft_and_rgt
 lock_space: 265
  lock_page: 12
   lock_rec: 54
  lock_data: 653, 1, 20, 653
*************************** 2. row ***************************
    lock_id: 56159:265:12:54
lock_trx_id: 56159
  lock_mode: X
  lock_type: RECORD
 lock_table: `redmine_test`.`issues`
 lock_index: index_issues_on_root_id_and_lft_and_rgt
 lock_space: 265
  lock_page: 12
   lock_rec: 54
  lock_data: 653, 1, 20, 653
2 rows in set, 1 warning (0.00 sec)

mysql> SHOW ENGINE INNODB STATUS \G;
------------------------
LATEST DETECTED DEADLOCK
------------------------
2017-04-23 14:38:12 0x7f308c273700

*** (1) TRANSACTION:
TRANSACTION 56161, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 1
LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 9, OS thread handle 139846486136576, query id 1081 localhost root Sending data
SELECT `issues`.`id` FROM `issues` WHERE (root_id IN (SELECT root_id FROM issues WHERE id IN (658,655)))  ORDER BY `issues`.`id` ASC FOR UPDATE

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 265 page no 12 n bits 160 index index_issues_on_root_id_and_lft_and_rgt of table `redmine_test`.`issues` trx id 56161 lock_mode X waiting

Record lock, heap no 47 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
 0: len 4; hex 8000028d; asc     ;;
 1: len 4; hex 80000001; asc     ;;
 2: len 4; hex 80000016; asc     ;;
 3: len 4; hex 8000028d; asc     ;;

*** (2) TRANSACTION:
TRANSACTION 56159, ACTIVE 0 sec updating or deleting
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1136, 60 row lock(s), undo log entries 1
MySQL thread id 7, OS thread handle 139846486537984, query id 1134 localhost root updating
UPDATE `issues` SET lft = CASE WHEN lft > 5 THEN lft - 2 ELSE lft END, rgt = CASE WHEN rgt > 5 THEN rgt - 2 ELSE rgt END WHERE `issues`.`root_id` = 653 AND (lft > 5 OR rgt > 5)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 265 page no 12 n bits 160 index index_issues_on_root_id_and_lft_and_rgt of table `redmine_test`.`issues` trx id 56159 lock_mode X

Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

Record lock, heap no 47 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
 0: len 4; hex 8000028d; asc     ;;
 1: len 4; hex 80000001; asc     ;;
 2: len 4; hex 80000016; asc     ;;
 3: len 4; hex 8000028d; asc     ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 265 page no 12 n bits 160 index index_issues_on_root_id_and_lft_and_rgt of table `redmine_test`.`issues` trx id 56159 lock_mode X locks gap before rec insert intention waiting

Record lock, heap no 47 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
 0: len 4; hex 8000028d; asc     ;;
 1: len 4; hex 80000001; asc     ;;
 2: len 4; hex 80000016; asc     ;;
 3: len 4; hex 8000028d; asc     ;;

*** WE ROLL BACK TRANSACTION (1)

Should we put the index on :issues => [:root_id, :lft, :rgt] as unique? I think this would play a role in the number of records being locked when we do something such as:

  In remove_from_nested_set:
  self.class.where(:root_id => root_id).where("lft >= ? AND rgt <= ?", lft, rgt).update_all(...)

  In add_to_nested_set:
  self.class.where(:root_id => root_id).where("lft >= ? OR rgt >= ?", lft, lft).update_all(...)

Or is a reorder needed before the update_all clause ?

Am about to make the move to Ubuntu 16.04. Can I install Redmine 3.3 and stick with MySQL 5.7 now?

Mark Anderson wrote:

Am about to make the move to Ubuntu 16.04. Can I install Redmine 3.3 and stick with MySQL 5.7 now?

For what it is worth, we have been using an Ubuntu 16.04 + MariaDB 10.0.x setup for over six months now without any obvious issues.

• web server: Ubuntu 16.04, nginx/Passenger, mysql-client 5.7.x
• database server: MariaDB 10.0.x

Hi,
what is the highest tested version of MySql that is supported in Redmine 3.x - 4.x
thanks

yossi edri wrote:

Hi,
what is the highest tested version of MySql that is supported in Redmine 3.x - 4.x
thanks

Salam

redmine 3.2 work by last version of mysql but redmine 3.3 and higher not work well with mysql 5.5 or higher

Just installed Redmine in following ports for FreeBSD. There are included mysql56, and the lowest version available from ports is mysql55. I've been trying Redmine with mysql80. Seems working for the case of mysql. But I got this issue - http://www.redmine.org/boards/2/topics/55693.

Does it make sense to retry the transaction inside the Issue class?

In app/models/issue.rb:

  def self.transaction(options={}, &block)
    retry_count = 0
    begin
      super
    rescue ActiveRecord::StatementInvalid => error
      raise if connection.adapter_name !~ /mysql/i
      raise if error.message !~ /Deadlock found when trying to get lock/
      raise if retry_count == 10
      retry_count += 1
      wait_ms = rand(retry_count * 100)
      if logger
        logger.info("Deadlock found when saving #{self}: "\
          "Waiting for #{wait_ms} ms before restarting the transaction (retry ##{retry_count})")
      end
      sleep(wait_ms / 1000.0)
      retry
    end
  end

Even if the problem with deadlocks still exists on MySQL 5.7, I think that it is safe to say that Redmine 4.0 officially supports MySQL 5.7 and to close this issue.

CI server run the tests on MySQL 5.7 and the install documentation has been updated.

What do you think?

Even if the problem with deadlocks still exists on MySQL 5.7, I think that it is safe to say that Redmine 4.0 officially supports MySQL 5.7 and to close this issue.

CI server run the tests on MySQL 5.7 and the install documentation has been updated.

What do you think?

Loose tangent: Does the same also apply to MariaDB? FWIW, I've run Redmine on MariaDB 10.0 for years without (apparent) issue, hoping to upgrade to a newer MariaDB release in the near future.

Deoren Moor wrote:

Even if the problem with deadlocks still exists on MySQL 5.7, I think that it is safe to say that Redmine 4.0 officially supports MySQL 5.7 and to close this issue.

CI server run the tests on MySQL 5.7 and the install documentation has been updated.

What do you think?

Loose tangent: Does the same also apply to MariaDB? FWIW, I've run Redmine on MariaDB 10.0 for years without (apparent) issue, hoping to upgrade to a newer MariaDB release in the near future.

No, it applies only to MySQL 5.7. MariaDB is not supported (or at least officially) and an user reported some failing tests (please see #30367).

Marius BALTEANU wrote:

Even if the problem with deadlocks still exists on MySQL 5.7, I think that it is safe to say that Redmine 4.0 officially supports MySQL 5.7 and to close this issue.

CI server run the tests on MySQL 5.7 and the install documentation has been updated.

What do you think?

Tests skip on MySQL (r16926).

Toshi MARUYAMA wrote:

Marius BALTEANU wrote:

Even if the problem with deadlocks still exists on MySQL 5.7, I think that it is safe to say that Redmine 4.0 officially supports MySQL 5.7 and to close this issue.

CI server run the tests on MySQL 5.7 and the install documentation has been updated.

What do you think?

Tests skip on MySQL (r16926).

Yes, but the CI uses MySQL 5.7 so is quite strange to say we do not support MySQL 5.7, but we run the tests against MySQL 5.7.

Newer versions of MySQL, after 5.7 are still not applicable? MySQL 8.x.x etc? Also does mysql2 gem support higher versions of MySQL?

This concurrency issue is fixed in the upcoming Redmine 5.1.1 (it will be released soon) and 6.0.0, but it may require some changes to the database settings, please see all the comments from #39437 and the MySQL_configuration page.