Defect #19550
closed
Issue query: crash when processing saved query(filters) with cyrillic after upgrade to 3.0 from 2.*
0%
Description
After updrading from 2 to 3 version I've found a bug.
Some if saved custom queries have become broken. Those that have enumeration of values in filter with cyrillic symbols. I've attached some information from log.
When I create new custom query at 3.0 version in works properly.
Files
Updated by Toshi MARUYAMA over 9 years ago
- Status changed from New to Needs feedback
See Submissions.
Updated by Pavel Liavonau over 9 years ago
- File about_env.txt about_env.txt added
Added output of about script. What else do you need?
Updated by Toshi MARUYAMA over 9 years ago
Check your MySQL setting and characters.
http://www.redmine.org/boards/1/topics/36573?r=36587#message-36587
Updated by Pavel Liavonau over 9 years ago
- File mysql_encoding_info.txt mysql_encoding_info.txt added
Attached output of mysql character set information. What can you advice me to do?
Updated by Toshi MARUYAMA over 9 years ago
- Status changed from Needs feedback to Closed
- Resolution set to Invalid
You need to set "UTF-8" for all MySQL setting.
http://www.redmine.org/boards/1/topics/36573?r=36629#message-36629
And you need to convert all MySQL data from latin1 to UTF-8.
http://www.redmine.org/boards/2/topics/35979?r=36082#message-36082
Updated by Pavel Liavonau over 9 years ago
- File queries.filters_new_without_error.txt queries.filters_new_without_error.txt added
- File queries.filters_old_with_error.txt queries.filters_old_with_error.txt added
- Status changed from Closed to Reopened
Your advice has not helped me.
My all settings are now only for UTF-8, but error still occures.
It is the bug. Admit it.
Conversion of dump don't help because the problem is at queries.filters column.
I've added 2 files with content of this column for not old not working query and new created with the same functionality.
Look: old format is "binary" but new is not.
So, fix it, or propose me script that converts queries.filters field properly.
Thanks.
Updated by Toshi MARUYAMA over 9 years ago
- Status changed from Reopened to Closed
- Resolution changed from Invalid to Wont fix
Pavel Liavonau wrote:
Look: old format is "binary" but new is not.
Old format is Ruby 1.8 yaml format.
http://stackoverflow.com/questions/7576500/ruby-1-8-7-why-to-yaml-converts-some-strings-to-non-readable-bytes
It is the bug. Admit it.
It is not bug.
I don't know your old yaml data.
I think it is saved in non UTF-8.
Redmine treat yaml data as UTF-8 nevertheless it is Ruby 1.8 yaml data or not.
source:tags/3.0.3/app/models/custom_field.rb#L121
We cannot support in your case such as database encoding was not UTF-8.
You need to convert yaml data manually.
Updated by Toshi MARUYAMA over 9 years ago
Toshi MARUYAMA wrote:
I don't know your old yaml data.
I think it is saved in non UTF-8.
It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.
$ RAILS_ENV=development rails console
Loading development environment (Rails 4.2.2)
1.9.3-p551 :001 > x = YAML.load('
1.9.3-p551 :002'> - !binary |
1.9.3-p551 :003'>
1.9.3-p551 :004'> 0JDQl9CY0JzQo9Ci
1.9.3-p551 :005'>
1.9.3-p551 :006'> - !binary |
1.9.3-p551 :007'>
1.9.3-p551 :008'> 0JDQstGC0L7QutC+0LzQv9C+0LfQuNGC
1.9.3-p551 :009'>
1.9.3-p551 :010'> - !binary |
1.9.3-p551 :011'>
1.9.3-p551 :012'> 0JHQodCX
1.9.3-p551 :013'>
1.9.3-p551 :014'> - !binary |
1.9.3-p551 :015'>
1.9.3-p551 :016'> 0JHQldCb0JrQkNCg0JQ=
1.9.3-p551 :017'>
1.9.3-p551 :018'> - !binary |
1.9.3-p551 :019'>
1.9.3-p551 :020'> 0JHQvtCx0YDQkNCz0YDQvtC80LDRiA==
1.9.3-p551 :021'>
1.9.3-p551 :022'> - !binary |
1.9.3-p551 :023'>
1.9.3-p551 :024'> 0JLQmNCi0K/Ql9Cs
1.9.3-p551 :025'>
1.9.3-p551 :026'> - !binary |
1.9.3-p551 :027'>
1.9.3-p551 :028'> 0JPQldCk0JXQodCi
1.9.3-p551 :029'>
1.9.3-p551 :030'> - !binary |
1.9.3-p551 :031'>
1.9.3-p551 :032'> 0JfQnNCX
1.9.3-p551 :033'>
1.9.3-p551 :034'> - !binary |
1.9.3-p551 :035'>
1.9.3-p551 :036'> 0JrQnNCX
1.9.3-p551 :037'>
1.9.3-p551 :038'> - !binary |
1.9.3-p551 :039'>
1.9.3-p551 :040'> 0JrQntCg0J/Qo9Ch
1.9.3-p551 :041'>
1.9.3-p551 :042'> - !binary |
1.9.3-p551 :043'>
1.9.3-p551 :044'> 0JrQo9Cf0J7Qmw==
1.9.3-p551 :045'>
1.9.3-p551 :046'> - !binary |
1.9.3-p551 :047'>
1.9.3-p551 :048'> 0JvQtdCx0LXQtNGP0L3RjA==
1.9.3-p551 :049'>
1.9.3-p551 :050'> - !binary |
1.9.3-p551 :051'>
1.9.3-p551 :052'> 0JzQkNCX
1.9.3-p551 :053'>
1.9.3-p551 :054'> - !binary |
1.9.3-p551 :055'>
1.9.3-p551 :056'> 0JzQotCX
1.9.3-p551 :057'>
1.9.3-p551 :058'> - !binary |
1.9.3-p551 :059'>
1.9.3-p551 :060'> 0J3QldCk0JDQlw==
1.9.3-p551 :061'>
1.9.3-p551 :062'> - !binary |
1.9.3-p551 :063'>
1.9.3-p551 :064'> 0J/Qu9Cw0L3QsNGA
1.9.3-p551 :065'>
1.9.3-p551 :066'> - !binary |
1.9.3-p551 :067'>
1.9.3-p551 :068'> 0KDQtdGB0L/QuNGA0LDRgtC+0YA=
1.9.3-p551 :069'>
1.9.3-p551 :070'> - !binary |
1.9.3-p551 :071'>
1.9.3-p551 :072'> 0KDQo9CR0JjQnQ==
1.9.3-p551 :073'>
1.9.3-p551 :074'> - !binary |
1.9.3-p551 :075'>
1.9.3-p551 :076'> 0KHQkNCX
1.9.3-p551 :077'>
1.9.3-p551 :078'> - !binary |
1.9.3-p551 :079'>
1.9.3-p551 :080'> 0KHQsNC70Y7Rgg==
1.9.3-p551 :081'>
1.9.3-p551 :082'> - !binary |
1.9.3-p551 :083'>
1.9.3-p551 :084'> 0KHQrdCT0Jc=
1.9.3-p551 :085'>
1.9.3-p551 :086'> - !binary |
1.9.3-p551 :087'>
1.9.3-p551 :088'> 0KHRgtCw0L3QutC+0LDQs9GA0LXQs9Cw0YI=
1.9.3-p551 :089'>
1.9.3-p551 :090'> - !binary |
1.9.3-p551 :091'>
1.9.3-p551 :092'> 0KLRgNC+0LjRhtC6
1.9.3-p551 :093'>
1.9.3-p551 :094'> - !binary |
1.9.3-p551 :095'>
1.9.3-p551 :096'> 0KfQotCX
1.9.3-p551 :097'>
1.9.3-p551 :098'> ')
=> ["\xD0\x90\xD0\x97\xD0\x98\xD0\x9C\xD0\xA3\xD0\xA2", "\xD0\x90\xD0\xB2\xD1\x82\xD0\xBE\xD0\xBA\xD0\xBE\xD0\xBC\xD0\xBF\xD0\xBE\xD0\xB7\xD0\xB8\xD1\x82", "\xD0\x91\xD0\xA1\xD0\x97", "\xD0\x91\xD0\x95\xD0\x9B\xD0\x9A\xD0\x90\xD0\xA0\xD0\x94", "\xD0\x91\xD0\xBE\xD0\xB1\xD1\x80\xD0\x90\xD0\xB3\xD1\x80\xD0\xBE\xD0\xBC\xD0\xB0\xD1\x88", "\xD0\x92\xD0\x98\xD0\xA2\xD0\xAF\xD0\x97\xD0\xAC", "\xD0\x93\xD0\x95\xD0\xA4\xD0\x95\xD0\xA1\xD0\xA2", "\xD0\x97\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9E\xD0\xA0\xD0\x9F\xD0\xA3\xD0\xA1", "\xD0\x9A\xD0\xA3\xD0\x9F\xD0\x9E\xD0\x9B", "\xD0\x9B\xD0\xB5\xD0\xB1\xD0\xB5\xD0\xB4\xD1\x8F\xD0\xBD\xD1\x8C", "\xD0\x9C\xD0\x90\xD0\x97", "\xD0\x9C\xD0\xA2\xD0\x97", "\xD0\x9D\xD0\x95\xD0\xA4\xD0\x90\xD0\x97", "\xD0\x9F\xD0\xBB\xD0\xB0\xD0\xBD\xD0\xB0\xD1\x80", "\xD0\xA0\xD0\xB5\xD1\x81\xD0\xBF\xD0\xB8\xD1\x80\xD0\xB0\xD1\x82\xD0\xBE\xD1\x80", "\xD0\xA0\xD0\xA3\xD0\x91\xD0\x98\xD0\x9D", "\xD0\xA1\xD0\x90\xD0\x97", "\xD0\xA1\xD0\xB0\xD0\xBB\xD1\x8E\xD1\x82", "\xD0\xA1\xD0\xAD\xD0\x93\xD0\x97", "\xD0\xA1\xD1\x82\xD0\xB0\xD0\xBD\xD0\xBA\xD0\xBE\xD0\xB0\xD0\xB3\xD1\x80\xD0\xB5\xD0\xB3\xD0\xB0\xD1\x82", "\xD0\xA2\xD1\x80\xD0\xBE\xD0\xB8\xD1\x86\xD0\xBA", "\xD0\xA7\xD0\xA2\xD0\x97"]
1.9.3-p551 :099 > x.each { |i| puts i }
АЗИМУТ
Автокомпозит
БСЗ
БЕЛКАРД
БобрАгромаш
ВИТЯЗЬ
ГЕФЕСТ
ЗМЗ
КМЗ
КОРПУС
КУПОЛ
Лебедянь
МАЗ
МТЗ
НЕФАЗ
Планар
Респиратор
РУБИН
САЗ
Салют
СЭГЗ
Станкоагрегат
Троицк
ЧТЗ
=> ["\xD0\x90\xD0\x97\xD0\x98\xD0\x9C\xD0\xA3\xD0\xA2", "\xD0\x90\xD0\xB2\xD1\x82\xD0\xBE\xD0\xBA\xD0\xBE\xD0\xBC\xD0\xBF\xD0\xBE\xD0\xB7\xD0\xB8\xD1\x82", "\xD0\x91\xD0\xA1\xD0\x97", "\xD0\x91\xD0\x95\xD0\x9B\xD0\x9A\xD0\x90\xD0\xA0\xD0\x94", "\xD0\x91\xD0\xBE\xD0\xB1\xD1\x80\xD0\x90\xD0\xB3\xD1\x80\xD0\xBE\xD0\xBC\xD0\xB0\xD1\x88", "\xD0\x92\xD0\x98\xD0\xA2\xD0\xAF\xD0\x97\xD0\xAC", "\xD0\x93\xD0\x95\xD0\xA4\xD0\x95\xD0\xA1\xD0\xA2", "\xD0\x97\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9E\xD0\xA0\xD0\x9F\xD0\xA3\xD0\xA1", "\xD0\x9A\xD0\xA3\xD0\x9F\xD0\x9E\xD0\x9B", "\xD0\x9B\xD0\xB5\xD0\xB1\xD0\xB5\xD0\xB4\xD1\x8F\xD0\xBD\xD1\x8C", "\xD0\x9C\xD0\x90\xD0\x97", "\xD0\x9C\xD0\xA2\xD0\x97", "\xD0\x9D\xD0\x95\xD0\xA4\xD0\x90\xD0\x97", "\xD0\x9F\xD0\xBB\xD0\xB0\xD0\xBD\xD0\xB0\xD1\x80", "\xD0\xA0\xD0\xB5\xD1\x81\xD0\xBF\xD0\xB8\xD1\x80\xD0\xB0\xD1\x82\xD0\xBE\xD1\x80", "\xD0\xA0\xD0\xA3\xD0\x91\xD0\x98\xD0\x9D", "\xD0\xA1\xD0\x90\xD0\x97", "\xD0\xA1\xD0\xB0\xD0\xBB\xD1\x8E\xD1\x82", "\xD0\xA1\xD0\xAD\xD0\x93\xD0\x97", "\xD0\xA1\xD1\x82\xD0\xB0\xD0\xBD\xD0\xBA\xD0\xBE\xD0\xB0\xD0\xB3\xD1\x80\xD0\xB5\xD0\xB3\xD0\xB0\xD1\x82", "\xD0\xA2\xD1\x80\xD0\xBE\xD0\xB8\xD1\x86\xD0\xBA", "\xD0\xA7\xD0\xA2\xD0\x97"]
Updated by Pavel Liavonau over 9 years ago
Toshi MARUYAMA wrote:
It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.
what do you mean?
If it is UTF-8 already, what can I do now? I am not so strong in Ruby to write conversion script by myself.
I use ruby ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux].
Updated by Toshi MARUYAMA over 9 years ago
Pavel Liavonau wrote:
Toshi MARUYAMA wrote:
It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.what do you mean?
$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "日本語"
>>> s.decode("UTF-8").encode("EUC-JP")
'\xc6\xfc\xcb\xdc\xb8\xec'
$ irb
1.8.7-head :001 > require 'yaml'
=> true
1.8.7-head :002 > "\xc6\xfc\xcb\xdc\xb8\xec".to_yaml
=> "--- !binary |\nxvzL3Ljs\n\n"
1.8.7-head :003 > "日本語".to_yaml
=> "--- !binary |\n5pel5pys6Kqe\n\n"
Updated by Pavel Liavonau over 9 years ago
Ок. I see.
It is hard to create universal script for migration from ruby 1.8 to new versions.
This queries are not important for me and I've just deleted them with a sql query.
delete * from queries where filters like '%binary%';
Thank you very much for answering.