Redmine

I just noticed something weird with Mercurial.

When I try to remove the file mentionned above, mercurial did not success...
So the problem is maybe from Mercurial instead of Redmine.

I have the same issue with Bazaar.

I have the same issue too. My environment is a Redmine 0.8.4 in a Windows 2003 Server. My repo is Mercurial with some special character in file path, like 'ç', 'ã', 'õ'.

That's because Mercurial (and also Git) treats file names as byte string.
Here we need to convert them to UTF-8, but, there's no reliable info about file name encoding.

Wei Li wrote:

I have the same issue with Bazaar.

I'm not sure about Bazaar, but it must handle paths as UTF-8, so it seems strange.

I'm using redmine 0.9.3 on Windows Server 2003, has the same problem.

C:\redmine-0.9>ruby script/runner "Repository.fetch_changesets" -e production
c:/ruby/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/commands/runner.rb:48: c:/ruby/li
b/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/ab
stract_adapter.rb:219:in `log': Mysql::Error: Incorrect string value: '\xB2\xE2\
xCA\xD4\xB9\xDC...' for column 'path' at row 1: INSERT INTO `changes` (`changese
t_id`, `action`, `revision`, `branch`, `from_path`, `path`, `from_revision`) VAL
UES (
ActiveRecord::StatementInvalid)
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/connection_adapters/mysql_adapter.rb:323:in `execute'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/connection_adapters/abstract/database_statements.rb:259:in `insert_sql'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/connection_adapters/mysql_adapter.rb:333:in `insert_sql'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/connection_adapters/abstract/database_statements.rb:44:in `insert_without_query
_dirty'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/connection_adapters/abstract/query_cache.rb:18:in `insert'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/base.rb:2908:in `create_without_timestamps'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/timestamp.rb:53:in `create_without_callbacks'
from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record
/callbacks.rb:266:in `create'
... 30 levels...
from c:/ruby/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/commands/runner.rb:4
8
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `ge
m_original_require'
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `re
quire'
from script/runner:3

Yuya Nishihara wrote:

That's because Mercurial (and also Git) treats file names as byte string.
Here we need to convert them to UTF-8, but, there's no reliable info about file name encoding.

Wei Li wrote:

I have the same issue with Bazaar.

I'm not sure about Bazaar, but it must handle paths as UTF-8, so it seems strange.

I asked this Bazaar problem and #5578 at Mercurial-ja google group (in Japanese).
The reason is same with #5578.
Bazaar issue: want an option to set the output encoding, especially on win32 .
And I got a suggestion that XMLOutput plugin is better than "bzr log".

Toshi Maruyama wrote:

Git problem is reported at #5251.
I tried git and Bazaar and I could display multi-bytes characters path.
This patch is for git and Bazaar.

Git and Mercurial have absolutely the same problem, they treat filename as bytes, so the patch about Git seems reasonable.

But Bazaar's problem sounds different to me. It lies on the communication layer between Redmine and Bazaar. They should talk in UTF-8 but currently not.

To share my experence:
My system is Windows XP SP3, and Windows Server 2003.
My steps are:
1.Uninstall the redmine and reinstall it.
2.Creat hg repository in redmine folder.
3.Import the patch.
4.run "rake db:migrate RAILS_ENV=production" command
5.Restart the redmine service.

The path_encoding column was added successfully.

And I test the coding type in the list one by one, the "GBK" is correct for me.

Good luck for you!

By the way: if you have data in database, please backup it first and restore it after that the path_encoding column was added successfully.

Additionally, you need to delete repository setting created before patch applied and recreate the same repository from Redmine settings tab.

please fix this first in later version of redmine(like 1.1.2?) if #4455 Mercurial overhaul could not be done soon.
this problem stopped us from using hg for redmine completely.

I can confirm that the patches (see note 19) solve the problem for us.
Since the issue is blocking, we would like to know if
the is a method to backout the patches and undo the schema
migration when there will be an official release that addresses this issue.

Thanks

Paolo Losi wrote:

I can confirm that the patches (see note 19) solve the problem for us.
Since the issue is blocking, we would like to know if
the is a method to backout the patches and undo the schema
migration when there will be an official release that addresses this issue.

Answering myself:

rake db:migrate:down

Sorry for the noise

wow, these patches work great!!
it seems even better than before, at least now issues can be linked with r####
please make this to the next minor version 1.1.2. you have my vote. :)

there is a minor problem with the patches though. it doesn't work with codeview plugin. the error on the repository page:

NoMethodError in Code_review#update_revisions_view 
Showing vendor/plugins/redmine_code_review/app/views/code_review/_update_revisions.html.erb where line #6 raised: 

undefined method `review_count' for #<Changeset:0x63e7320>
Extracted source (around line #6): 

3: # and open the template in the editor.
4: %>
5: 
6: <script type="text/javascript">
7: <% @changesets.each do |changeset| %>
8:   <%
9:   if changeset.review_count > 0

Toshi MARUYAMA wrote:

These are patches for svn trunk r4799 and 1.1 stable r4800.

• 20110207-impl.diff is main patch.
• 20110207-db.diff is DB migration. If you have applied Yuya's issue-2664-0.9-stable-2010-04-11.patch, you don't need to apply this patch or you don't need to run "rake db:migrate". If you have not applied Yuya's patch, you need to apply this patch, and run "rake db:migrate".
• 20110207-git-cvs-fs.diff is for Git, CVS and Filesystem.

bo ye wrote:

please make this to the next minor version 1.1.2. you have my vote. :)

This feature has big behaviour change and has a db migrate.
So, I think it is difficult to apply 1.1 stable.
But, we need to consider to apply 1.2.

Yuya, what do you think?

Toshi MARUYAMA wrote:

bo ye wrote:

please make this to the next minor version 1.1.2. you have my vote. :)

This feature has big behaviour change and has a db migrate.
So, I think it is difficult to apply 1.1 stable.
But, we need to consider to apply 1.2.

Yuya, what do you think?

Same idea. For now, you can work around the issue by:

1. put lib/redmine/scm/adapters/path_encodable_wrapper.rb
2. apply the patch only for app/models/repository.rb
3. and replace the content of def new_scm method in place of db:migrate:
   

   scm = Redmine::Scm::Adapters::PathEncodableWrapper.new(scm, path_encoding) unless path_encoding.blank?
   

   
   by
   

   scm = Redmine::Scm::Adapters::PathEncodableWrapper.new(scm, 'encoding-name-of-your-repo')
   

Ruby 1.9 compatibility and tests are very serious.
Please see source:trunk/test/unit/lib/redmine/scm/adapters/git_adapter_test.rb@4810#L77 .

Japanese Shift_JIS and Traditional Chinese Big5 have 0x5c(backslash) problem and these are incompatible with ASCII.
Japanese EUC-JP is compatible with ASCII.

Ruby uses ANSI api to fork a process on Windows.

Subversion supports URL encoding for path and Redmine uses it.
I think Redmine Mercurial adapter need to wrap command line path of cat, diff and annotate such as Yuya's Mercurial overhaul helper extension.

"Files" module has similar strange behavior on my Japanese Windows Ruby 1.9.2.
I give up fix it.