Defect #11089
closed
UTF-8 encoding not showing correctly when looking highlighted php file contents
Added by Troex Nevelin almost 12 years ago. Updated almost 11 years ago.
0%
Description
Ruby version 1.9.3 (x86_64-linux) RubyGems version 1.8.11 Rack version 1.4 Rails version 3.2.5 Database adapter mysql2 Database schema version 20120422150750 Git version 1.7.2.5
When I request file to see it contents (repository/revisions/HASH/entry) instead UTF-8 text I get '???'.
I'm using Git SCM and my files are valid UTF-8 (without BOM). I have this problem with Chineses, Russian, Thai and other scripts than latin.
However seeing diff's and attached utf-8 files are okay.
Files
| utf-8-not-shown-in-file-contents-view.png (50.4 KB) utf-8-not-shown-in-file-contents-view.png | Troex Nevelin, 2012-06-04 19:13 | ||
| diff-view-is-okay.png (56.6 KB) diff-view-is-okay.png | Troex Nevelin, 2012-06-04 19:13 | ||
| gh-new-d7e2a66d.png (53.8 KB) gh-new-d7e2a66d.png | Toshi MARUYAMA, 2012-06-05 15:13 | ||
| git-show-component.php.txt (2.61 KB) git-show-component.php.txt | Troex Nevelin, 2012-06-05 15:36 | ||
| git-show-component.php (2.61 KB) git-show-component.php | the same file but with different extension | Troex Nevelin, 2012-06-05 15:39 | |
| issue-attached-files.png (15.2 KB) issue-attached-files.png | Troex Nevelin, 2012-06-05 15:45 | ||
| test-file-with-php-ext.png (145 KB) test-file-with-php-ext.png | Troex Nevelin, 2012-06-05 15:45 | ||
| test-file-with-txt-ext.png (93.5 KB) test-file-with-txt-ext.png | Troex Nevelin, 2012-06-05 15:45 | ||
| def.php (31 Bytes) def.php | gehao liu, 2012-06-11 11:09 | ||
| def.py (38 Bytes) def.py | gehao liu, 2012-06-11 11:09 | ||
| def.txt (31 Bytes) def.txt | gehao liu, 2012-06-11 11:09 | ||
| def.php.png (3.93 KB) def.php.png | Toshi MARUYAMA, 2013-06-04 15:40 |
Related issues
Updated by Etienne Massip almost 12 years ago
Did you set any value in the Attachments and repositories encodings setting (in Administration/Settings General tab)?
If not, try to?
Updated by Troex Nevelin almost 12 years ago
Yes I have tried setting it to UTF-8 but it has no effect
Updated by Toshi MARUYAMA almost 12 years ago
- Subject changed from UTF-8 encoding not showing correctly when looking file contents to Git: encoding not showing correctly when looking file contents
Updated by Toshi MARUYAMA almost 12 years ago
Redmine uses "git show".
source:tags/2.0.1/lib/redmine/scm/adapters/git_adapter.rb#L372
Git 1.7.3.4, "git show --help" says
The contents of the blob objects are uninterpreted sequences of bytes. There is no encoding translation at the core level.
Updated by Troex Nevelin almost 12 years ago
I understand that git stores files in binary form, but calling from console:
git show --no-color HEAD:.../lang/ru/component.php
returns UTF-8 valid text, as I understand Redmine tries to guess encoding and sanitise content making sure no invalid characters pass to view.
For example source:trunk/config/locales/ja.yml this displays up correctly (but it uses SVN).
I think there is encoding guess problem in source:tags/2.0.1/lib/redmine/codeset_util.rb#L84 calling .to_utf8_by_setting_internal(str) sets ASCII-8BIT encoding on line 94?
Updated by Toshi MARUYAMA almost 12 years ago
- File gh-new-d7e2a66d.png gh-new-d7e2a66d.png added
I cannot reproduce.
https://github.com/redmine/redmine/commit/d7e2a66d
Could you attach this "git show" output file?
Updated by Troex Nevelin almost 12 years ago
git show --no-color HEAD:.../lang/ru/component.php > git-show-component.php.txt
I'm running Redmine on Debian 6, with ruby 1.9.3p125 (2012-02-16) [x86_64-linux] package compiled from debian ruby repository, using unicorn rack server.
I'm almost sure this is local related problem. Can you guide me how to debug this problem? I'm familier with ruby and ror. I have tried to output raw content in app/views/common/_file.html.erb but it gives me ActionView::Template::Error (incompatible character encodings: UTF-8 and ASCII-8BIT) error
Updated by Troex Nevelin almost 12 years ago
- File git-show-component.php git-show-component.php added
Updated by Troex Nevelin almost 12 years ago
- File issue-attached-files.png issue-attached-files.png added
- File test-file-with-php-ext.png test-file-with-php-ext.png added
- File test-file-with-txt-ext.png test-file-with-txt-ext.png added
I've made one more test on my setup, I've attached the same file to an issue but with different extensions .txt and .php and when trying to see attached file I get an issue with viewing syntax highlighted file. So this is not only Git related problem.
But no issue here in this ticket.
# grep coderay Gemfile.lock
coderay (1.0.6)
coderay (~> 1.0.6)
Updated by Toshi MARUYAMA almost 12 years ago
- Subject changed from Git: encoding not showing correctly when looking file contents to UTF-8 encoding not showing correctly when looking file contents
- Category deleted (
SCM)
Updated by gehao liu almost 12 years ago
- Status changed from New to Resolved
same problem!!!!!!!!!!
txt extname is OK,
python extname .py is OK.
php extname is wrong,
problem is viewing syntax highlighted!!!!!!!!!!
Updated by Toshi MARUYAMA almost 12 years ago
- Status changed from Resolved to New
Updated by gehao liu almost 12 years ago
this is coderay 1.0.6's bug,only php file.
Updated by András Kolesár about 11 years ago
coderay php encoding issue has been solved:
https://github.com/rubychan/coderay/issues/40
checked, works fine with updated coderay/scanners/php.rb file
Updated by Etienne Massip about 11 years ago
- Subject changed from UTF-8 encoding not showing correctly when looking file contents to UTF-8 encoding not showing correctly when looking highlighted file contents
- Category set to Text formatting
- Status changed from New to Confirmed
- Target version set to Candidate for next minor release
Upgrade dep to 1.0.9 or 1.1.
Updated by Toshi MARUYAMA almost 11 years ago
- Related to Defect #13692: warning: already initialized constant on Ruby 1.8.7 added
Updated by Toshi MARUYAMA almost 11 years ago
- Related to deleted (Defect #13692: warning: already initialized constant on Ruby 1.8.7)
Updated by Toshi MARUYAMA almost 11 years ago
- File def.php.png def.php.png added
- Status changed from Confirmed to Closed
Coderay version is defined "~> 1.0.6" in source:tags/2.3.1/Gemfile#L6
So, Coderay 1.0.9 is installed in 2.3.1.
This is note 12 php image.
Updated by Toshi MARUYAMA almost 11 years ago
- Subject changed from UTF-8 encoding not showing correctly when looking highlighted file contents to UTF-8 encoding not showing correctly when looking highlighted php file contents
Updated by Toshi MARUYAMA almost 11 years ago
- Target version deleted (
Candidate for next minor release)
Updated by Toshi MARUYAMA almost 11 years ago
- Has duplicate Defect #14445: <code> inside <pre> destroy polish diacritic of PHP added