Project

General

Profile

Actions

Feature #2371

closed

character encoding for attachment file

Feature #2371: character encoding for attachment file

Added by youngseok yi almost 18 years ago. Updated almost 15 years ago.

Status:
Closed
Priority:
Low
Assignee:
Toshi MARUYAMA
Category:
Attachments
Target version:
Resolution:
Fixed

Description

As r814, default encoding for repository can be configured.

diff or patch attachment requires similar configuration.

  • default encoding for diff or patch attachment (Admin -> Settings -> Attachment -> diff/patch encodings ?).
  • follow encoding of repository. (source:/trunk/app/helpers/repositories_helper.rb@1900#L109)

I thinks 2nd option may be enough and useful.


Files

attachment-encoding.patch (1017 Bytes) attachment-encoding.patch Yuya Nishihara, 2010-06-27 12:14
general-settings.png (28.2 KB) general-settings.png Toshi MARUYAMA, 2011-11-17 11:17

Related issues 4 (0 open — 4 closed)

Related to Redmine - Defect #9143: Partial diff comparison should be done on actual code, not on htmlClosedJean-Philippe LangActions
Related to Redmine - Defect #4608: Mail attachment name encoding is incorectly handledClosedActions
Has duplicate Redmine - Feature #4577: convert text file attached an issue to utf-8.ClosedActions
Has duplicate Redmine - Defect #3652: Unicode Support for TXT-FilesClosedActions

Updated by Yuya Nishihara over 16 years ago Actions #1

youngseok yi wrote:

  • follow encoding of repository.

Attached patch implements it with minimal changes. attachment-encoding.patch

Proper solution will be something like:
  1. move to_utf8 to separate module, e.g. RepoFilesHelper
  2. make AttachmentsHelper and RepositoriesHelper include RepoFilesHelper

Updated by Toshi MARUYAMA over 15 years ago Actions #2

  • Assignee set to Toshi MARUYAMA

Updated by Toshi MARUYAMA over 15 years ago Actions #3

  • Target version set to Candidate for next major release

Updated by Toshi MARUYAMA over 15 years ago Actions #4

  • Target version changed from Candidate for next major release to 1.3.0

Updated by Toshi MARUYAMA about 15 years ago Actions #5

  • Subject changed from encoding for diff or patch attachment file to encoding for attachment file

Updated by Toshi MARUYAMA about 15 years ago Actions #6

  • Subject changed from encoding for attachment file to character encoding for attachment file

Updated by Etienne Massip about 15 years ago Actions #7

Toshi, won't your last commit prevent me from attaching an iso8859-1 encoded patch to this issue and seeing it fine?

Updated by Toshi MARUYAMA about 15 years ago Actions #8

Etienne Massip wrote:

Toshi, won't your last commit prevent me from attaching an iso8859-1 encoded patch to this issue and seeing it fine?

This feature issue goal is that attachment file and patch encoding are converted by repositories setting.

Updated by Etienne Massip about 15 years ago Actions #9

I'm not sure this is a good idea; repositories may return data using a specific encoding, but attachments are usually stored on FS without transformation, so assuming that they're "very likely to be encoded the same way data in SCM is" is not necessarily true.

For example, my encoding list starts with UTF-8 and my locale (Fr) would assume that files uploaded by users are probably encoded in ISO-8859-15/CP1252; so assuming that the text files uploaded are in UTF-8 mean that they will be rendered stripped and that I will probably often loose some chars, which is the actual situation.

I would prefer to be able to specify a distinct default encoding for text attachments which would be ISO-8859-15/CP1252 (could be defaulted to default server encoding) and render with something like bom_present?(str) ? str : Iconv.conv('UTF-8', Setting.default_encoding).

Updated by Toshi MARUYAMA about 15 years ago Actions #10

UTF-8 is very strict.
It is very rare case that miss understanding ISO-8859-1 characters as UTF-8.
http://groups.google.com/group/thg-dev/browse_thread/thread/6c258628e3fce8/09e9dbe4a030e51d

Updated by Toshi MARUYAMA about 15 years ago Actions #11

Redmine 1.2.2 repository converting encoding is this line.
source:tags/1.2.2/app/helpers/repositories_helper.rb#L140

In case of "UTF-8,ISO-8859-1",
if converting error in "UTF-8", Redmine converts from ISO-8859-1.

Japanese use three encoding, UTF-8, EUC-JP and Shift-JIS (CP932).
This Redmine feature is big advantage in Japan.

Updated by Etienne Massip about 15 years ago Actions #12

So if I understand well, according to encoding list order, it will try and fail to convert the ISO-8859-1 file from UTF-8 to UTF-8 and then will try and success to convert it from ISO-8859-1 to UTF-8?

Guess it will work...

Updated by Etienne Massip about 15 years ago Actions #13

What if the administrator does not set UTF-8 at the start of the list?
Can't you str.is_utf8? ? str : try Iconv.conv('UTF-8', Setting.encodings)?

Updated by Toshi MARUYAMA about 15 years ago Actions #14

Etienne Massip wrote:

repositories may return data using a specific encoding,

It is not true.
SCMs does not have encoding information (meta data) of file contents.
http://mercurial.selenic.com/wiki/EncodingStrategy?action=recall&rev=21#Unknown_byte_strings

Updated by Etienne Massip about 15 years ago Actions #15

Toshi MARUYAMA wrote:

It is not true.
SCMs does not have encoding information (meta data) of file contents.

Well, that's why I said may :-)

Updated by Toshi MARUYAMA about 15 years ago Actions #16

Etienne Massip wrote:

What if the administrator does not set UTF-8 at the start of the list?

This is very rare case in Japan.
It is popular "UTF-8,EUC-JP,Shift_JIS in Japan.
This order is strict order.

If Single Byte Character Set (e.g. ISO-8859-1) is the start of the list, all characters are converted to UTF-8.
But, I think this is very rare case in the whole world.

Can't you str.is_utf8? ? str : try Iconv.conv('UTF-8', Setting.encodings)?

Default repository encoding setting is empty.
This is equivalent that default is UTF-8.
And I think it is better that administrator set UTF-8 in the start of the list explicitly.

Updated by Toshi MARUYAMA almost 15 years ago Actions #17

  • % Done changed from 0 to 100

Updated by Anton Statutov almost 15 years ago Actions #18

Is this feature fixes #4608?

Updated by Mischa The Evil almost 15 years ago Actions #19

Anton Statutov wrote:

Is this feature fixes #4608?

I don't think so.

Updated by Toshi MARUYAMA almost 15 years ago Actions #20

  • Status changed from New to Closed
  • Resolution set to Fixed

Committed in r7885.

Actions

Also available in: PDF Atom