Project

General

Profile

Actions

Feature #7037

closed

CSV export encoding and excel. UTF-8 and BOM

Added by John Yani about 13 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Category:
Issues
Target version:
Start date:
2010-12-03
Due date:
% Done:

0%

Estimated time:
Resolution:
Fixed

Description

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

Maybe there is some setting to enforce BOM while doing csv export?

References:

Files

7037-csv-bom.diff (4.53 KB) 7037-csv-bom.diff Toshi MARUYAMA, 2015-04-02 10:37
7037-csv-bom-1.diff (4.53 KB) 7037-csv-bom-1.diff Toshi MARUYAMA, 2015-04-02 10:42
7037-csv-bom-2.diff (4.54 KB) 7037-csv-bom-2.diff Toshi MARUYAMA, 2015-04-02 11:02
excel2007.png (38.4 KB) excel2007.png Jean-Philippe Lang, 2015-05-10 11:29
libreoffice.png (46.7 KB) libreoffice.png Toshi MARUYAMA, 2015-05-10 12:48
Office2013Pro.png (41.6 KB) Office2013Pro.png Toshi MARUYAMA, 2015-05-10 13:01

Related issues

Related to Redmine - Defect #7461: Special chars / umlauts broken in CSV / PDF exportClosed

Actions
Related to Redmine - Defect #7039: CSV export. Line breaks. ExcelClosed2010-12-03

Actions
Related to Redmine - Defect #19523: problem in exporting Arabic text to csvClosed

Actions
Related to Redmine - Defect #11116: CSV export encoding problem when issues and interface languages are not equalClosed

Actions
Related to Redmine - Patch #17902: CSV encoding should be UTF-8 in French localeClosedJean-Philippe Lang

Actions
Has duplicate Redmine - Feature #11301: Add "byte order mark" to the export CSV fileClosed

Actions
Actions #1

Updated by Terence Mill about 13 years ago

Is that fixed?

Actions #2

Updated by John Yani about 13 years ago

Well, the fix exists. But AFAIK it is not merged yet.
The correct way would be to add some setting to enable/disable BOM instead of just hard-coding it.

Actions #3

Updated by Alexander Kriventsov over 12 years ago

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

Actions #4

Updated by John Yani over 12 years ago

Alexander Kriventsov wrote:

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

I've edited the "general_csv_encoding" value in corresponding "/config/locales/*.yml" file.

Actions #5

Updated by Etienne Massip over 12 years ago

  • Category set to Issues
Actions #6

Updated by Nick Tan almost 11 years ago

hi,

i have the exactly same issue, with the plugin https://github.com/sprosin/redmine_csv_with_bom now the csv export works as expected.

but i think this patch should be merge into the redmine core, not in form of a plugin with monkey patching style

Actions #7

Updated by Yuri Rumega about 9 years ago

The problem is still actual in 2.6. Had to adapt "redmine_csv_with_bom" plugin.

Actions #8

Updated by Toshi MARUYAMA almost 9 years ago

  • Tracker changed from Defect to Feature
Actions #9

Updated by Toshi MARUYAMA almost 9 years ago

  • Related to Defect #19523: problem in exporting Arabic text to csv added
Actions #10

Updated by Toshi MARUYAMA almost 9 years ago

This is patch for trunk r14155.

Actions #11

Updated by Toshi MARUYAMA almost 9 years ago

Removed extra white space from previous patch.

Actions #12

Updated by Toshi MARUYAMA almost 9 years ago

Previous patch has mistake.
This patch fixed.

-+    encoding = Setting.csv_utf8_bom ? "UTF-8" : l(:general_csv_encoding)
++    encoding = Setting.csv_utf8_bom? ? "UTF-8" : l(:general_csv_encoding)
Actions #13

Updated by Toshi MARUYAMA almost 9 years ago

  • Related to Defect #11116: CSV export encoding problem when issues and interface languages are not equal added
Actions #14

Updated by Jean-Philippe Lang almost 9 years ago

Doesn't work for me when opening the CSV with Excel 2007 (which is supposed to support UTF-8 with BOM), tested with 7037-csv-bom-2.diff applied :

Actions #15

Updated by Toshi MARUYAMA almost 9 years ago

Could you check "csv_utf8_bom" is on in setting?

Actions #16

Updated by Jean-Philippe Lang almost 9 years ago

Yes, it was checked. The csv file is recognized as UTF-8 with BOM by my text editor but not by Excel 2007.

Actions #17

Updated by Toshi MARUYAMA almost 9 years ago

$ hexdump -C issues.csv 
00000000  ef bb bf 23 3b 54 72 61  63 6b 65 72 3b 53 74 61  |...#;Tracker;Sta|
00000010  74 75 74 3b 50 72 69 6f  72 69 74 c3 a9 3b 53 75  |tut;Priorit..;Su|
00000020  6a 65 74 3b 41 73 73 69  67 6e c3 a9 20 c3 a0 3b  |jet;Assign.. ..;|
00000030  4d 69 73 2d c3 a0 2d 6a  6f 75 72 0a 38 37 3b 42  |Mis-..-jour.87;B|
00000040  75 67 3b 4e 65 77 3b 4e  6f 72 6d 61 6c 3b 4e 6f  |ug;New;Normal;No|
00000050  6e 2d 41 53 43 49 49 20  c3 a7 c3 a9 c3 a0 c3 b9  |n-ASCII ........|
00000060  3b 22 22 3b 31 30 2f 30  35 2f 32 30 31 35 20 31  |;"";10/05/2015 1|
00000070  39 3a 32 35 0a                                    |9:25.|
00000075

$ hexdump -c issues.csv 
0000000 357 273 277   #   ;   T   r   a   c   k   e   r   ;   S   t   a
0000010   t   u   t   ;   P   r   i   o   r   i   t 303 251   ;   S   u
0000020   j   e   t   ;   A   s   s   i   g   n 303 251     303 240   ;
0000030   M   i   s   - 303 240   -   j   o   u   r  \n   8   7   ;   B
0000040   u   g   ;   N   e   w   ;   N   o   r   m   a   l   ;   N   o
0000050   n   -   A   S   C   I   I     303 247 303 251 303 240 303 271
0000060   ;   "   "   ;   1   0   /   0   5   /   2   0   1   5       1
0000070   9   :   2   5  \n                                            
0000075
$ libreoffice --version
LibreOffice 4.0.4.2

$ libreoffice issues.csv 

Actions #18

Updated by Toshi MARUYAMA almost 9 years ago

I can open on my Japanese MS Office 2013 Professional with general_csv_separator change.

diff --git a/config/locales/fr.yml b/config/locales/fr.yml
--- a/config/locales/fr.yml
+++ b/config/locales/fr.yml
@@ -158,7 +158,8 @@ fr:
   general_text_no: 'non'
   general_text_yes: 'oui'
   general_lang_name: 'French (Fran├žais)'
-  general_csv_separator: ';'
+  # general_csv_separator: ';'
+  general_csv_separator: ','
   general_csv_decimal_separator: ','
   general_csv_encoding: ISO-8859-1
   general_pdf_fontname: freesans

Actions #19

Updated by Toshi MARUYAMA almost 9 years ago

#7039#note-15:

German wikipedia

The delimiter semicolon is expected when file is in ANSI.
The delimiter TAB is expected when file is in Unicode.

I don't know about French.

Actions #20

Updated by Jean-Philippe Lang almost 9 years ago

Is there any situation where we want to use UTF-8 without BOM?
Because I'm not sure that a new setting is required, we could add the BOM when CSV is encoded in UTF-8 just like the redmine_csv_with_bom plugin does.

Actions #21

Updated by Jean-Philippe Lang over 8 years ago

  • Status changed from New to Closed
  • Assignee set to Jean-Philippe Lang
  • Resolution set to Fixed
Actions #22

Updated by Go MAEDA over 8 years ago

  • Has duplicate Feature #11301: Add "byte order mark" to the export CSV file added
Actions #23

Updated by Toshi MARUYAMA about 8 years ago

  • Related to Patch #17902: CSV encoding should be UTF-8 in French locale added
Actions

Also available in: Atom PDF