Project

General

Profile

Actions

Patch #31076

open

Issues CSV / PDF export via ActiveJob

Added by Jens Krämer almost 5 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Issues
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Why?
Exporting large amounts of issues and/or issues with a lot of custom fields as CSV or PDF can take a long time, to the point where users run into HTTP timeouts and cannot successfully do their exports.

How?
This patch which has been extracted from Planio allows for such exports to be run in a queueing backend like DelayedJob (that's what we use, but any supported ActiveJob backend should be fine). Exports with less than 1000 issues or exports that take less than 30 seconds to finish will be delivered to the user immediately as a download (just as it is without this patch), but after 30 seconds, the controller gives up waiting and returns the user to the previous issues listing. An email with a download link will be sent to the user once the export is finished.

Without a configured ActiveJob backend everything will be as before and all exports are processed in-request. The background export logic itself is generic and we use the same approach for running time entry exports in the background as well. I'll post that as a separate patch if there's interest in integrating this into Redmine at all.

Patches are against current master. Patch no 1 adds a lazy_issues method to IssueQuery which fetches issues in batches (to avoid high memory usage). Patch no 2 implements the background execution.


Files


Related issues

Related to Redmine - Feature #24477: CSV export feature for thousand of issuesNew

Actions
Related to Redmine - Defect #24473: CSV Export Very slow for issues with lots of Custom FieldsNew

Actions
Related to Redmine - Feature #36691: Background job and dedicated status for project deletionClosedMarius BĂLTEANU

Actions
Actions #1

Updated by Jens Krämer almost 5 years ago

related to #24477, #24473

Actions #2

Updated by Go MAEDA almost 5 years ago

  • Target version set to Candidate for next major release
Actions #3

Updated by Go MAEDA almost 5 years ago

  • Related to Feature #24477: CSV export feature for thousand of issues added
Actions #4

Updated by Go MAEDA almost 5 years ago

  • Related to Defect #24473: CSV Export Very slow for issues with lots of Custom Fields added
Actions #5

Updated by Go MAEDA almost 5 years ago

  • Target version changed from Candidate for next major release to 4.1.0
Actions #6

Updated by Stephane Evr almost 5 years ago

Awesome!

Actions #7

Updated by Koichi Tanimoto over 4 years ago

Please let me know the following points:

1. Why do you skip filesize validation?
2. How do I remove attachment file after download finished. Can I remove it on browser?

Actions #8

Updated by Jens Krämer over 4 years ago

Good points, thanks for bringing them up.

1. Why do you skip filesize validation?

We do this to avoid running into a filesize validation error in the background worker. We want these background exports to succeed even if the file violates size constraints. We feel this is OK since the resulting files are only temporary and will not stay on the system for long time, anyway (see below).

2. How do I remove attachment file after download finished. Can I remove it on browser?

The resulting attachments are orphaned, without any container object. As such, they are fairly easy to detect and clean up i.e. by a cron job running rake redmine:attachments:prune daily.

Actions #9

Updated by Koichi Tanimoto over 4 years ago

I understood above. Thanks your explaination!

Actions #10

Updated by Go MAEDA over 4 years ago

  • Target version changed from 4.1.0 to 4.2.0
Actions #11

Updated by Marius BĂLTEANU almost 3 years ago

  • Target version changed from 4.2.0 to 5.0.0
Actions #12

Updated by Marius BĂLTEANU about 2 years ago

  • Related to Feature #36691: Background job and dedicated status for project deletion added
Actions #13

Updated by Marius BĂLTEANU almost 2 years ago

  • Target version changed from 5.0.0 to Candidate for next major release
Actions

Also available in: Atom PDF