keystonejs/keystone-classic

CSV download in /api/list/download.js if interrupted will crash node

Open

#4595 opened on Mar 8, 2018

View on GitHub
 (0 comments) (0 reactions) (0 assignees)JavaScript (14,656 stars) (2,288 forks)batch import
4.x candidatebughelp wanted

Description

Expected behavior

When trying to download > 100k records using the CSV download button. It should either download, warn that the size is too large or timeout gracefully.

Actual/Current behavior

A blank page is presented while the babyparse.unparse is running. If this is interrupted by the user the entire keystone site can not be accessed until it is restarted by pm2. Memory is maxed out. CPU is low. Nothing in the logs.

Steps to reproduce the actual/current behavior

Create a mongo collection with 200k+ records. Click download csv from the collections admin page.

Environment

Software Version
Keystone keystone@4.0.0-beta.5
Node v7.10.1

Linux ip-172-30-0-112 4.4.0-1022-aws #31-Ubuntu SMP Tue Jun 27 11:27:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

From Top:- 825 ubuntu 20 0 1983552 308980 2296 R 0.3 62.0 0:29.66 node keystone.js

Contributor guide

Tech stack
javascriptnodejsmongodb
Domain
backendapi
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Node.jsStreamsError handling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue is in /api/list/download.js. The current implementation uses babyparse.unparse which builds the entire CSV in memory, causing crashes on large datasets. A solution would involve streaming the CSV generation using Node.js streams or paginating the database query. Consider using a stream transform to pipe data directly to the response, and handle client disconnect (e.g., request.aborted). Look at the Keystone source code for existing streaming patterns.