nextflow-io/nextflow

splitCsv handles commas incorrectly when cells contain commas within quoted strings

Open

#4,874 opened on Apr 2, 2024

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Groovy (3,382 stars) (784 forks)batch import
good first issue

Description

Bug report

I have a csv with cells that contain paired-end fastq globs formatted for use as inputs to fromFilePairs(). This means the globs contain commas (i.e. "data{1,2}.fq.gz"). When using commas as the csv separator, the editor (libreoffice) wraps the fastq strings in quotation marks. When loading this file with splitCsv(), the commas enclosed within the quotes used by libreoffice are treated as cell delimiters rather than as part of the cell contents.

Expected behavior and actual behavior

Given a demo.csv file with the following plaintext contents:

path,ref
"file{1,2}.fq.gz",hg38

I expect that running the following program:

workflow { channel.fromPath("demo.csv") | splitCsv(header: true) | view }

Will produce the output:

[path:file{1,2}.fq.gz", ref:hg38]

It actually outputs:

[path:"file{1, ref:2}.fq.gz"]

Changing the file format to tsv and calling splitCsv with sep: '\t' solves the problem, but I'd like to be able to support csvs as well.

Environment

  • Nextflow version: 23.10.1 build 5891
  • Java version: openjdk 20.0.2-internal 2023-07-18
  • Operating system: Linux
  • Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

Contributor guide

Tech stack
None
Domain
databackend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
CSV quotingNextflow basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Look at the splitCsv implementation in the Nextflow codebase, likely in the groovy files under nextflow/csv/. The current parser does not handle commas inside quoted strings. Research existing CSV parsing libraries or fix the custom parsing to respect double quotes. Check comments in the issue (#4874) for any discussion or pointers. A fix would involve modifying the parsing logic to ignore commas within quoted fields.