[CH] The CSV sep option is ignored by the native text scan

Open
#13,203 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
clickhouse, scala
Domain
backend, data

Research direction

Read FileSourceScanExecTransformer.getProperties and compare its CSV option mapping with Spark's CSVExprUtils.toDelimiterStr; inspect the relevant CSVSuite alias test and how ClickHouse handles native text reader delimiters. Verify sep and escaped tab handling, and ensure multi-character or non-ASCII delimiters fall back rather than being misread; the issue notes the ClickHouse backend could not be run.

Written by the indexing model from the issue text.

Description

Backend

ClickHouse (only ClickHouse offloads CSV scans; Velox and Bolt fall back).

Bug description

FileSourceScanExecTransformer.getProperties builds the native text reader options from the CSV options, and it only forwards the legacy delimiter key. sep, which is Spark's canonical key and what spark.read.option("sep", ...) and OPTIONS (sep '|') set, is ignored, so the native reader splits on its default comma:

// file content: "a|1\nb|2\n"
spark.read.schema("c1 string, c2 int").option("sep", "|").csv(path).show()
// vanilla: (a, 1), (b, 2); ClickHouse native: c1 = "a|1", c2 = null

The same table read with delimiter '|' is correct. Spark's own "test aliases sep and encoding for delimiter and charset" in CSVSuite is excluded for ClickHouse.

Two related gaps in the same mapping:

  • Spark decodes the delimiter with CSVExprUtils.toDelimiterStr, so sep '\t' written as a backslash and a t is a tab. The raw string is forwarded, and the native reader splits on its first byte, the backslash.
  • The native reader only uses the first byte of the delimiter, so a multi-character delimiter such as || or a non-ASCII one is read wrongly instead of falling back.

The ClickHouse path was traced by reading the code; I could not run the ClickHouse backend.

Gluten version

main (2681cc9a3)

Spark version

Spark 3.5

This issue was written with the assistance of AI (Claude Opus).

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.