Snowflake CSV - use parallel scan settings for uncompressed CSV

Open
  • Is something wrong?
#408 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
shell
Domain
databases

Research direction

Locate the Snowflake data loading test and compare its current input handling with duckdb/benchmark.sh at line 18. Update the test to load an uncompressed CSV using MULTI_LINE=FALSE and COMPRESSION=NONE, then verify that the benchmark still runs with those settings.

Written by the indexing model from the issue text.

Description

I noticed that for duckdb the CSV file is first uncompressed before loading. https://github.com/ClickHouse/ClickBench/blob/main/duckdb/benchmark.sh#L18C1-L18C5

For Snowflake, we also support faster parallel scanning of uncompressed CSVs.

Can we modify the Snowflake data loading test such that it loads a uncompressed CSV with MULTI_LINE=FALSE and COMPRESSION=NONE?

https://medium.com/snowflake/recap-of-snowflake-ingestion-cost-and-performance-improvements-large-csv-demo-911e6588d626?source=friends_link&sk=38a754b71aa06f51f269c6974b24abd8

Dominant language
Shell
Stars
1.1k
Forks
328
Avg merge
4h 12m
Merged PRs (30d)
605

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ClickHouse/ClickBench

All issues in ClickHouse/ClickBench

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.