[Enh]: Document and test chDB interoperability through Arrow

Open
  • Is something wrong?
#3,932 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
65/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
documentation, testing

Research direction

Start by reading the Arrow interoperability documentation and existing tests for nw.from_arrow; the issue does not name specific files. Add coverage for the listed value types, empty results, and row order, then include a chDB example explaining that conversion materializes results and later operations are not pushed down. Check performance on representative result sizes; done means the tests, documentation, and performance check cover the stated scope.

Written by the indexing model from the issue text.

Description

Context

chDB DataStore exposes the Arrow C Stream interface. Its results can already be passed to Narwhals with:

nw.from_arrow(datastore, backend="pyarrow")

This allows Narwhals-based libraries to consume chDB results without converting through pandas.

Scope of this issue

This proposal covers Arrow interoperability only.

  • Add an interoperability test using nw.from_arrow
  • Cover nullable values, unsigned integers, strings, nested types, empty results, and row order
  • Add a short chDB example to the Arrow interoperability documentation
  • Clearly document that the conversion executes and materializes the chDB result
  • Check conversion performance on representative result sizes

After conversion, subsequent Narwhals operations run on the selected Arrow backend. They are not pushed down to chDB.

Possible follow-up: native chDB backend

If there is community interest, we can explore a native Narwhals backend for chDB as a separate follow-up.

A native backend would keep Narwhals expressions lazy and translate operations such as filtering, projection, sorting, and aggregation into chDB queries. This would:

  • avoid materializing unnecessary intermediate results
  • reduce data transfer by reading only the required rows and columns
  • keep query planning and execution inside chDB
  • make ClickHouse execution available to libraries built on Narwhals
  • improve performance for larger-than-memory and multi-step analytical workloads

We would appreciate feedback on the initial Arrow integration and whether native chDB execution would be useful to the Narwhals community.

Dominant language
Python
Stars
1.7k
Forks
242
Avg merge
1d 5h
Merged PRs (30d)
49

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from narwhals-dev/narwhals

All issues in narwhals-dev/narwhals

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.