Proposed ArrowRecordBatchAdapter to replace ParquetReader with `read_from_memory_tables=True`

Open
  • Is something wrong?
#297 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

  • #519 by @arhamchopra — closed without merging

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
data-engineering, stream-processing

Research direction

Start by reading the existing ParquetReader path that handles read_from_memory_tables=True and determine how it consumes in-memory Arrow tables or record batches. Define the scope of a dedicated ArrowRecordBatchAdapter, including conversion of lists and structs and whether timestamp grouping is included. Done should provide a discoverable adapter with efficient, ideally zero-copy handling and behavior covered by appropriate tests.

Written by the indexing model from the issue text.

Description

adapter: parquet

Is your feature request related to a problem? Please describe.
The ability to stream a sequence of in-memory arrow tables into csp is very powerful, but currently a bit hidden within the ParquetReader implementation (by setting read_from_memory_tables=True), making it hard for users to find. It can also be a bit tricky to use properly as many of the arguments to the parquet reader are ignored in this mode.

Describe the solution you'd like
A dedicated pull adapter that consumes a serquence of arrow record batches (or tables) as efficiently as possible. i.e. ArrowRecordBatchAdapter.
An initial implementation could just delegate to the ParquetReader for implementation.

Also, the solution should ideally be zero-copy on the underlying arrow tables (I am not positive this is the case at the moment).

Describe alternatives you've considered
Continuing to use the parquet reader is confusing and non-intuitive, especially for cases where the arrow tables come from other sources.

Additional context
Other features to consider

  • Being able to convert arrow lists as numpy arrays (like the current parquet reader)
  • Being able to convert arrow structs as csp structs
  • An option such at that each timestamp, it ticks an arrow record batches containing all the records with that timestamp (in cases where we want to minimize any copy overhead and just operate on arrow types directly). This is essentially a "group-by" on the timestamp column.

Note that since there are a number of other tools that can efficiently and flexibly produce sequences of arrow tables from different sources (polars, duckdb, arrow-odbc, ray), having a generic ArrowRecordBatchAdapter will allow an even greater number of historical data connections with very little additional effort.

Dominant language
Python
Stars
450
Forks
91
Avg merge
18h 8m
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Point72/csp

All issues in Point72/csp

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.