[FEA] Explicitly guarantee row group ordering in the parquet reader. · rapidsai/cudf#15697

(0 comments) (0 reactions) (0 assignees)C++ (6,000 stars) (735 forks)batch import

cuIOfeature requestgood first issueimprovementlibcudf

Description

From @devavret , the question came up as to whether we guarantee the relative ordering of row groups across multiple input files in the parquet reader. That is, if you have two files [f1, f2] and the row groups within the files (in one column) are specified as [[r0,r3], [r0,r1]], do we guarantee the output ordering would be [f1r0, f1r3, f2r0, f2r1]

The code does in fact do this for both the explicitly specified case and the unspecified (empty user input / all row groups), but we don't make any guarantees about it. Seems like a safe and easy thing to add.

https://github.com/rapidsai/cudf/blob/5d244dfc13f4db0b1e41ded3029942fec50c98f6/cpp/src/io/parquet/reader_impl_helpers.cpp#L663

Contributor guide

Tech stack: cpp
Domain: data
Issue type: feature
Difficulty: 3
Estimated time: 1-3 hours
Activity status: fresh
Clarity: clear
Prerequisites: Basic understanding of parquet file formatFamiliarity with C++
Newbie friendliness: 75
Research direction: Examine the code at cpp/src/io/parquet/reader impl helpers.cpp line 663 where row group ordering is currently implicit. The goal is to explicitly guarantee the ordering of row groups across multiple input files. This likely involves adding a comment or documentation in the header or API, ensuring that the existing behavior is contractually guaranteed. Look for any existing tests for row group ordering and consider adding a test to enforce the guarantee.