rapidsai/cudf

[FEA] Expand ORC and Parquet benchmarks to cover different stripe/rowgroup sizes

Open

#10100 opened on Jan 21, 2022

View on GitHub
 (5 comments) (0 reactions) (0 assignees)C++ (6,000 stars) (735 forks)batch import
PerformancecuIOfeature requestgood first issuelibcudftests

Description

Add a set of benchmarks with varying stripe/rowgroup sizes to each affected component:

  • ORC reader
  • ORC writer
  • Parquet reader
  • Parquet writer

Use the new benchmarks to evaluate the effects of these options and potentially determine the optimal settings.

Contributor guide

Tech stack
cpppython
Domain
dataperformancetesting
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++PythonGPU programmingfile format knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Examine the existing benchmark infrastructure in the cuDF repository, likely under `benchmarks/`. Review the current ORC and Parquet reader/writer benchmarks to understand the pattern. Look through the issue comments for suggestions on stripe/rowgroup sizes. Propose a set of sizes (e.g., 1MB, 10MB, 100MB) to test and determine optimal settings. Consider using existing benchmark utilities to minimize new code.