rapidsai/cudf

[FEA] Expand ORC and Parquet benchmarks to cover different stripe/rowgroup sizes

Open

#10,100 创建于 2022年1月21日

在 GitHub 查看
 (5 评论) (0 反应) (0 负责人)C++ (6,000 star) (735 fork)batch import
PerformancecuIOfeature requestgood first issuelibcudftests

描述

Add a set of benchmarks with varying stripe/rowgroup sizes to each affected component:

  • ORC reader
  • ORC writer
  • Parquet reader
  • Parquet writer

Use the new benchmarks to evaluate the effects of these options and potentially determine the optimal settings.

贡献者指南

技术栈
cpppython
领域
dataperformancetesting
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
3-5 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
clear
前置要求
C++PythonGPU programmingfile format knowledge
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
25
研究方向
Examine the existing benchmark infrastructure in the cuDF repository, likely under `benchmarks/`. Review the current ORC and Parquet reader/writer benchmarks to understand the pattern. Look through the issue comments for suggestions on stripe/rowgroup sizes. Propose a set of sizes (e.g., 1MB, 10MB, 100MB) to test and determine optimal settings. Consider using existing benchmark utilities to minimize new code.