rapidsai/cudf
GitHub で見る[FEA] Update IO benchmarks for consistency between formats
Open
#12,739 opened on 2023年2月8日
2 - In ProgresscuIOfeature requestgood first issuelibcudf
Repository metrics
- Stars
- (6,000 stars)
- PR merge metrics
- (平均マージ 17d 21h) (30d で 230 merged PRs)
説明
Is your feature request related to a problem? Please describe.
- #12678
- #12675
- add device buffer data source to CSV reader benchmark (code pointer)
- migrate CSV writer to nvbench. Currently using gbench.
- add JSON writer benchmark. This benchmark could be modeled after CSV writer.
- add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after
BM_csv_read_io -
#12700 -
#12674 -> add compression toBM_csv_read_io(?) - add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
- for reader benchmarks, verify that the roundtripped table matches the starting table
- convert
compressionandioto string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation. - rename
pmu,efs,trcin ORC writer chunks topeak_memory_usage,encoded_file_size,total_rows, to conform with the other ORC, PQ, CSV, text benchmarks
Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.