rapidsai/cudf

[FEA] Update IO benchmarks for consistency between formats

Open

#12,739 opened on 2023年2月8日

GitHub で見る
 (0 comments) (0 reactions) (0 assignees)C++ (735 forks)batch import
2 - In ProgresscuIOfeature requestgood first issuelibcudf

Repository metrics

Stars
 (6,000 stars)
PR merge metrics
 (平均マージ 17d 21h) (30d で 230 merged PRs)

説明

Is your feature request related to a problem? Please describe.

  • #12678
  • #12675
  • add device buffer data source to CSV reader benchmark (code pointer)
  • migrate CSV writer to nvbench. Currently using gbench.
  • add JSON writer benchmark. This benchmark could be modeled after CSV writer.
  • add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after BM_csv_read_io
  • #12700
  • #12674 -> add compression to BM_csv_read_io (?)
  • add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
  • for reader benchmarks, verify that the roundtripped table matches the starting table
  • convert compression and io to string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation.
  • rename pmu, efs, trc in ORC writer chunks to peak_memory_usage, encoded_file_size, total_rows, to conform with the other ORC, PQ, CSV, text benchmarks

Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.

コントリビューターガイド