rapidsai/cudf

[FEA] Update IO benchmarks for consistency between formats

Open

#12,739 建立於 2023年2月8日

在 GitHub 查看
 (0 留言) (0 反應) (0 負責人)C++ (735 fork)batch import
2 - In ProgresscuIOfeature requestgood first issuelibcudf

倉庫指標

Star
 (6,000 star)
PR 合併指標
 (平均合併 17天 21小時) (30 天內合併 230 個 PR)

描述

Is your feature request related to a problem? Please describe.

  • #12678
  • #12675
  • add device buffer data source to CSV reader benchmark (code pointer)
  • migrate CSV writer to nvbench. Currently using gbench.
  • add JSON writer benchmark. This benchmark could be modeled after CSV writer.
  • add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after BM_csv_read_io
  • #12700
  • #12674 -> add compression to BM_csv_read_io (?)
  • add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
  • for reader benchmarks, verify that the roundtripped table matches the starting table
  • convert compression and io to string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation.
  • rename pmu, efs, trc in ORC writer chunks to peak_memory_usage, encoded_file_size, total_rows, to conform with the other ORC, PQ, CSV, text benchmarks

Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.

貢獻者指南