rapidsai/cudf

[FEA] Update IO benchmarks for consistency between formats

已關閉

#12,739 建立於 2023年2月8日

 (0 則留言) (0 個反應) (0 位負責人)C++ (735 個分叉)batch import
2 - In ProgresscuIOfeature requestgood first issuelibcudf

倉庫指標

星標
 (6,000 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Is your feature request related to a problem? Please describe.

  • #12678
  • #12675
  • add device buffer data source to CSV reader benchmark (code pointer)
  • migrate CSV writer to nvbench. Currently using gbench.
  • add JSON writer benchmark. This benchmark could be modeled after CSV writer.
  • add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after BM_csv_read_io
  • #12700
  • #12674 -> add compression to BM_csv_read_io (?)
  • add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
  • for reader benchmarks, verify that the roundtripped table matches the starting table
  • convert compression and io to string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation.
  • rename pmu, efs, trc in ORC writer chunks to peak_memory_usage, encoded_file_size, total_rows, to conform with the other ORC, PQ, CSV, text benchmarks

Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.

貢獻者指南