rapidsai/cudf

[FEA] Update IO benchmarks for consistency between formats

Open

#12.739 aperta il 8 feb 2023

Vedi su GitHub
 (0 commenti) (0 reazioni) (0 assegnatari)C++ (735 fork)batch import
2 - In ProgresscuIOfeature requestgood first issuelibcudf

Metriche repository

Star
 (6000 star)
Metriche merge PR
 (Merge medio 17g 21h) (230 PR mergiate in 30 g)

Descrizione

Is your feature request related to a problem? Please describe.

  • #12678
  • #12675
  • add device buffer data source to CSV reader benchmark (code pointer)
  • migrate CSV writer to nvbench. Currently using gbench.
  • add JSON writer benchmark. This benchmark could be modeled after CSV writer.
  • add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after BM_csv_read_io
  • #12700
  • #12674 -> add compression to BM_csv_read_io (?)
  • add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
  • for reader benchmarks, verify that the roundtripped table matches the starting table
  • convert compression and io to string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation.
  • rename pmu, efs, trc in ORC writer chunks to peak_memory_usage, encoded_file_size, total_rows, to conform with the other ORC, PQ, CSV, text benchmarks

Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.

Guida contributor