rapidsai/cudf
Auf GitHub ansehen[FEA] Update IO benchmarks for consistency between formats
Open
#12.739 geöffnet am 8. Feb. 2023
2 - In ProgresscuIOfeature requestgood first issuelibcudf
Repository-Metriken
- Stars
- (6.000 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 17T 21h) (230 gemergte PRs in 30 T)
Beschreibung
Is your feature request related to a problem? Please describe.
- #12678
- #12675
- add device buffer data source to CSV reader benchmark (code pointer)
- migrate CSV writer to nvbench. Currently using gbench.
- add JSON writer benchmark. This benchmark could be modeled after CSV writer.
- add JSON reader benchmark with file data source (NESTED_JSON only does parsing and only on device buffers). This benchmark could be modeled after
BM_csv_read_io -
#12700 -
#12674 -> add compression toBM_csv_read_io(?) - add benchmark coverage for parquet chunked reader. Perhaps modeled after parquet_read_io_compression
- for reader benchmarks, verify that the roundtripped table matches the starting table
- convert
compressionandioto string axis type. see this discussion in nvbench. the goal is to choose other values from the CLI without having to run all values in automation. - rename
pmu,efs,trcin ORC writer chunks topeak_memory_usage,encoded_file_size,total_rows, to conform with the other ORC, PQ, CSV, text benchmarks
Additional context The initial set of topics came from a comparison of file read throughput across the supported formats in cuIO. We are also preparing for a comparison of memory footprint across cuIO, especially with Zstd compression/decompression.