rapidsai/cudf

[FEA] `cudf.read_text` does not use the GDS-enabled datasource

开放

#13,880 创建于 2023年8月14日

 (2 条评论) (0 个反应) (1 位负责人)C++ (735 个派生)batch import
PerformancePythoncuIOfeature requestgood first issuelibcudf

仓库指标

星标
 (6,000 个星标)
PR 合并指标
 (平均合并 17天 21小时) (30 天内合并 230 个 PR)

描述

libcudf's multibyte_split has multiple ways to specify input data source when reading from a file:

  1. auto const source = cudf::io::text::make_source_from_file(temp_file_name); // no GDS

  2. auto const datasource = cudf::io::datasource::create(temp_file_name); auto const source = cudf::io::text::make_source(*datasource); // supports GDS

cudf.read_text uses the first option to create the data_chunk_source object, so it does not benefit from potential direct reads with GDS.

cudf.read_text should create a cudf::io::datasource object to improve performance on GDS-enabled systems.

贡献者指南