rapidsai/cudf

[FEA] `cudf.read_text` does not use the GDS-enabled datasource

Open

#13,880 opened on Aug 14, 2023

View on GitHub
 (2 comments) (0 reactions) (1 assignee)C++ (735 forks)batch import
PerformancePythoncuIOfeature requestgood first issuelibcudf

Repository metrics

Stars
 (6,000 stars)
PR merge metrics
 (Avg merge 17d 21h) (230 merged PRs in 30d)

Description

libcudf's multibyte_split has multiple ways to specify input data source when reading from a file:

  1. auto const source = cudf::io::text::make_source_from_file(temp_file_name); // no GDS

  2. auto const datasource = cudf::io::datasource::create(temp_file_name); auto const source = cudf::io::text::make_source(*datasource); // supports GDS

cudf.read_text uses the first option to create the data_chunk_source object, so it does not benefit from potential direct reads with GDS.

cudf.read_text should create a cudf::io::datasource object to improve performance on GDS-enabled systems.

Contributor guide

Research direction
Modify the cudf.read text implementation to create a cudf::io::datasource from the file path and pass it to make source instead of using make source from file directly.
Tech stack
cpp
Domain
backendapiperformance
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++cudf::io::datasource
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60