apache/pinot
GitHub で見る[Segment ingestion job] Multi-threaded segment copy from staging dir to final destination
Open
#7,510 opened on 2021年10月4日
enhancementhelp wantedingestion
Repository metrics
- Stars
- (4,937 stars)
- PR merge metrics
- (平均マージ 6d 7h) (30d で 186 merged PRs)
説明
SparkIngestionJob uses single threaded copy for segments copy from staging dir to final deep store dir(e.g. S3): https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
For a few hundred segments being uploaded, it can take forever. In this case, we should consider generating a list of source and destination paths for segment copy then use RDD to parallel this job.