apache/pinot
[Segment ingestion job] Multi-threaded segment copy from staging dir to final destination
开放
#7,510 创建于 2021年10月4日
enhancementhelp wantedingestion
仓库指标
- 星标
- (4,937 个星标)
- PR 合并指标
- (PR 指标待抓取)
描述
SparkIngestionJob uses single threaded copy for segments copy from staging dir to final deep store dir(e.g. S3): https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
For a few hundred segments being uploaded, it can take forever. In this case, we should consider generating a list of source and destination paths for segment copy then use RDD to parallel this job.