apache/pinot
在 GitHub 查看[Segment ingestion job] Multi-threaded segment copy from staging dir to final destination
Open
#7,510 创建于 2021年10月4日
enhancementhelp wantedingestion
仓库指标
- Star
- (4,937 star)
- PR 合并指标
- (平均合并 6天 7小时) (30 天内合并 186 个 PR)
描述
SparkIngestionJob uses single threaded copy for segments copy from staging dir to final deep store dir(e.g. S3): https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
For a few hundred segments being uploaded, it can take forever. In this case, we should consider generating a list of source and destination paths for segment copy then use RDD to parallel this job.