apache/pinot
在 GitHub 查看[Segment ingestion job] Multi-threaded segment copy from staging dir to final destination
Open
#7,510 建立於 2021年10月4日
enhancementhelp wantedingestion
倉庫指標
- Star
- (4,937 star)
- PR 合併指標
- (平均合併 6天 7小時) (30 天內合併 186 個 PR)
描述
SparkIngestionJob uses single threaded copy for segments copy from staging dir to final deep store dir(e.g. S3): https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
For a few hundred segments being uploaded, it can take forever. In this case, we should consider generating a list of source and destination paths for segment copy then use RDD to parallel this job.