apache/pinot
[Segment ingestion job] Multi-threaded segment copy from staging dir to final destination
オープン
#7,510 opened on 2021/10/04
enhancementhelp wantedingestion
Repository metrics
- Stars
- (4,937 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
SparkIngestionJob uses single threaded copy for segments copy from staging dir to final deep store dir(e.g. S3): https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
For a few hundred segments being uploaded, it can take forever. In this case, we should consider generating a list of source and destination paths for segment copy then use RDD to parallel this job.