apache/pinot

Improving server segment download code

开放

#9,152 创建于 2022年8月3日

 (14 条评论) (1 个反应) (0 位负责人)Java (1,234 个派生)batch import
good first issue

仓库指标

星标
 (4,937 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

  1. Currently if we use the default code path for server segment download we won't retry download on untar failure. In situations as downloading partial tar file, this would not be helpful. There are two solutions:

    • improve the default code path in BaseTableDataManager
    • use the new stream download untar code path pinot.server.instance.segment.stream.download.untar
  2. The new stream download untar code path currently doesn't support stream decryption, as should be theoretically possible with block cipher. This potential improvement to stream segment download can be explored.

User doc for stream download untar: https://docs.pinot.apache.org/operators/tutorials/performance-optimization-configurations PR: https://github.com/apache/pinot/pull/8753

贡献者指南