apache/pinot

Improving server segment download code

開放

#9,152 建立於 2022年8月3日

 (14 則留言) (1 個反應) (0 位負責人)Java (1,234 個分叉)batch import
good first issue

倉庫指標

星標
 (4,937 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

  1. Currently if we use the default code path for server segment download we won't retry download on untar failure. In situations as downloading partial tar file, this would not be helpful. There are two solutions:

    • improve the default code path in BaseTableDataManager
    • use the new stream download untar code path pinot.server.instance.segment.stream.download.untar
  2. The new stream download untar code path currently doesn't support stream decryption, as should be theoretically possible with block cipher. This potential improvement to stream segment download can be explored.

User doc for stream download untar: https://docs.pinot.apache.org/operators/tutorials/performance-optimization-configurations PR: https://github.com/apache/pinot/pull/8753

貢獻者指南