apache/pinot

Inconsistent segment file extension for TarGzPush and MetadataPush

开放

#10,375 创建于 2023年3月3日

 (6 条评论) (0 个反应) (1 位负责人)Java (1,234 个派生)batch import
good first issue

仓库指标

星标
 (4,937 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Our SegmentGenerationJobRunner adds tar.gz extension to the segment file name.

  • TarGzPush: the controller side segment upload logic removes tar.gzextension when copying data to deep storage. The segment file will be stored as fileName.
  • MetadataPush: we use pinotFS to copy file from local to dst. Segment file will be stored as fileName.tar.gz.

It's going to be good if we can make the name convension the same. But, this will change the file name convension to the deepstorage. We should at least consider to add the config to remove the extension to have the clear migration path. (e.g. some people can start the cluster using TarGzPush and want to migrate to MetadataPush and keep the file names the same convention in the deep storage)

贡献者指南