apache/pinot

Unify some behavior in TableDataManager

Open

#9,752 opened on 2022年11月7日

GitHub で見る
 (2 comments) (2 reactions) (0 assignees)Java (1,234 forks)batch import
bugcleanupgood first issue

Repository metrics

Stars
 (4,937 stars)
PR merge metrics
 (平均マージ 6d 7h) (30d で 186 merged PRs)

説明

In the TableDataManager:

  • void addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig) is never used in production code (the only non-testing caller addOfflineSegment() is not wired up)
  • void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig) can be simplified because TableConfig is included in the IndexLoadingConfig
  • void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata) should be broken into 2 methods: addSegment() and refreshSegment()
  • Loading committed segment into real-time table should share the same logic as loading segment into offline table. Currently the logic is diverged, which causes moving tier not happening for real-time table

Ideally we want to keep the following methods:

  • addSegment
  • addConsumingSegment (real-time only)
  • refreshSegment
  • reloadSegment
  • removeSegment

コントリビューターガイド