apache/pinot

Add sanity check to the output of Realtime Provisioning Tool

開放

#8,339 建立於 2022年3月11日

 (8 則留言) (0 個反應) (0 位負責人)Java (1,234 個分叉)batch import
good first issue

倉庫指標

星標
 (4,937 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Currently RTP tool generates a mutable segments based on the given schema characteristics (or a given sample segment) and then linearly extrapolates the required memory, segment size, etc for different combinations of the input parameters. We've recently seen people have used 5GB segments because of the tool's suggestion. It's fine if the tool outputs 5GB because of the chosen input parameters, but it should give some warnings about potential problematic values. Here are some suggestions:

  • Change the output data structure to include an array of warnings
  • If a suggested segment size is > 500MB, add a warning message that the desired segment size should be < 500MB.
  • If number of segments is greater than a threshold (maybe 5K?), then the tools add a warning for that.
  • Sometimes tools outputs NA if the memory goes above the provided availableHostMemory parameter. Instead of NA, we should output the actual value, but add a warning that memory for some combinations goes above available host memory

貢獻者指南