apache/pinot

Add sanity check to the output of Realtime Provisioning Tool

Open

#8,339 opened on Mar 11, 2022

View on GitHub
 (8 comments) (0 reactions) (0 assignees)Java (1,234 forks)batch import
good first issue

Repository metrics

Stars
 (4,937 stars)
PR merge metrics
 (Avg merge 6d 7h) (186 merged PRs in 30d)

Description

Currently RTP tool generates a mutable segments based on the given schema characteristics (or a given sample segment) and then linearly extrapolates the required memory, segment size, etc for different combinations of the input parameters. We've recently seen people have used 5GB segments because of the tool's suggestion. It's fine if the tool outputs 5GB because of the chosen input parameters, but it should give some warnings about potential problematic values. Here are some suggestions:

  • Change the output data structure to include an array of warnings
  • If a suggested segment size is > 500MB, add a warning message that the desired segment size should be < 500MB.
  • If number of segments is greater than a threshold (maybe 5K?), then the tools add a warning for that.
  • Sometimes tools outputs NA if the memory goes above the provided availableHostMemory parameter. Instead of NA, we should output the actual value, but add a warning that memory for some combinations goes above available host memory

Contributor guide

Research direction
Study the Realtime Provisioning Tool's output generation code to understand how memory, segment size, and segment count are computed. Identify where warnings can be injected for segment size > 500MB, segment count > 5000, and memory exceeding availableHostMemory.
Tech stack
java
Domain
backenddatabaseinfrastructure
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaApache Pinot
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60