apache/pinot

Support dynamic updates to existing derived columns in table

Open

#8,700 opened on May 14, 2022

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Java (1,234 forks)batch import
featurehelp wanted

Repository metrics

Stars
 (4,937 stars)
PR merge metrics
 (Avg merge 6d 7h) (186 merged PRs in 30d)

Description

Currently Pinot doesn't support runtime changes to existing derived columns (for e.g change to ingestion transform expression) in table. Only way to make such updates is to either delete/recreate the table and ingest from scratch or add a new derived column with the changes, neither of which are ideal from usability perspective.

Can we consider improving this behavior so users can easily make adjustments to the data via table config/schema change and subsequently doing a segment reload ? cc @Jackie-Jiang @mayankshriv

Contributor guide

Research direction
Understand how Pinot applies ingestion transform expressions during segment generation and explore how segment reload could trigger recomputation of derived columns without full re ingestion. Look at the table config, schema, and segment metadata APIs.
Tech stack
java
Domain
databasedistributed systemsbackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Apache PinotJavaOLAP concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40