apache/pinot

Refactor to centralize Job Specification properties for reusability across ingestion jobs

Open

#14,272 建立於 2024年10月22日

在 GitHub 查看
 (1 留言) (0 反應) (1 負責人)Java (1,234 fork)batch import
cleanupgood first issueingestion

倉庫指標

Star
 (4,937 star)
PR 合併指標
 (平均合併 6天 7小時) (30 天內合併 186 個 PR)

描述

Currently some of the the job specification properties are defined at each implementation of batch ingestion like

Hadoop job runner has defined the job spec constants in it's corresponding implementation:

public static final String SEGMENT_GENERATION_JOB_SPEC = "segmentGenerationJobSpec";

// Field names in job spec's executionFrameworkSpec/extraConfigs section private static final String DEPS_JAR_DIR_FIELD = "dependencyJarDir"; private static final String STAGING_DIR_FIELD = "stagingDir";

Similarly spark ingestion runner also defined the same constants in its own classes

private static final Logger LOGGER = LoggerFactory.getLogger(SparkSegmentGenerationJobRunner.class); private static final String DEPS_JAR_DIR = "dependencyJarDir"; private static final String STAGING_DIR = "stagingDir";

Better to centralize them at common place and reuse to avoid deviations in the future.

貢獻者指南