apache/pinot

Refactor to centralize Job Specification properties for reusability across ingestion jobs

Open

#14.272 aberto em 22 de out. de 2024

Ver no GitHub
 (0 comments) (0 reactions) (0 assignees)Java (4.937 stars) (1.234 forks)batch import
cleanupgood first issueingestion

Description

Currently some of the the job specification properties are defined at each implementation of batch ingestion like

Hadoop job runner has defined the job spec constants in it's corresponding implementation:

public static final String SEGMENT_GENERATION_JOB_SPEC = "segmentGenerationJobSpec";

// Field names in job spec's executionFrameworkSpec/extraConfigs section private static final String DEPS_JAR_DIR_FIELD = "dependencyJarDir"; private static final String STAGING_DIR_FIELD = "stagingDir";

Similarly spark ingestion runner also defined the same constants in its own classes

private static final Logger LOGGER = LoggerFactory.getLogger(SparkSegmentGenerationJobRunner.class); private static final String DEPS_JAR_DIR = "dependencyJarDir"; private static final String STAGING_DIR = "stagingDir";

Better to centralize them at common place and reuse to avoid deviations in the future.

Guia do colaborador

Pilha de tecnologia
java
Domain
backend
Tipo Issue
refactor
DifficultyDificuldade de implementação estimada para um novo contribuidor, de 1 para alterações muito pequenas a 5 para trabalho de nível especializado.
2
Tempo estimadoUm intervalo de tempo aproximado para um colaborador experiente investigar, implementar, testar e preparar um pull request.
half day
Status da atividadeQuão disponível o issue aparece agora: novo, ativo, obsoleto, bloqueado ou aguardando entrada do mantenedor.
fresh
ClarityCom que clareza o issue explica a mudança esperada, os critérios de aceitação e a próxima etapa.
clear
Prerequisites
Java basicsPinot ingestion concepts
Simpatia para novatosUma pontuação de 1 a 100 que estima o quão acessível este issue é para colaboradores iniciantes.
70
Direção de pesquisa
Examine the files HadoopJobRunner.java and SparkSegmentGenerationJobRunner.java to identify duplicated constants like SEGMENT GENERATION JOB SPEC, DEPS JAR DIR FIELD, STAGING DIR FIELD. Propose a common class in a shared package (e.g., under pinot ingestion common) to define these constants. Check if there are other ingestion runners with similar duplication. Leave a comment on the issue to discuss the proposed location and get maintainer feedback before implementing.