Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[FEATURE] Introduce flexible handling of emission failures

Abierto
  • ¿Algo no está bien?
#4,810 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 4 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
java, spark
Área
data-engineering

Línea de trabajo

Start by reading the Spark integration’s error handling in the facet builders, event builder, and OpenLineageSparkListener, then trace how the circuit breaker and execution context handle exceptions. The issue proposes a policy with SILENT, PARTIAL, and FAIL_FAST behavior plus a failure metric; done means failures follow those policies and emission failures can be counted.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:integration/spark kind:feature state:needs-triage
Motivation

Spark integration handles errors in event creation or emission silently, this is intentional so that failures in lineage collection do not affect the processing (and was introduced as a feature when the integration was much less stable).

This however can become an issue while fixing problems with the pipeline, as we may not have information about successful/failed runs.

Description

Add a configuration parameter openlineage.eventEmissionHandlingPolicy with options

  • FAIL_FAST - failure in building/emitting event causes app failure
  • PARTIAL - some failures like facet building failure could be ok but fail in building event in general or emitting it would cause app failure
  • SILENT - current behaviour, OL does what it can to not disturb application

Also we could add a counter similar to openlineage.spark.event.sql.start in openlineage metrics if we change the execution context to throw exception or return non-void type or propagate exception handled in circuit breaker run. This enables user to configure failure handling and exposes emission metrics to external systems.

It would require changing the way handle errors (wherever convenient) to some more deliberate one. Right now we have a lot of error handling in different places, propagating exceptions or SneakyThrows. So we would have to establish scopes of error handling.

  • immediate - for when exceptions are not even considered errors but are used as conditions e.g. NoSuchMethodException while checking something with reflection. Always handled
  • non-critical - propagated if FAIL_FAST policy is set, facet building errors etc. Event will miss some information but can be build.
  • critical - propagated if FAIL_FAST or PARTIAL policy is set. Event build or emission errors.

So on facet builder level we would have

catch(Exception e) {
  doSomething();
  if(FAIL_POLICY == FAIL_FAST){
    throw new FacetBuilingException("Error while building facet", e);
  }
}

Event builder would have

catch(Exception e) {
  doSomething();
  if(FAIL_POLICY != SILENT){
    throw new EventBuildingOrEmittingException("Error while building or emitting event", e);
  }
}

on the OpenLineageSparkListener level, less crude version of:

Boolean failOrTimeExceeded = circuitBreaker.run(
  () -> {
    try {
      getSparkApplicationExecutionContext().end(applicationEnd);
      return true;
    } catch (FacetBuilingException e) {
      if (FAIL_POLICY == FAIL_FAST) {
        throw e;
      } else {
        return false;
      }
    } catch (EventBuildingOrEmittingException e) {
      if (FAIL_POLICY != SILENT) {
        throw e;
      } else {
        return false;
      }
    }
  });
if (failOrTimeExceeded == null || !failOrTimeExceeded) {
  if (FAIL_POLICY != SILENT) {
    throw e;
  }
  meterRegistry.counter("openlineage.spark.event.fail").increment();
}
Related issues

No response

Do you plan to make this contribution yourself?
  • I am interested in doing this work
Lenguaje dominante
Java
Estrellas
2.7k
Forks
545
Merge medio
4 d 23 h
PR fusionados (30 d)
57

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de OpenLineage/OpenLineage

Todos los issues de OpenLineage/OpenLineage

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.