[PROPOSAL] Introduce explicit lineage to support complex ETL-like jobs

Start by reading issue #3458 and the OpenLineage specification sections for inputs, outputs, and Column Level Lineage, which this proposal discusses. The issue asks for feedback on a new explicit lineage model and its alternatives; there are no implementation files or tests named, and completion would require agreeing on the design before implementation.

Following up on the discussion in issue https://github.com/OpenLineage/OpenLineage/issues/3458, because we are facing the same questions and challenges while adopting OpenLineage in our tools, and given the expressed interest from the community in understanding where the "parent run" approach might fall short for certain workloads, we have put together this proposal for discussion.

Summary

Currently OpenLineage relies on implicit dataset level lineage, i.e. if a Run has inputs=[A] and outputs=[B], an edge A -> B is assumed. However, this way the number of links will be the cartesian product of inputs by outputs, with all inputs influencing all outputs in the same Run.

While this works well for lazy-evaluation engines like Apache Spark (where an "Action" tightly couples specific inputs to just one specific output), it creates significant semantic distortion for eager-execution engines like traditional ETL tools (e.g. Pentaho).

The specification also supports explicit column level lineage via the dataset's Column Level Lineage facet, but this solution is insufficient for many use cases. The facet solves the problem at the "microscopic" level (column-to-column) but leaves a gap at the "macroscopic" level (dataset-to-dataset). Integrators often know that "Table A derived Table B" without having the granularity to map specific columns. Currently, if an integrator cannot provide column level details, they end up back to the mentioned cartesian product inference.

We propose making lineage a first-class citizen by introducing a mechanism to define explicit complete lineage. This would separate the audit of what was accessed (inputs, outputs) from the definition of what and how was used (lineage), and allow integrators to report accurate lineage for complex control flows without artificially breaking jobs into fictitious sub-runs.

The Problem

The spec doesn't distinguish between IO (Input/Output) and the flow of data.

• IO event: "File A was read by Job X."
  • This is a fact useful for audit trails and security (who touched the data?).
• Flow of data event: "Column B was calculated by Job X using Column A."
  • This is a fact useful for impact analysis and debugging.

By inferring the data flow from IO events, the system is somewhat "guessing" the lineage. The standard should allow producers to be more precise.

In Spark, the unit of work is defined by an Action (.write(), .collect(), .save()). Transformations accumulate in a logical plan and only materialize (lazy evaluation) when the action is invoked. As such, a Run maps to a specific optimized plan writing to a single destination.

In ETL tools, a Run often represents a flow / collection of steps that performs multiple, independent data movements sequentially or in parallel.

If a single Job reads datasets A and B and writes to C and D, the current spec forces us to list all of them in the inputs and outputs lists. Consumers then infer dependencies as (A,B) -> (C,D).

Real Flow:

• A -> C
• B -> D

Inferred OpenLineage Flow (Implicit):

• A -> C (True)
• A -> D (False Positive)
• B -> C (False Positive)
• B -> D (True)

As job complexity increases, the "noise" in the lineage graph scales. For a bulk ETL job processing 10 independent tables, the lineage graph becomes 90% false positives.

Why the "parent run" approach does not work

It has been suggested in issue #3458 that integrators should break complex jobs into "child runs" (one per output) to fit the Spark-like model. But that approach can be fundamentally flawed for some types of workflows.

In ETL tools, the unit of execution is the "Job", which may contain dozens of write operations. Breaking a single logical job into artificial sub-runs creates a granularity mismatch that disconnects metadata from actual orchestrator logs, and the increased volume of events can lead to unnecessary traffic, storage and processing costs.

Furthermore, in some contexts writes to multiple tables can occur within a single ACID transaction. Separate runs misrepresent transaction boundaries as independent events, misleading users about the atomicity of the operation.

Finally, this pattern introduces some observability noise, skewing operational metrics by reporting, for example, a single daily job as a handful of fragmented runs rather than a single unit of work.

The Proposal

To shift from inferring relationships from inputs and outputs to explicitly defining the lineage graph, we propose introducing lineage as a root-level property of the OpenLineage event, alongside run, job, inputs, and outputs.

With it we gain:

• Centralization: All lineage logic for the unit of work is contained in one list, not inferred from inputs/outputs or scattered across multiple Column Level Lineage facets.
• Availability: If added to all event types (Run, Job and Dataset), it serves both runtime and static/design-time lineage.
• Decoupling: It separates Access (Security/Audit) from Usage (Lineage).
  • Inputs property: "What did this job touch?" (e.g., Config tables, lookup tables).
  • Lineage property: "What data actually flowed into the output?".

{
  "eventType": "RUNNING",
  "run": { ... },
  "job": { ... },
  "inputs": [ ... ],    // Audit: Everything read (Access)
  "outputs": [ ... ],   // Audit: Everything written (Access)
  "lineage": [          // Definition: The data lineage (Usage)
    {
      "namespace": "ns",
      "name": "Output_C",
      "inputs": [
        { "namespace": "ns", "name": "Input_A", "transformations": [...] }
      ]
    },
    {
      "namespace": "ns",
      "name": "Output_D",
      "inputs": [
        { "namespace": "ns", "name": "Input_B", "transformations": [...] }
      ]
    }
  ]
}

It will be a list of connections (edges) between datasets and/or fields. This will allow column level lineage (a field and its input fields), dataset level lineage (a dataset and its input datasets), and other mixed scenarios where the fields read or written are known or unknown. Each input can include an array descriptive of the transformations, like in the Column Level Lineage facet.

Not all datasets read (inputs) need to be in the lineage list (they may have been read but not used by the job). In principle, almost all written datasets (outputs) will be in the lineage list, except those written without any data from the inputs. It might make sense – or not – to mandate that the lineage list only refers datasets and fields declared in the inputs and outputs.

Mixed granularity

This structure allows mixing macroscopic (dataset level) and microscopic (column level) lineage in the same payload, solving the "all-or-nothing" problem of the current Column Level Lineage facet.

It even allows cases where the producer can only gather column level information from one side of the relation.

Example Payload:

"lineage": [
  // Dataset Level Lineage (Macro)
  {
    "namespace": "NS1",
    "name": "X",
    "inputs": [{ "namespace": "NS1", "name": "A" }]
  },
  // Column Level Lineage (Micro)
  {
    "namespace": "NS1",
    "name": "Y",
    "field": "y1",
    "inputs": [{ "namespace": "NS1", "name": "B", "field": "b1" }]
  },
  // Column from table(s): column level lineage with no input fields
  {
    "namespace": "NS1",
    "name": "Z",
    "field": "z1",
    "inputs": [
      { "namespace": "NS1", "name": "A" }
    ]
  },
  // Table from column(s): column level lineage with no target field
  {
    "namespace": "NS1",
    "name": "Z",
    "inputs": [
      { "namespace": "NS1", "name": "B", "field": "b2" }
    ]
  }
]

Static / design-time lineage

For the case of static lineage (or others where it makes sense), it is necessary to support an edge removal syntax.

It is suggested to utilize the "_deleted": true that already exists for facet removal, but adapted to the context of a list (removing items identified by the remaining key properties: namespace, name, and field).

Example Payload:

"lineage": [
  // Removing a column level lineage edge
  {
    "_deleted": true,
    "namespace": "NS1",
    "name": "Z",
    "field": "z2",
    "inputs": [
      // Will delete only B.b2 -> Z.z2
      // i.e. if A.a2 -> Z.z2 exists, it will be kept
      { "namespace": "NS1", "name": "B", "field": "b2" }
    ]
  }
]

Migration & Compatibility

If done carefully, e.g. with the new property as optional, this can be a non-breaking change.

This list does not replace the list of inputs and outputs (they continue to be the datasets read and written in this Run).

This list completely replaces the Column Level Lineage facet. Consumers can still process it for backwards compatibility, but if both are provided in the same Run, the new property should take precedence.

• Producers: Legacy integrations continue just sending inputs/outputs, and eventually column level lineage as a facet. Advanced integrations add the new property.
• Consumers:
  • If not supporting the new property:
    • Processes the events as before
  • Else If the new property is present:
    • Use it to build the lineage graph
  • Else:
    • Fallback to current logic

Alternative proposal

If introducing a new root property is too disruptive, we can alternatively try to implement this via a standard facet.

However, the question arises: A facet of what?

Option A: Dataset Facet

• Concept: Attach lineage information to outputs[].facets (like the current Column Level Lineage facet).
• Pros: Consistent with current Column Level Lineage facet.
• Cons:
  • Decentralization: Lineage logic is scattered across multiple output objects rather than centralized.
  • Category Error: Why is information about movement and origin of data a property/characteristic of a dataset?
  • Ambiguity: As with the current Column Level Lineage facet, it is unclear what lineage on an input dataset means. And in a Dataset Event, does it augment or replace the existing lineage information associated with the dataset?

Option B: Run Facet

• Concept: Attach the lineage list to run.facets.
• Pros: Centralized logic.
• Cons:
  • Static Lineage Gap: Run facets are not present in Job Events and Dataset Events. This would make it impossible to define design-time lineage.

Option C: Job Facet

• Concept: Attach the lineage list to job.facets.
• Pros:
  • Universal Availability: Valid in both Run Event (runtime) and Job Event (static/design-time).
  • Semantic Fit: "Job A transforms Data X into Data Y" is possibly cleaner than treating it as a property of the data.
  • Centralization: All lineage logic for the unit of work is contained in one list.

---

Thoughts? We welcome your feedback and look forward to discussing this further.