Metadata Store

Non vengono indicati file del repository né test. Inizia leggendo l'ambito di v1 e l'epic OpenLineage collegato (#856), quindi confronta i percorsi API proposti e le considerazioni sull'archiviazione PostgreSQL descritte qui. Il lavoro è completato quando sono concordati il design del servizio MVP, i confini API supportati, l'approccio all'archiviazione degli eventi e l'ambito del deployment basato su Helm.

Stackable Metadata Store - v1

[!NOTE]
Yes, I'm aware the name is dull and technical. Anyone with a better idea is welcome to propose one.

[!IMPORTANT]
This is very much under construction

SDP deploys a dozen or more products, some of which hold data and metadata worth knowing about. We want one service that collects as much of the platform's metadata as it can, stores it, and makes it available to other services such as our Cockpit UI.

Scope

[!NOTE]
We have many ideas to extend this later. For our first version, this section describes the scope of what we want to achieve.

A read-only service that ingests OpenLineage events, stores them, and serves one or two clients/APIs:

• Cockpit probably via a custom (but documented and usable by others) API
• Stretch/Optional: OpenLineage out as well (so it can function as some kind of proxy)
  • Investigate this briefly as I do not want us to offer a Push API as part of this. If that is the only way for OpenLineage to gather data.

Technical details

• Unless there are really really good arguments otherwise I'd like this to be a Rust based service, using Axum and PostgreSQL as the sole supported backing store.
• I don't know enough about this but from what I understand we probably don't need an operator for this: A Helm chart should be enough
• The "URL space" API should be ours and it should be designed so we can extend it later with more APIs to ingest and export.

/ingest/v1/openlineage, /ingest/v1/<foosource>, /v1/ingest/openlineage or /openlineage are all up for discussion. Two things to keep in mind: for OpenLineage there will be ingest and output at the same time, and it is an open question whether our own API and the external ones share a prefix or get separate ones (/api/ and /ingest/openlineage, or all the same).

No layout will be perfect, so pick one rather than overthinking it. Especially for this very first version. We will mark it experimental so we can learn a bit.

All tools we support can configure the path, so we can pick whatever. The default some of them use is /api/v1/lineage.

Just remember that each of these APIs might behave slightly different also with authentication & authorisation.

OpenLineage - Technical details

[!NOTE]
This is A LOT, but it cost me hours to distill this. It's important and hopefully helpful. Do NOT skip this.

OpenLineage is complex and troublesome. Also see our OpenLineage epic and talk to the colleagues if it helps.

It emits several kinds of event and we need to decite how we want to store which.

Here is one example (this might not be 100% correct, AI generated based on it reading the code)

Apache Spark reads raw.orders and writes the Iceberg table sales.orders_daily through a Hive catalog:

{
  "eventType": "COMPLETE",
  "job": {
    "namespace": "prod-lineage",
    "name":      "orders_etl.execute_replace_table_as_select_command"
  },
  "run": { "runId": "c7f2…-01b8", "facets": {} },
  "inputs": [
    { "namespace": "s3a://warehouse", "name": "raw.db/orders" }
  ],
  "outputs": [{
    "namespace": "s3a://warehouse",
    "name":      "sales.db/orders_daily",
    "facets": {
      "symlinks": { "identifiers": [
        { "namespace": "hive://hive-metastore:9083",
          "name":      "sales.orders_daily",
          "type":      "TABLE" }
      ]}
    }
  }]
}

Trino then queries that same table through catalog lakehouse and writes sales.orders_summary:

{
  "eventType": "COMPLETE",
  "job": {
    "namespace": "prod-lineage",
    "name":      "20260902_101500_00042_ab3kd"   // default $QUERY_ID
  },
  "run": { "runId": "9d13…-77e0" },
  "inputs": [{
    "namespace": "trino://trino-coordinator-default-0.trino-coordinator-default.prod.svc.cluster.local:8443",
    "name":      "lakehouse.sales.orders_daily"
  }],
  "outputs": [{
    "namespace": "trino://trino-coordinator-default-0.…:8443",
    "name":      "lakehouse.sales.orders_summary"
  }]
}

Both emitters are configured with the same OpenLineage namespace, and both describe the same physical table. Unfortunately, they do not agree on what it is called:

Emitted by	Namespace	Name
Spark, primary	s3a://warehouse	sales.db/orders_daily
Spark, symlink	hive://hive-metastore:9083	sales.orders_daily
Trino	trino://trino-coordinator-default-0.…:8443	lakehouse.sales.orders_daily

• The scheme differs. s3a against hive against trino: physical storage, catalog service, query engine. All three correct, but...annoying for us.
• The separator differs. Spark writes sales.db/orders_daily from the warehouse path, its own symlink writes sales.orders_daily, Trino writes dots throughout.
• Trino prefixes its catalog name. lakehouse. is a Trino-side concept with no counterpart anywhere in Spark's output. You cannot resolve it without knowing which catalog points at which metastore.

So, even the symlinks thing doesn’t help us. It gives us a hive:// when we need to match a trino://

In this MVP we will not be able to resolve this and it’s out of scope too. We should be able to do clever things later because the information is in TrinoCatalog and so on AND we have the source code for all those OpenLineage things under control so we’ll have to tweak this later but for now this matching between products etc. is out of scope.

That means (namespace, name) is not an identity.
We can still use it for lookups and it probably should get a unique index.

But it doesn’t uniquely identify “this is our customer table”.

This is the definition of an OpenLineage dataset: {namespace, name, facets} (no id) and a name is unique only within a namespace.

Dataset, asset, subject

Which is why I think we end up with three layers. It took me a long time to arrive here and it might still be wrong, please challenge me.

I’d like us to only build the very first of these layers now but the schema should make sure that the others are possible later.

Look at the three events from the table above, and add more product: NiFi reading order events from a Kafka topic and writing them into that same table from above.

• dataset: Every one of those names (namespace,name) from above (trino://.../lakehouse.sales.order_daily, s3a://..) plus the new kafka://../orders one. This is the terminology OpenLineage uses. Four rows.
• asset: The physical thing those names refer to. In this case two different things: The orders_daily table which is really what the three things from the table name and the orders topic from Kafka. Two things.
• subject: The logical thing spanning several assets. order data covering both the table and the topic. One thing.

I made the names asset and subject up for this.

We can’t just get thisf rom lineage data, we need to reconstruct this (especially assets) later by looking e.g. at catalogs and other smart stuff. And some of it will have to be manual (especially subjects). Just because a job has something as input and output doesn’t mean that they are “the same thing”.

So, why bother? Because I think it could be a very very nice feature if you can tag and describe at every level:

• "Customer data must not be modified" belongs on the subject. And it immediately applies everywhere.
• "This one table is deprecated" belongs on the asset.
• I can’t really come up with where a dataset is useful here but... I’m sure we’ll find something.

This is all “Zukunftsmusik”. I just didn’t want to forget about this.

Out of scope for all of it

• A write path: We'll want this later, but not for this.
• Any other APIs than the ones I mentioned above (Cockpit/OpenLineage)
• Any kind of Web UI -> all of that lives in Cockpit
• Any kind of pruning of old data. The store grows without limit for now.