Metadata store: MVP

The payload names no repository files, tests, or entry points. Start by resolving the storage model and service boundaries against the parent issue's example events, then define the Trino-to-store-to-Cockpit path; done means a real Trino query appears as one or more datasets in Cockpit with the listed operational requirements addressed.

Metadata store: MVP

We want a view in Cockpit which shows all datasets that reach the Metadata Store as OpenLineage events from Trino.

This issue is the end-to-end work on that: configuring Trino to emit, the store itself, and the view.

Scope

• Trino can post OpenLineage to our new product
• The metadata store stores the data in PostgreSQL
  • Storage model is the big open question (see below)
• Cockpit can retrieve the data and show a list of datasets (the frontend part of this can be a demo/throwaway thing entirely for now)
• Server TLS
• NO authentication on either end
• NO authorisation
• Trino posts synchronously with a default timeout of 5s. If we’re slow or down we stall each query. We therefore could consider a configurable async mode where you trade potential data loss vs. performance
  • Related: We should also consider improving Trino here and sending this in a background thread and also allow for retries...ping me should you want to do anything on this before you start.
• Graceful shutdown: So we don't lose any requests in flight/accepted (especially re: point above)
• Readiness/Health endpoint (should check for PostgreSQL health/availability, canary queries etc.)
• Set up the service for logging (so it fits with our Vector stuff), metrics (Prometheus) and traces (OpenTelemetry). This is about the service's own behaviour
• Bring your own PostgreSQL (we don’t do anything about shipping one!)

Done when

A real Trino query on a real cluster shows up as one (or more) datasets in Cockpit.

Open Questions

• The biggest one: Storage model. We receive OpenLineage (and later potentially other stuff): How do we store it? Do we store the verbatim data we received and/or some processed form?
  • I’m leaning towards both.
    • If we want to serve OpenLineage again later we should probably just store it in an append-only JSONB thing.
    • If we want to make it useful for Cockpit et. al. then we should try to extract information from it and for that we need something structured. I don’t know what yet exactly besides a “dataset” table. We probably should start small and evolve with use-cases
    • And if we later change our mind about the extracted information, we can backfill it from the source data.
• Without authentication/authorisation we do not have any information about the source who sent something to us. Not sure if we should already design such a column? I’m not even sure how it’d look like. We might actually need two columns, one for the “type” and one for the actual principal if it exists (e.g. we could also have a type “inferred”)
• Helm chart or operator (I see no reason for an operator but...not an expert)
• Minimum PostgreSQL version
• Is there existing Rust OpenLineage stuff or do we need to build our own?

Storage Model

I put some thought into this but this is not final or mandatory, that said I have probably spent more time (so far) than most at the company thinking about OpenLineage and these other formats.

I think we need at least three tables to start with:

• events -> This is OpenLineage terminology and captures the raw jsonb events we receive, it should have a few more columns maybe for event type, event time, job namespace etc. that are useful to extract (append only)
• datasets -> This is one of our most important tables and it almost certainly will evolve. For now it needs to have a unique id, and a namespace and name column and a unique index on those, then something something “learned from” but there could be more than one sources...so not sure.
• event_datasets linking an event to a dataset potentially with a direction (input/output)

See the parent issue for example events and more information. The namespace has a scheme which we should parse. The rest of the namespace is tool dependent and we can leave it for now (trino, s3a, hive etc.)

Hint: OpenLineage supports “symlinks” and I think we just want to expand them into datasets maybe of a type symlink but...not sure.

For this version I only want us to focus on the dataset layer, also in the cockpit. Even if that means that the same thing will be shown three times.

Expected effort

15 days. If it takes longer, stop and let me know.