Lineage map: an ordered "Architecture Layer" option next to Service / Domain / Data Product (medallion / Data Vault)

Aperta
  • Qualcosa non va?
#35,872 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
data, data-engineering

Direzione di ricerca

Start by reading the lineage map work in #29204 and #32042, then trace the Lineage Layer selector and how dbt tags are ingested. The request defines an ordered architecture layer, rule based and dbt assignment, map grouping, and later layout, styling, flow checks, and Impact Analysis changes. Done means supporting the core grouping and drill-down behavior; the issue leaves implementation boundaries and sequencing open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement
Is your feature request related to a problem? Please describe.

The lineage map coming in 2.1 (#29204, under #32042) groups the graph with a Lineage Layer selector (Service, Domain, Data Product) and rolls connections up, so "a single line can stand for many asset-level connections". That makes a large graph readable, but it misses the grouping most warehouse teams think in: the architecture layer.

Warehouses are built in layers with a fixed direction of flow: medallion (bronze → silver → gold) or Data Vault (sources → staging → raw vault → business vault → information marts). "Which layer is this in, and does data flow the right way between layers?" is usually the first question a reader of the lineage has.

None of the existing dimensions answers it:

  • Schema is often where a layer lives, but only by naming convention, and it is not a grouping option of the map. It breaks when a layer spans several schemas or databases, when schema names carry environment or user prefixes (DBT_<user>_STAGING), or when one schema holds two layers.
  • Domain and data product are business groupings (Finance, Recruitment); using them for technical layers blocks their real use.
  • Tier expresses importance, not position in the flow. Service is usually one warehouse for every layer.
  • Tags/classifications can carry a layer (we set them per dbt folder), but they have no order and are only shown in the side panel, not on the graph.

In the asset-level graph, the automatic layout orders nodes by hop distance, so a mart three hops from one source and five from another lands in different columns, and staging, vault and mart tables share a column.

Describe the solution you'd like

Core: an "Architecture Layer" option in the lineage map's Lineage Layer selector.

  1. An ordered layer dimension, defined once by an admin, e.g. Source < Staging < Raw Vault < Business Vault < Mart or Bronze < Silver < Gold. A natural fit: a classification flagged as "architecture layer", whose tags carry an order. An asset has at most one layer.
  2. Assignment without hand work: by rule on service / database / schema name pattern (*STAGING* → Staging), inherited downwards; and from dbt (meta or tags per folder), which the dbt ingestion already reads.
  3. "Architecture Layer" next to Service / Domain / Data Product: the map rolls up to Staging → Raw Vault → Business Vault → Mart with edge counts, and drills into the assets of a layer like the other options do.

Building on it, in the asset-level graph:

  1. Layout bands: a column (or swimlane) per layer, in layer order, instead of hop depth alone.
  2. A colour or stripe per layer on the node card, with a legend, and a filter to show a range of layers.
  3. Flow checks (optional): highlight edges that go backwards (a lower layer reading a higher one) or skip layers the organisation does not allow (e.g. a mart reading staging directly). Many teams enforce such rules in code review; showing violations in the catalog makes the rule visible to everyone.
  4. Impact Analysis: a Layer column and filter.
Describe alternatives you've considered
  • Grouping by schema: not offered by the map, and would only work where layer == schema, see above.
  • Domains per layer: the Domain option then groups visually, but domains are lost for business ownership.
  • Tags per layer (what we use today): searchable and filterable in Explore and Impact Analysis, but not on the lineage node, no order, no effect on layout.
  • A separate lineage diagram outside OpenMetadata: possible from the API, but splits the catalog experience.
Additional context
  • Related:
    • #29204 (hierarchical lineage maps)
    • #32042 (hierarchical lineage at scale)
    • #35862 (DIRECT vs INDIRECT edges: a layered layout reads better when indirect edges carry less weight).
  • Our setup: OpenMetadata 2.0.5, Snowflake, dbt-core 1.12 with a Data Vault (AutomateDV) in staging / raw vault / business vault / mart schemas. Layer tags set per dbt folder already arrive in OpenMetadata as dbtTags.layer_*.
Pre-submission checklist
  • I searched for duplicate feature requests.
Lingua principale
TypeScript
Stelle
15.4k
Fork
2.4k
Merge medio
1g 2h
PR unite (30g)
693

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di open-metadata/OpenMetadata

Tutte le issue di open-metadata/OpenMetadata

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.