mtg-graph: cut /diagnostics latency by collapsing its Neo4j round trips

Aperta
  • Qualcosa non va?
#383 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
39/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
neo4j, python
Ambito
backend, databases, performance

Direzione di ricerca

Start in services/mtg-graph/src/deck_lab/diagnostics.py and read diagnose() and the listed query callers; compare the seven sequential Neo4j queries and trace the unaccounted gap around deck_theme_breakdown. Profile DECK_CARD_TYPES and DECK_CARD_RESOURCES before and after any rewrite, and update the monkeypatches in tests/test_diagnostics.py and tests/test_api_tracing.py. Done means a comparable trace shows the endpoint well under 100 ms, with before and after numbers posted on the issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Tracking issue for speeding up POST /api/graph/diagnostics (diagnose() in services/mtg-graph/src/deck_lab/diagnostics.py).

Baseline

Trace 3eae5143f2b97765f1412bffd26b77d7 (91-card deck, cache miss): 194 ms end to end, 169 ms in the mtg-graph endpoint. The mtg side (session, identity, account, proxy) adds only ~4 ms.

Seven Neo4j queries run sequentially, all UNWIND $rows … MATCH (c:Card {oracle_id}) over the same deck, together ~133 ms:

ms query caller
20.1 DECK_CARDS fetch_deck
17.7 DECK_CARD_RESOURCES (BROADER*0..) deck_card_resources
12.9 DECK_CARD_ROLES deck_card_roles
43.2 DECK_CARD_TYPES deck_card_types
9.0 DECK_ROLE_WEIGHTS deck_role_weights
16.2 DECK_PRODUCED deck_resource_balance
14.0 DECK_WANTED deck_resource_balance

Plus a ~25 ms gap with no query between DECK_CARD_ROLES and DECK_CARD_TYPES.

Even the trivial DECK_ROLE_WEIGHTS costs 9 ms, which points at per-query overhead (round trip, planning, record decoding in the Python driver) rather than work in the database. Fewer round trips is the lever. Moving to Postgres was considered and rejected: the bottleneck is not the engine, and the rest of mtg-graph relies on PARENT_OF*0.. / BROADER*0.. traversals, so a split would mean keeping two stores in sync from ingest.

Tasks

  • Derive the three aggregate queries in Python (~39 ms, low risk)
    • deck_role_weights: sum(weight * qty) per role from card_roles, which diagnose() already holds
    • DECK_PRODUCED: cards and names per expanded resource, from resources_by_card[*]["produces"] + cards
    • DECK_WANTED: reads CARES_ABOUT without hierarchy expansion, unlike resources_by_card["cares_about"]. Have DECK_CARD_RESOURCES also return the direct cares-about set, then derive locally
    • Keep build_diagnostics pure; update the monkeypatches in tests/test_diagnostics.py and tests/test_api_tracing.py
  • Rewrite DECK_CARD_TYPES with pattern comprehensions instead of three chained OPTIONAL MATCH … collect / WITH stages; check with PROFILE before and after. Same check for DECK_CARD_RESOURCES
  • Run the remaining per-card fetches concurrently, or as one query (fetch_deck, resources, roles, types depend only on deck): either a ThreadPoolExecutor (wall time = slowest query instead of the sum) or a single combined per-card query (one round trip, one MATCH per card)
  • Find the ~25 ms gap: add a span around deck_theme_breakdown (and the eminence/cEDH steps) or profile with py-spy, then decide whether it needs work
  • (optional) Materialize hierarchy closures at ingest (e.g. expanded PRODUCES/CARES_ABOUT edges or list properties) so per-deck reads need no *0.. traversal. Only if the steps above leave the resource queries dominant

Done when

A new trace for a comparable deck shows the endpoint well under 100 ms, with the before/after numbers posted here.

Lingua principale
TypeScript
Stelle
0
Fork
1
Merge medio
9h 39m
PR unite (30g)
144

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di myOmikron/code

Tutte le issue di myOmikron/code

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.