Connector-centric registry: one place to add an integration

Aperta
  • Qualcosa non va?
#22 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, typescript
Ambito
backend, backend-api-design, database, frontend

Direzione di ricerca

Read docs/architecture-review.md first, then inspect the existing dispatch in src/dapier/engine/init.py, validation in src/dapier/triggers/email_triggers.py, and the designer catalog in designer/src/catalog.ts. A first incremental milestone is the registry wrapper described in migration step 1; run make test and consider it done when dispatch uses the registry while the existing tests still pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Full analysis in docs/architecture-review.md. The trigger→action engine is sound; what doesn't scale is that connector knowledge lives in four hand-maintained registries across two languages.

Problem

Adding one action today touches:

  1. src/dapier/engine/actions/<x>.py
  2. the run_x import + if/elif dispatch in src/dapier/engine/__init__.py
  3. ACTION_SPECS validation in src/dapier/triggers/email_triggers.py
  4. the hand-mirrored designer/src/catalog.ts (comment says "Mirrors the run_* dispatch")

Adding one trigger source touches:

  1. worker.normalize_payload branches
  2. template.yaml (per-source queue/table/env)
  3. api/router.py hook routing
  4. a load_workflows() hook in matching.all_workflows

Nothing links these copies; they drift silently. Related structural limits: no data flow between steps (blocks #6, #7, #12), O(all-workflows) resolution per event (published-table scan capped at 200, silently), and per-action auth folklore (credential_id vs connection_id vs auth_secret_id).

Proposal

  • src/dapier/connectors/<name>/ — one package per integration: a manifest (label, events, actions + field schema), normalize(raw) -> event per ingress it owns, and run(action, ctx) per action. A connectors/registry.py replaces the dispatch chain, ACTION_SPECS, and the trigger-load hooks.
  • GET /api/catalog — serve the manifests as JSON; the designer renders palette/inspector from it (keep catalog.ts as a dev fallback). One registry, three consumers: engine dispatch, validation, UI.
  • ActionContext — ctx.connection(name) → live token (auto-refreshed), plus event, workflow, run id, and a steps namespace so later steps can template earlier outputs (first enabler for #6).
  • One workflows table with a connector#event GSI — merge the published/email/hook/schedule stores; matching becomes a query per event instead of four scans + a linear pass. Instant-publish semantics unchanged.
  • Generic ingress — a route/queue/source → connector mapping with one ingress handler; per-connector queues become declared resources instead of template.yaml surgery.

Migration order (incremental, no big bang)

  1. Registry wrapping the existing run_* functions; dispatch reads the registry. Tests keep passing.
  2. Move normalize_payload branches into connectors; keep the SQS contract.
  3. /api/catalog; designer consumes it; delete the TS mirror and ACTION_SPECS duplication.
  4. ActionContext + steps + unified connection_id (keep credential_id/auth_secret_id as deprecated aliases).
  5. Single workflows table + trigger GSI.
  6. Then #6/#7/#10 become small changes on top instead of structural ones.

The 23 src/*.py back-compat shims are already removed (2026-09-25); make test was 426 passing at filing time.

Lingua principale
Python
Stelle
0
Fork
0
Merge medio
6m
PR unite (30g)
10

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di DataTalksClub/dapier

Tutte le issue di DataTalksClub/dapier

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.