Connector-centric registry: one place to add an integration

Abierto
  • ¿Algo no está bien?
#22 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, typescript
Área
backend, backend-api-design, database, frontend

Línea de trabajo

Read docs/architecture-review.md first, then inspect the existing dispatch in src/dapier/engine/init.py, validation in src/dapier/triggers/email_triggers.py, and the designer catalog in designer/src/catalog.ts. A first incremental milestone is the registry wrapper described in migration step 1; run make test and consider it done when dispatch uses the registry while the existing tests still pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Full analysis in docs/architecture-review.md. The trigger→action engine is sound; what doesn't scale is that connector knowledge lives in four hand-maintained registries across two languages.

Problem

Adding one action today touches:

  1. src/dapier/engine/actions/<x>.py
  2. the run_x import + if/elif dispatch in src/dapier/engine/__init__.py
  3. ACTION_SPECS validation in src/dapier/triggers/email_triggers.py
  4. the hand-mirrored designer/src/catalog.ts (comment says "Mirrors the run_* dispatch")

Adding one trigger source touches:

  1. worker.normalize_payload branches
  2. template.yaml (per-source queue/table/env)
  3. api/router.py hook routing
  4. a load_workflows() hook in matching.all_workflows

Nothing links these copies; they drift silently. Related structural limits: no data flow between steps (blocks #6, #7, #12), O(all-workflows) resolution per event (published-table scan capped at 200, silently), and per-action auth folklore (credential_id vs connection_id vs auth_secret_id).

Proposal

  • src/dapier/connectors/<name>/ — one package per integration: a manifest (label, events, actions + field schema), normalize(raw) -> event per ingress it owns, and run(action, ctx) per action. A connectors/registry.py replaces the dispatch chain, ACTION_SPECS, and the trigger-load hooks.
  • GET /api/catalog — serve the manifests as JSON; the designer renders palette/inspector from it (keep catalog.ts as a dev fallback). One registry, three consumers: engine dispatch, validation, UI.
  • ActionContext — ctx.connection(name) → live token (auto-refreshed), plus event, workflow, run id, and a steps namespace so later steps can template earlier outputs (first enabler for #6).
  • One workflows table with a connector#event GSI — merge the published/email/hook/schedule stores; matching becomes a query per event instead of four scans + a linear pass. Instant-publish semantics unchanged.
  • Generic ingress — a route/queue/source → connector mapping with one ingress handler; per-connector queues become declared resources instead of template.yaml surgery.

Migration order (incremental, no big bang)

  1. Registry wrapping the existing run_* functions; dispatch reads the registry. Tests keep passing.
  2. Move normalize_payload branches into connectors; keep the SQS contract.
  3. /api/catalog; designer consumes it; delete the TS mirror and ACTION_SPECS duplication.
  4. ActionContext + steps + unified connection_id (keep credential_id/auth_secret_id as deprecated aliases).
  5. Single workflows table + trigger GSI.
  6. Then #6/#7/#10 become small changes on top instead of structural ones.

The 23 src/*.py back-compat shims are already removed (2026-09-25); make test was 426 passing at filing time.

Lenguaje dominante
Python
Estrellas
0
Forks
0
Merge medio
6 min
PR fusionados (30 d)
10

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de DataTalksClub/dapier

Todos los issues de DataTalksClub/dapier

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.