Connector-centric registry: one place to add an integration

オープン
  • 何か問題がありますか？
#22 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python, typescript
領域
backend, backend-api-design, database, frontend

調査の方向性

Read docs/architecture-review.md first, then inspect the existing dispatch in src/dapier/engine/init.py, validation in src/dapier/triggers/email_triggers.py, and the designer catalog in designer/src/catalog.ts. A first incremental milestone is the registry wrapper described in migration step 1; run make test and consider it done when dispatch uses the registry while the existing tests still pass.

索引モデルが issue の本文から書いたものです。

説明

Full analysis in docs/architecture-review.md. The trigger→action engine is sound; what doesn't scale is that connector knowledge lives in four hand-maintained registries across two languages.

Problem

Adding one action today touches:

  1. src/dapier/engine/actions/<x>.py
  2. the run_x import + if/elif dispatch in src/dapier/engine/__init__.py
  3. ACTION_SPECS validation in src/dapier/triggers/email_triggers.py
  4. the hand-mirrored designer/src/catalog.ts (comment says "Mirrors the run_* dispatch")

Adding one trigger source touches:

  1. worker.normalize_payload branches
  2. template.yaml (per-source queue/table/env)
  3. api/router.py hook routing
  4. a load_workflows() hook in matching.all_workflows

Nothing links these copies; they drift silently. Related structural limits: no data flow between steps (blocks #6, #7, #12), O(all-workflows) resolution per event (published-table scan capped at 200, silently), and per-action auth folklore (credential_id vs connection_id vs auth_secret_id).

Proposal

  • src/dapier/connectors/<name>/ — one package per integration: a manifest (label, events, actions + field schema), normalize(raw) -> event per ingress it owns, and run(action, ctx) per action. A connectors/registry.py replaces the dispatch chain, ACTION_SPECS, and the trigger-load hooks.
  • GET /api/catalog — serve the manifests as JSON; the designer renders palette/inspector from it (keep catalog.ts as a dev fallback). One registry, three consumers: engine dispatch, validation, UI.
  • ActionContext — ctx.connection(name) → live token (auto-refreshed), plus event, workflow, run id, and a steps namespace so later steps can template earlier outputs (first enabler for #6).
  • One workflows table with a connector#event GSI — merge the published/email/hook/schedule stores; matching becomes a query per event instead of four scans + a linear pass. Instant-publish semantics unchanged.
  • Generic ingress — a route/queue/source → connector mapping with one ingress handler; per-connector queues become declared resources instead of template.yaml surgery.

Migration order (incremental, no big bang)

  1. Registry wrapping the existing run_* functions; dispatch reads the registry. Tests keep passing.
  2. Move normalize_payload branches into connectors; keep the SQS contract.
  3. /api/catalog; designer consumes it; delete the TS mirror and ACTION_SPECS duplication.
  4. ActionContext + steps + unified connection_id (keep credential_id/auth_secret_id as deprecated aliases).
  5. Single workflows table + trigger GSI.
  6. Then #6/#7/#10 become small changes on top instead of structural ones.

The 23 src/*.py back-compat shims are already removed (2026-09-25); make test was 426 passing at filing time.

主要言語
Python
スター
0
フォーク
0
平均マージ
6分
マージ済み PR（30日）
10

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

DataTalksClub/dapier のほかの issue

DataTalksClub/dapier の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。