Expand agents-server conformance coverage for Cloud/reference parity
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 静か
- 技術スタック
- typescript
調査の方向性
packages/agents-server-conformance-tests/src/electric-agents-tests.ts と packages/agents-server/test/conformance.test.ts の参照ハーネスから始め、次に server-claim-write-token.test.ts や runners-router.test.ts などの P0 claim-token および runner テストを確認します。外部から見えるシナリオのみを機能別のスイートに移植し、実装のみに関するテストは packages/agents-server/test に残します。完了とは、参照実装と Cloud が同じ選択された conformance の動作を実行することです。
索引モデルが issue の本文から書いたものです。
説明
Context
We confirmed that Electric Cloud is primarily running the agents server conformance suite for agents-server compatibility. That means any agents-server behavior that only exists in packages/agents-server/test/*.test.ts is not necessarily being validated against Cloud.
Current conformance package:
packages/agents-server-conformance-tests- Main suite:
packages/agents-server-conformance-tests/src/electric-agents-tests.ts - Used by reference implementation via:
packages/agents-server/test/conformance.test.ts
The reference server conformance harness currently calls:
runElectricAgentsConformanceTests(config)runCliConformanceTests(config)when CLI existsrunMockAgentTests(config)runMockAgentCliTests(config)when CLI exists
The conformance suite already covers a lot of core public entity runtime behavior, but a large amount of important agents-server behavior remains only in reference implementation tests under packages/agents-server/test.
We should expand the conformance suite so Cloud and the reference implementation are tested against the same externally visible contract.
Current conformance coverage
The conformance suite already covers many core public behaviors:
Entity lifecycle
- spawn registered entity type
- reject unknown entity type
- reject duplicate entity URL
- spawn without explicit body type
- get/list/ps
- status filters
- kill
- send after kill rejected
- stream persists after kill
- full spawn → send → webhook → read → kill lifecycle
Entity type registry
- register/list/inspect/delete entity type
- duplicate registration upserts
- reject malformed registration
- minimal type registration without
creation_schema
Typed spawn / schema validation
- validate
creation_schema - reject invalid spawn args
- parent spawn handling
- explicit/default tags at spawn
- reject non-string tags at spawn
Send/schema validation
- validate
input_schemas - reject invalid/unknown typed messages
- behavior with no schemas / empty schemas
Schema evolution
- schema amendment adding message types
- reject modifications to existing schema keys
- multiple schema amendments against existing entities
- amendment on deleted type
Serve endpoint
- register type via serve endpoint
- reject serve endpoint name mismatch
State Protocol send format
- inbox event shape
type/key/value/headerscontract- ordering expectations
- parseability by runtime webhook handler expectations
Error paths
- kill missing entity
- send missing payload
- delete missing type
- stopped tag write rejected without claim
Auth boundary
- direct writes to entity stream require claim/write token
- wrong/no token rejected
- spawn/get do not leak write tokens
- unauthenticated
sendremains allowed - non-entity stream writes remain unauthenticated
Concurrency / property-based
- randomized action sequences preserve safety invariants
- concurrent spawns under same type
- sequential tag writes through claim token
Problem
The conformance suite does not yet cover several important externally visible agents-server behaviors that are currently tested only by the reference implementation.
The reference server has many additional test files under:
packages/agents-server/test
Several of these tests validate public API or protocol behavior that Cloud should also satisfy.
Important gaps include:
- claim-scoped write-token lifecycle
- runner API and pull-wake behavior
- dispatch policy routing
- scheduling APIs
- wake/manifest semantics
- fork behavior
- multi-tenant / owner-scoped isolation
- shared-state / Durable Streams proxy behavior
Goal
Move or port the externally visible, black-box agents-server behavior tests into packages/agents-server-conformance-tests, so they can run against:
- the reference implementation
- Electric Cloud
- any future compatible agents-server implementation
The conformance suite should define the public compatibility contract. Reference-only implementation details should stay in packages/agents-server/test.
Non-goals
Do not move implementation-specific unit tests into conformance.
Examples that should generally stay reference-only:
- migration folder resolution
- environment variable parsing
- low-level route hook composition
- schema middleware unit tests
- URL helper unit tests
- Postgres-specific scheduler internals
- LMDB/local token store internals
- outbox retry/dead-letter implementation details
- server startup cleanup internals
- OSS dashboard/router wrapper behavior
- low-level adapter mechanics unless exposed as public protocol behavior
Proposed structure
Split the conformance package into capability-based suites, instead of one very large catch-all runner.
Suggested exports:
runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)
runForkConformanceTests(config)
runCliConformanceTests(config)
runMockAgentTests(config)
runMockAgentCliTests(config)
runElectricAgentsConformanceTests(config) can continue to exist as a broad/default suite, but Cloud should be able to opt into capability-specific suites explicitly.
Example:
runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)
This also makes it easier to skip suites for capabilities Cloud does not yet expose.
Suggested priority
P0: Claim-scoped write-token lifecycle
Source/reference tests to inspect:
packages/agents-server/test/server-claim-write-token.test.tspackages/agents-server/test/claim-write-token-store.test.ts
Port the externally visible behavior into conformance.
Candidate conformance scenarios:
- claim returns a scoped write token
- active claim token can write to entity stream
- no token is rejected
- wrong token is rejected
- stale token is rejected after a newer claim
- stale
donecannot mark a newer active claim idle - kill clears/invalidates active claim token
- tag writes accept the active claim token
- claim token is accepted in supported headers, e.g.
Authorization/electric-claim-token - claim-scoped writes validate output/state schemas
- unknown event types are rejected when output schemas exist
- arbitrary events are accepted when no state schemas exist
- writes to stopped entities are rejected even with a token
- claim-scoped tag writes reject non-string values
- claim-scoped tag writes support merge/delete semantics
Why P0:
- This is security-sensitive.
- It is core to the agent wake/write contract.
- Cloud/reference divergence here would be serious.
P0: Runner API / pull-wake contract
Source/reference tests to inspect:
packages/agents-server/test/runners-router.test.tspackages/agents-server/test/pull-wake-subscription-stack.test.tspackages/agents-server/test/dispatch-policy-routing.test.tspackages/agents-server/test/webhook-forward-routing.test.tspackages/agents-server/test/runtime-registry.test.ts
Port public API/protocol behavior into conformance.
Candidate conformance scenarios:
- register a runner
- list runners for authenticated owner
- reject unauthenticated runner listing/claims
- reject runner registration for a different owner
- canonicalize owner principal inputs
- runner health includes expected public diagnostics
- runner diagnostics are sanitized
- runner is unhealthy when lease expires
- runner is degraded/unhealthy when stream is disconnected/disabled
- spawn entity with runner-targeted dispatch policy
- send creates a runner wake
- runner claim returns enriched notification
- claim conflicts map to 409
- callback/done acknowledges wake
- pull-wake subscription receives pre-existing wake events
- missing runner dispatch subscription is recreated before send
- runner dispatch subscription creation conflicts are treated idempotently
Why P0:
- This is likely a major Cloud/reference parity surface.
- Runners are a core part of the agents architecture.
- The current conformance suite mostly exercises webhook-style wake handling, not runner pull-wake behavior.
P1: Scheduling APIs
Source/reference tests to inspect:
packages/agents-server/test/electric-agents-routes.test.tspackages/agents-server/test/scheduler-integration.test.tspackages/agents-server/test/scheduler.test.tspackages/agents-server/test/server-start.test.ts
Port only public behavior. Avoid Postgres/LISTEN/NOTIFY internals.
Candidate conformance scenarios:
- future/delayed send schedule can be created
- delayed send lands exactly once
- delayed send can be replaced
- delayed send can be cancelled/deleted
- cron schedule can be created
- cron schedule rejects invalid/missing expression
- cron schedule emits at least one expected send/wake
- deleting cron schedule prevents future delivery
- scheduled payload validation happens before enqueueing
Why P1:
- Scheduling is externally visible.
- It is easy for Cloud and reference implementation to diverge due to infrastructure differences.
P1: Wake / manifest semantics
Source/reference tests to inspect:
packages/agents-server/test/wake-registry.test.tspackages/agents-server/test/wake-registry-sync.test.tspackages/agents-server/test/manifest-side-effects.test.tspackages/agents-server/test/horton-spawn-worker.test.tspackages/agents-server/test/horton-title-generation.test.tspackages/agents-server/test/horton-pull-wake-e2e.test.ts
Candidate conformance scenarios:
- spawn with wake registers a condition
- child
runFinishedwake is delivered to parent/subscriber runFinishedwake includes child text response by defaultincludeResponse: falseomits response- failed child run includes error
- unrelated/unscoped errors are ignored
- event append triggers collection/change wake
- wake event contains source/change details
- timeout wake is delivered
- debounce coalesces rapid events
- debounced
runFinishedpreserves requested status/includeResponse options
Why P1:
- Wake semantics are core to agent composition.
- Some of this is currently deeply tested in the reference server but not necessarily in Cloud.
P1: Fork behavior
Source/reference tests to inspect:
packages/agents-server/test/electric-agents-routes.test.tspackages/agents-server/test/electric-agents-status.test.tspackages/agents-server/test/stream-client-fork.test.ts
Candidate conformance scenarios:
- fork endpoint returns public entity/entities
- forked streams preserve source history
- JSON appends to forked streams are accepted
- fork is rejected or waits while subtree is active, depending on spec
- sends to subtree entities are rejected while fork snapshotting is in progress
- fork locks are released after failure
Why P1:
- Forking is visible API behavior if exposed in Cloud.
- It has concurrency/consistency semantics that should be tested across implementations.
P2: Multi-tenant / owner isolation
Source/reference tests to inspect:
packages/agents-server/test/entity-projector.test.tspackages/agents-server/test/entity-type-registry.test.tspackages/agents-server/test/wake-registry.test.tspackages/agents-server/test/tag-stream-outbox-registry.test.tspackages/agents-server/test/tag-stream-outbox-drainer.test.tspackages/agents-server/test/host.test.tspackages/agents-server/test/server-utils.test.ts
This likely requires extending the conformance DSL/config to support auth headers, tenant IDs, or principal injection.
Candidate conformance scenarios:
- tenant A/B entity type names do not collide
- tenant A cannot list tenant B entities
- tenant A cannot inspect tenant B entity type
- runner listing is scoped to authenticated owner
- runner runtime diagnostics shapes are owner-scoped
- no cross-tenant wake delivery
- tenant-specific entity streams are isolated
Why P2:
- Important for Cloud.
- May require DSL/auth harness changes before tests can be written cleanly.
P2: Shared-state / Durable Streams proxy behavior
Source/reference tests to inspect:
packages/agents-server/test/electric-agents-routes.test.tspackages/agents-server/test/stream-client.test.ts
Candidate conformance scenarios, if part of the public contract:
- shared-state stream traffic routes correctly
- subscription control-plane traffic routes correctly
- reserved
__dspaths behave as expected - configured bearer auth is forwarded for service-scoped subscription traffic
- subscription-scoped bearer auth is preserved for ack/proxy traffic
- subscription stream paths are relative to the resolved stream root
Some of these may be adapter-specific. Only move the externally visible compatibility requirements.
Cleanup: remove runtime tool tests from server conformance
runElectricAgentsConformanceTests currently includes a block named:
Electric Agents - Tool Tests
These tests import directly from:
../../agents-runtime/src/tools
They test things like:
- bash tool stdout/stderr capture
- bash timeout
- read_file path/binary/size guard
- web_search/fetch_url tool interface
These are not really agents-server conformance tests. They depend on monorepo source layout and runtime implementation details.
Suggested change:
- Move these tests back to
packages/agents-runtime/test, or - create a separate
runRuntimeToolConformanceTestsexport if we intentionally want runtime/tool conformance.
They should not run as part of black-box agents-server compatibility against Cloud.
Implementation plan
Step 1: Add conformance suite structure
Create additional files under packages/agents-server-conformance-tests/src, for example:
src/core-entity-tests.ts
src/claim-write-token-tests.ts
src/runner-tests.ts
src/schedule-tests.ts
src/wake-tests.ts
src/fork-tests.ts
Update src/index.ts to export the new suite runners.
Keep backward compatibility:
export function runElectricAgentsConformanceTests(config) {
runCoreEntityConformanceTests(config)
// optionally include currently existing broad set, or keep as alias during migration
}
Step 2: Extend the DSL/config as needed
The current DSL in:
packages/agents-server-conformance-tests/src/electric-agents-dsl.ts
may need support for:
- default request headers
- auth/principal headers
- tenant IDs
- runner registration helpers
- claim helpers
- schedule helpers
- wait/poll helpers
- stream assertion helpers
Suggested config shape:
export interface ElectricAgentsTestOptions {
baseUrl: string
headers?: Record<string, string> | (() => Record<string, string> | Promise<Record<string, string>>)
capabilities?: {
runners?: boolean
schedules?: boolean
wakes?: boolean
fork?: boolean
multiTenant?: boolean
}
}
Need to be careful not to make tests silently skip too much. Capability skips should be explicit and visible.
Step 3: Port P0 claim-token tests
Port from reference tests to black-box HTTP scenarios.
Avoid importing reference server internals.
Step 4: Port P0 runner/pull-wake tests
Port public runner API and wake claiming behavior.
Avoid asserting internal database rows or implementation-specific subscription IDs unless they are part of the public response contract.
Step 5: Port P1 schedule/wake/fork tests
Move in batches. Each batch should be runnable against both:
- local reference server
- Cloud target
Step 6: Wire reference implementation to run all conformance suites
Update:
packages/agents-server/test/conformance.test.ts
to call the new suite exports.
Step 7: Wire Cloud test job to run all supported conformance suites
Wherever Cloud currently runs conformance, update it to include the newly exported suites.
Cloud should report which conformance capabilities were run/skipped.
Acceptance criteria
- Conformance package exports separate suite runners for core, claim-token, runner, schedule, wake, and fork behavior.
- Reference server conformance harness runs the new suites.
- Cloud conformance job runs all supported suites.
- Claim-scoped write-token lifecycle is covered in conformance.
- Runner registration/list/claim/health and pull-wake delivery are covered in conformance.
- Public scheduling API behavior is covered in conformance, if supported by Cloud.
- Public wake/manifest behavior is covered in conformance, if supported by Cloud.
- Fork behavior is covered in conformance, if supported by Cloud.
- Runtime tool tests are removed from the agents-server conformance runner or split into a separate runtime/tool suite.
- Implementation-specific tests remain in
packages/agents-server/testand are not moved to conformance. - Conformance tests do not import reference server internals.
- Conformance tests can run against an arbitrary
baseUrl. - Capability skips are explicit and visible in test output.
Notes
The highest-value work is probably:
- claim-scoped write-token conformance
- runner/pull-wake conformance
These are security/correctness-critical and most likely to reveal Cloud/reference divergence.
- 主要言語
- TypeScript
- スター
- 10.4k
- フォーク
- 399
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
electric-sql/electric のほかの issue
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error value対応中かも @Tyagiquamar が 19 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)対応中かも @mohanedmoh が 37 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
electric-sql/electric#4826 · コメント 1 件 ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wake対応中かも @adityavkk が 133 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
electric-sql/electric#4687 ·
-
Listen IP addressオープンfeature request triage
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
electric-sql/electric#4150 · コメント 1 件 ·
electric-sql/electric の issue をすべて見る
似ている issue
-
clawsweeper:fix-shape-clear clawsweeper:queueable-fix clawsweeper:source-repro impact:ux-friction issue-rating: 🦞 diamond lobster no-stale P2
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
openclaw/openclaw#199885 · コメント 2 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
-
feature needs-triage
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
BoardGamesEmpire/board-games-empire-backend#1306 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
sequant-io/sequant#1689 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100