Expand agents-server conformance coverage for Cloud/reference parity

未关闭
#4,358 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
冷清
技术栈
typescript

调研方向

先从 packages/agents-server-conformance-tests/src/electric-agents-tests.ts 和 packages/agents-server/test/conformance.test.ts 中的参考 harness 开始,然后检查 P0 claim-token 和 runner 测试,例如 server-claim-write-token.test.ts 和 runners-router.test.ts。只将外部可见的场景移植到按 capability 划分的 suite 中,并将仅针对实现的测试保留在 packages/agents-server/test 中。完成的标准是参考实现和 Cloud 运行相同的所选 conformance 行为。

由索引模型根据 Issue 内容生成。

描述

Context

We confirmed that Electric Cloud is primarily running the agents server conformance suite for agents-server compatibility. That means any agents-server behavior that only exists in packages/agents-server/test/*.test.ts is not necessarily being validated against Cloud.

Current conformance package:

  • packages/agents-server-conformance-tests
  • Main suite: packages/agents-server-conformance-tests/src/electric-agents-tests.ts
  • Used by reference implementation via: packages/agents-server/test/conformance.test.ts

The reference server conformance harness currently calls:

  • runElectricAgentsConformanceTests(config)
  • runCliConformanceTests(config) when CLI exists
  • runMockAgentTests(config)
  • runMockAgentCliTests(config) when CLI exists

The conformance suite already covers a lot of core public entity runtime behavior, but a large amount of important agents-server behavior remains only in reference implementation tests under packages/agents-server/test.

We should expand the conformance suite so Cloud and the reference implementation are tested against the same externally visible contract.

Current conformance coverage

The conformance suite already covers many core public behaviors:

Entity lifecycle
  • spawn registered entity type
  • reject unknown entity type
  • reject duplicate entity URL
  • spawn without explicit body type
  • get/list/ps
  • status filters
  • kill
  • send after kill rejected
  • stream persists after kill
  • full spawn → send → webhook → read → kill lifecycle
Entity type registry
  • register/list/inspect/delete entity type
  • duplicate registration upserts
  • reject malformed registration
  • minimal type registration without creation_schema
Typed spawn / schema validation
  • validate creation_schema
  • reject invalid spawn args
  • parent spawn handling
  • explicit/default tags at spawn
  • reject non-string tags at spawn
Send/schema validation
  • validate input_schemas
  • reject invalid/unknown typed messages
  • behavior with no schemas / empty schemas
Schema evolution
  • schema amendment adding message types
  • reject modifications to existing schema keys
  • multiple schema amendments against existing entities
  • amendment on deleted type
Serve endpoint
  • register type via serve endpoint
  • reject serve endpoint name mismatch
State Protocol send format
  • inbox event shape
  • type/key/value/headers contract
  • ordering expectations
  • parseability by runtime webhook handler expectations
Error paths
  • kill missing entity
  • send missing payload
  • delete missing type
  • stopped tag write rejected without claim
Auth boundary
  • direct writes to entity stream require claim/write token
  • wrong/no token rejected
  • spawn/get do not leak write tokens
  • unauthenticated send remains allowed
  • non-entity stream writes remain unauthenticated
Concurrency / property-based
  • randomized action sequences preserve safety invariants
  • concurrent spawns under same type
  • sequential tag writes through claim token

Problem

The conformance suite does not yet cover several important externally visible agents-server behaviors that are currently tested only by the reference implementation.

The reference server has many additional test files under:

packages/agents-server/test

Several of these tests validate public API or protocol behavior that Cloud should also satisfy.

Important gaps include:

  1. claim-scoped write-token lifecycle
  2. runner API and pull-wake behavior
  3. dispatch policy routing
  4. scheduling APIs
  5. wake/manifest semantics
  6. fork behavior
  7. multi-tenant / owner-scoped isolation
  8. shared-state / Durable Streams proxy behavior

Goal

Move or port the externally visible, black-box agents-server behavior tests into packages/agents-server-conformance-tests, so they can run against:

  • the reference implementation
  • Electric Cloud
  • any future compatible agents-server implementation

The conformance suite should define the public compatibility contract. Reference-only implementation details should stay in packages/agents-server/test.

Non-goals

Do not move implementation-specific unit tests into conformance.

Examples that should generally stay reference-only:

  • migration folder resolution
  • environment variable parsing
  • low-level route hook composition
  • schema middleware unit tests
  • URL helper unit tests
  • Postgres-specific scheduler internals
  • LMDB/local token store internals
  • outbox retry/dead-letter implementation details
  • server startup cleanup internals
  • OSS dashboard/router wrapper behavior
  • low-level adapter mechanics unless exposed as public protocol behavior

Proposed structure

Split the conformance package into capability-based suites, instead of one very large catch-all runner.

Suggested exports:

runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)
runForkConformanceTests(config)
runCliConformanceTests(config)
runMockAgentTests(config)
runMockAgentCliTests(config)

runElectricAgentsConformanceTests(config) can continue to exist as a broad/default suite, but Cloud should be able to opt into capability-specific suites explicitly.

Example:

runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)

This also makes it easier to skip suites for capabilities Cloud does not yet expose.

Suggested priority

P0: Claim-scoped write-token lifecycle

Source/reference tests to inspect:

  • packages/agents-server/test/server-claim-write-token.test.ts
  • packages/agents-server/test/claim-write-token-store.test.ts

Port the externally visible behavior into conformance.

Candidate conformance scenarios:

  • claim returns a scoped write token
  • active claim token can write to entity stream
  • no token is rejected
  • wrong token is rejected
  • stale token is rejected after a newer claim
  • stale done cannot mark a newer active claim idle
  • kill clears/invalidates active claim token
  • tag writes accept the active claim token
  • claim token is accepted in supported headers, e.g. Authorization / electric-claim-token
  • claim-scoped writes validate output/state schemas
  • unknown event types are rejected when output schemas exist
  • arbitrary events are accepted when no state schemas exist
  • writes to stopped entities are rejected even with a token
  • claim-scoped tag writes reject non-string values
  • claim-scoped tag writes support merge/delete semantics

Why P0:

  • This is security-sensitive.
  • It is core to the agent wake/write contract.
  • Cloud/reference divergence here would be serious.
P0: Runner API / pull-wake contract

Source/reference tests to inspect:

  • packages/agents-server/test/runners-router.test.ts
  • packages/agents-server/test/pull-wake-subscription-stack.test.ts
  • packages/agents-server/test/dispatch-policy-routing.test.ts
  • packages/agents-server/test/webhook-forward-routing.test.ts
  • packages/agents-server/test/runtime-registry.test.ts

Port public API/protocol behavior into conformance.

Candidate conformance scenarios:

  • register a runner
  • list runners for authenticated owner
  • reject unauthenticated runner listing/claims
  • reject runner registration for a different owner
  • canonicalize owner principal inputs
  • runner health includes expected public diagnostics
  • runner diagnostics are sanitized
  • runner is unhealthy when lease expires
  • runner is degraded/unhealthy when stream is disconnected/disabled
  • spawn entity with runner-targeted dispatch policy
  • send creates a runner wake
  • runner claim returns enriched notification
  • claim conflicts map to 409
  • callback/done acknowledges wake
  • pull-wake subscription receives pre-existing wake events
  • missing runner dispatch subscription is recreated before send
  • runner dispatch subscription creation conflicts are treated idempotently

Why P0:

  • This is likely a major Cloud/reference parity surface.
  • Runners are a core part of the agents architecture.
  • The current conformance suite mostly exercises webhook-style wake handling, not runner pull-wake behavior.
P1: Scheduling APIs

Source/reference tests to inspect:

  • packages/agents-server/test/electric-agents-routes.test.ts
  • packages/agents-server/test/scheduler-integration.test.ts
  • packages/agents-server/test/scheduler.test.ts
  • packages/agents-server/test/server-start.test.ts

Port only public behavior. Avoid Postgres/LISTEN/NOTIFY internals.

Candidate conformance scenarios:

  • future/delayed send schedule can be created
  • delayed send lands exactly once
  • delayed send can be replaced
  • delayed send can be cancelled/deleted
  • cron schedule can be created
  • cron schedule rejects invalid/missing expression
  • cron schedule emits at least one expected send/wake
  • deleting cron schedule prevents future delivery
  • scheduled payload validation happens before enqueueing

Why P1:

  • Scheduling is externally visible.
  • It is easy for Cloud and reference implementation to diverge due to infrastructure differences.
P1: Wake / manifest semantics

Source/reference tests to inspect:

  • packages/agents-server/test/wake-registry.test.ts
  • packages/agents-server/test/wake-registry-sync.test.ts
  • packages/agents-server/test/manifest-side-effects.test.ts
  • packages/agents-server/test/horton-spawn-worker.test.ts
  • packages/agents-server/test/horton-title-generation.test.ts
  • packages/agents-server/test/horton-pull-wake-e2e.test.ts

Candidate conformance scenarios:

  • spawn with wake registers a condition
  • child runFinished wake is delivered to parent/subscriber
  • runFinished wake includes child text response by default
  • includeResponse: false omits response
  • failed child run includes error
  • unrelated/unscoped errors are ignored
  • event append triggers collection/change wake
  • wake event contains source/change details
  • timeout wake is delivered
  • debounce coalesces rapid events
  • debounced runFinished preserves requested status/includeResponse options

Why P1:

  • Wake semantics are core to agent composition.
  • Some of this is currently deeply tested in the reference server but not necessarily in Cloud.
P1: Fork behavior

Source/reference tests to inspect:

  • packages/agents-server/test/electric-agents-routes.test.ts
  • packages/agents-server/test/electric-agents-status.test.ts
  • packages/agents-server/test/stream-client-fork.test.ts

Candidate conformance scenarios:

  • fork endpoint returns public entity/entities
  • forked streams preserve source history
  • JSON appends to forked streams are accepted
  • fork is rejected or waits while subtree is active, depending on spec
  • sends to subtree entities are rejected while fork snapshotting is in progress
  • fork locks are released after failure

Why P1:

  • Forking is visible API behavior if exposed in Cloud.
  • It has concurrency/consistency semantics that should be tested across implementations.
P2: Multi-tenant / owner isolation

Source/reference tests to inspect:

  • packages/agents-server/test/entity-projector.test.ts
  • packages/agents-server/test/entity-type-registry.test.ts
  • packages/agents-server/test/wake-registry.test.ts
  • packages/agents-server/test/tag-stream-outbox-registry.test.ts
  • packages/agents-server/test/tag-stream-outbox-drainer.test.ts
  • packages/agents-server/test/host.test.ts
  • packages/agents-server/test/server-utils.test.ts

This likely requires extending the conformance DSL/config to support auth headers, tenant IDs, or principal injection.

Candidate conformance scenarios:

  • tenant A/B entity type names do not collide
  • tenant A cannot list tenant B entities
  • tenant A cannot inspect tenant B entity type
  • runner listing is scoped to authenticated owner
  • runner runtime diagnostics shapes are owner-scoped
  • no cross-tenant wake delivery
  • tenant-specific entity streams are isolated

Why P2:

  • Important for Cloud.
  • May require DSL/auth harness changes before tests can be written cleanly.
P2: Shared-state / Durable Streams proxy behavior

Source/reference tests to inspect:

  • packages/agents-server/test/electric-agents-routes.test.ts
  • packages/agents-server/test/stream-client.test.ts

Candidate conformance scenarios, if part of the public contract:

  • shared-state stream traffic routes correctly
  • subscription control-plane traffic routes correctly
  • reserved __ds paths behave as expected
  • configured bearer auth is forwarded for service-scoped subscription traffic
  • subscription-scoped bearer auth is preserved for ack/proxy traffic
  • subscription stream paths are relative to the resolved stream root

Some of these may be adapter-specific. Only move the externally visible compatibility requirements.

Cleanup: remove runtime tool tests from server conformance

runElectricAgentsConformanceTests currently includes a block named:

Electric Agents - Tool Tests

These tests import directly from:

../../agents-runtime/src/tools

They test things like:

  • bash tool stdout/stderr capture
  • bash timeout
  • read_file path/binary/size guard
  • web_search/fetch_url tool interface

These are not really agents-server conformance tests. They depend on monorepo source layout and runtime implementation details.

Suggested change:

  • Move these tests back to packages/agents-runtime/test, or
  • create a separate runRuntimeToolConformanceTests export if we intentionally want runtime/tool conformance.

They should not run as part of black-box agents-server compatibility against Cloud.

Implementation plan

Step 1: Add conformance suite structure

Create additional files under packages/agents-server-conformance-tests/src, for example:

src/core-entity-tests.ts
src/claim-write-token-tests.ts
src/runner-tests.ts
src/schedule-tests.ts
src/wake-tests.ts
src/fork-tests.ts

Update src/index.ts to export the new suite runners.

Keep backward compatibility:

export function runElectricAgentsConformanceTests(config) {
  runCoreEntityConformanceTests(config)
  // optionally include currently existing broad set, or keep as alias during migration
}
Step 2: Extend the DSL/config as needed

The current DSL in:

packages/agents-server-conformance-tests/src/electric-agents-dsl.ts

may need support for:

  • default request headers
  • auth/principal headers
  • tenant IDs
  • runner registration helpers
  • claim helpers
  • schedule helpers
  • wait/poll helpers
  • stream assertion helpers

Suggested config shape:

export interface ElectricAgentsTestOptions {
  baseUrl: string
  headers?: Record<string, string> | (() => Record<string, string> | Promise<Record<string, string>>)
  capabilities?: {
    runners?: boolean
    schedules?: boolean
    wakes?: boolean
    fork?: boolean
    multiTenant?: boolean
  }
}

Need to be careful not to make tests silently skip too much. Capability skips should be explicit and visible.

Step 3: Port P0 claim-token tests

Port from reference tests to black-box HTTP scenarios.

Avoid importing reference server internals.

Step 4: Port P0 runner/pull-wake tests

Port public runner API and wake claiming behavior.

Avoid asserting internal database rows or implementation-specific subscription IDs unless they are part of the public response contract.

Step 5: Port P1 schedule/wake/fork tests

Move in batches. Each batch should be runnable against both:

  • local reference server
  • Cloud target
Step 6: Wire reference implementation to run all conformance suites

Update:

packages/agents-server/test/conformance.test.ts

to call the new suite exports.

Step 7: Wire Cloud test job to run all supported conformance suites

Wherever Cloud currently runs conformance, update it to include the newly exported suites.

Cloud should report which conformance capabilities were run/skipped.

Acceptance criteria

  • Conformance package exports separate suite runners for core, claim-token, runner, schedule, wake, and fork behavior.
  • Reference server conformance harness runs the new suites.
  • Cloud conformance job runs all supported suites.
  • Claim-scoped write-token lifecycle is covered in conformance.
  • Runner registration/list/claim/health and pull-wake delivery are covered in conformance.
  • Public scheduling API behavior is covered in conformance, if supported by Cloud.
  • Public wake/manifest behavior is covered in conformance, if supported by Cloud.
  • Fork behavior is covered in conformance, if supported by Cloud.
  • Runtime tool tests are removed from the agents-server conformance runner or split into a separate runtime/tool suite.
  • Implementation-specific tests remain in packages/agents-server/test and are not moved to conformance.
  • Conformance tests do not import reference server internals.
  • Conformance tests can run against an arbitrary baseUrl.
  • Capability skips are explicit and visible in test output.

Notes

The highest-value work is probably:

  1. claim-scoped write-token conformance
  2. runner/pull-wake conformance

These are security/correctness-critical and most likely to reveal Cloud/reference divergence.

主要语言
TypeScript
星标
10.4k
派生
399
PR 合并指标
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

electric-sql/electric 的其他 Issue

查看 electric-sql/electric 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。