Entities become silently unreachable after `deleteEntity`+respawn or `agents stop`/`start` — stale wake/runner bindings

未关闭
#4,368 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
45/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
冷清
技术栈
docker, postgresql, typescript

调研方向

首先通过 electric-ax agents CLI 以及对 registerTypes()、spawnEntity()、deleteEntity() 和 sendEntityMessage() 的 bridge 调用,复现这两个触发器。结合列出的 PostgreSQL 表,跟踪 agents-server 的 wake/runner 生命周期及其进程内存状态。完成标准是:重启和使用相同 URL 的 respawn 能保留活动 binding,清理过期状态,并且没有活动 binding 的消息绝不会静默收到 204。

由索引模型根据 Issue 内容生成。

描述

Summary

There are two ways to leave a spawned entity silently unreachable. In both, the entity row exists in postgres and the agents UI shows the entity as present, but messages sent to it are never processed. The two have different triggers and slightly different failure signatures, but they look like the same underlying problem: the wake/runner binding for an entity URL is not reconciled, so the entity is "alive" in storage but has nothing wired up to actually run its handler.

Filing them together because a fix for one likely addresses the other.

Environment

  • @electric-ax/agents-runtime 0.2.2
  • @electric-ax/durable-streams-client-beta 0.3.1
  • agents-server 0.4.x (local electric-ax agents Docker stack)
  • electric-ax CLI 0.2.2
  • macOS, Docker Desktop

Setup in all repros: a long-running host process ("the bridge") that calls registerTypes() to register entity types against its own webhook URL, then spawns entities and relays messages to them.

Trigger 1 — entities unreachable after agents stop && agents start (no volume wipe)

Repro
  1. Start the stack: electric-ax agents start.
  2. From the bridge: registerTypes(), then spawnEntity({ type, id }). Verify the entity wakes and processes a sendEntityMessage (works).
  3. electric-ax agents stop (no --remove-volumes).
  4. electric-ax agents start.
  5. Restart the bridge process.
  6. sendEntityMessage to the entity → server returns HTTP 204, but no wake is ever dispatched to the bridge webhook. The entity never runs. A direct HTTP send to the entity's inbox endpoint behaves the same: 204, silent, no wake.
Scope

Only entities that existed at the moment of agents stop are affected. An entity spawned after the restart works normally — until the next stop.

Diagnostic that isolated it

Changing the entity id to a value that had never been used, then restarting the bridge, produced a fresh entity that woke normally. We then spawned a new entity inside that working post-restart session — it worked — and ran agents stop && agents start a second time. The new entity was now also dead. So the discriminator is the substrate stop event itself, not how recently the entity was spawned or which bridge process created it.

Reading

Entity state in postgres survives the restart intact. The wake-registration that binds "entity URL → webhook URL" is not rebound to the restarted bridge's webhook on agents start, so wake dispatch never fires.

Trigger 2 — entities unreachable after deleteEntity + respawn on the same URL

Repro
  1. Start the stack and spawn an entity; confirm it wakes on sendEntityMessage (works).
  2. client.deleteEntity('/type/id') → returns 2xx success.
  3. client.spawnEntity({ type, id }) again on the same URL → returns success, no error.
  4. sendEntityMessage to that URL → 409 {"code":"NOT_RUNNING","message":"Entity is stopped"}.

No agents stop is involved — this happens within a single substrate uptime.

What we investigated
  • Postgres is not cleaned by deleteEntity. After step 2, SELECT url FROM entities WHERE url LIKE '/type/%' still returns the row.
  • A direct SQL purge does not fix it. We DELETEd the URL's rows directly across all 8 per-entity tables (entities, wake_registrations, entity_dispatch_state, entity_manifest_sources, scheduled_tasks, tag_stream_outbox, wake_notifications, consumer_claims), then respawned. The send still returns 409.
  • entity_dispatch_state is half-initialized after respawn. The row exists, but every active_* column (active_consumer_id, active_runner_id, active_epoch, active_claimed_at, active_lease_expires_at) is NULL — no runner is bound.
  • Only a process restart clears it. electric-ax agents stop --remove-volumes && electric-ax agents start lets a fresh spawn on the same URL work again. Since a clean postgres alone (via the SQL purge above) does not fix it, the broken state appears to live in agents-server process memory, not in postgres.

Common thread

In both cases: the entity row exists, postgres looks normal, the agents UI shows the entity — but no runner/wake binding is actually live for it. A full process restart of agents-server is the only thing that reliably clears the bad state.

Worst-shaped part

Neither failure surfaces at spawn time. spawnEntity returns success and the agents UI shows the entity green. Trigger 1 is the worse of the two: an inbox write to an orphaned entity returns HTTP 204 — a success response — and the message is silently dropped. Application code has no signal that the message will never be processed. Trigger 2 at least returns an explicit 409.

Suggested fixes

  • On agents start, walk the entity registry and re-validate wake-registrations against the webhook URLs registered by the current registerTypes() calls. Rebind rows whose target entity still exists; drop rows whose entity is gone. Log a summary of rehydrated vs. dropped registrations.
  • Make deleteEntity truly destructive — clear postgres rows, in-process cache, and wake/runner bindings — so a URL is genuinely free for reuse; or make spawnEntity on a stale URL re-initialize the in-process lifecycle state and bind a runner.
  • At minimum: an inbox write or sendEntityMessage to an entity with no live wake binding should return a loud error (e.g. 409 / 503), never a silent 204. A successful HTTP response should mean the message will actually be dispatched.

Workaround

In application code: never reuse an entity URL. Generate a unique suffix per session (e.g. /type/id-<sessionId>); old entities become dead rows in postgres but never collide with new ones. Also avoid agents stop && agents start on a substrate with live entities you still need. Both workarounds leak dead entity rows over time.

主要语言
TypeScript
星标
10.4k
派生
399
PR 合并指标
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

electric-sql/electric 的其他 Issue

查看 electric-sql/electric 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。