Multiple static Retry-After values throughout the codebase

未关闭
#4,295 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
重构
描述清晰度
基本清楚
活跃度
冷清
技术栈
elixir, typescript

调研方向

从 Electric.AdmissionControl.try_acquire/3 及其过载 telemetry 开始,然后跟踪 lib/electric/plug/serve_shape_plug.ex、lib/electric/shapes/api.ex 和 lib/electric/shapes/api/params.ex 中列出的调用点。将它们当前的 Retry-After 路径与 lib/electric/shapes/api/response.ex 进行比较,后者应继续作为 response 层的 serializer。完成意味着瞬时请求路径使用同一个 policy,其增长、jitter、cap 和 reset 行为都有覆盖,并且 router/API 测试不再断言静态值。

由索引模型根据 Issue 内容生成。

描述

Problem

Retry-After is currently calculated in several separate places in packages/sync-service, mostly as fixed values. This makes retry behavior inconsistent and prevents the server from adapting during overload, redeploy, or stack transition events.

This is especially risky because the TypeScript client treats server Retry-After as a floor and combines it with client-side jitter. A fixed server value like 10 can dominate early client backoff attempts and cause many clients to retry together.

Current call sites

  • lib/electric/plug/serve_shape_plug.ex
    • DB connection errors set retry-after: 10 directly.
    • admission-control overload calls a private calculate_retry_after/2, currently random 6..10 seconds.
  • lib/electric/shapes/api.ex
    • shape creation failure: retry_after: 1
    • stack not ready after timeout: retry_after: 5
    • snapshot creation errors: retry_after: 10
    • long-poll timeout while stack is not up/read-only: retry_after: 10
  • lib/electric/shapes/api/params.ex
    • DB unavailable while validating shape params: retry_after: 1
  • lib/electric/shapes/api/response.ex
    • only serializes the value into the retry-after header and exposes it via CORS; this should likely remain a dumb response-layer concern.

AdmissionControl already owns the core overload decision, per-stack/per-kind counters, and reject telemetry, but it currently returns only {:error, :overloaded}. The caller then invents the retry delay locally.

Proposal

Centralize Retry-After calculation in logic rooted in Electric.AdmissionControl.

For admission overloads, try_acquire/3 should either return retry metadata, e.g. {:error, {:overloaded, retry_after}}, or expose a central helper that callers use immediately after rejection.

The retry policy should use exponential backoff with jitter, keyed by overload context such as:

  • stack_id
  • request kind: :initial or :existing
  • reason: admission overload, stack not ready, DB unavailable, snapshot pressure, etc.

Suggested behavior:

  • use jitter for every transient 503
  • increase delays under sustained rejection/overload
  • reset or decay the backoff when capacity recovers
  • cap the initial implementation, e.g. at 120s
  • keep Cache-Control: no-store and Surrogate-Control: no-store on transient overload responses
  • emit telemetry including retry_after, reason, kind, current count, limit, and rejection/backoff state

Other transient 503 paths should call the same policy instead of hard-coding 1, 5, or 10.

Acceptance criteria

  • No hard-coded sync-service transient Retry-After values remain in request handling paths.
  • Admission-control rejection responses get Retry-After from the central policy.
  • The policy has unit coverage for exponential growth, jitter bounds, cap behavior, and reset/decay.
  • Router/API tests assert policy behavior rather than specific static values.
  • Docs that mention fixed Retry-After values are updated if the public behavior changes.

Open questions

  • Should the exponential “attempt” be server-side overload streak state, client-provided attempt metadata, or a hybrid?
  • Should DB unavailable, stack not ready, and snapshot pressure share one policy with different baselines, or use separate reason-specific profiles?
主要语言
TypeScript
星标
10.4k
派生
399
PR 合并指标
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

electric-sql/electric 的其他 Issue

查看 electric-sql/electric 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。