stats.sonicjs.com: /admin/content 500 — D1 overloaded by CI playwright ingest bursts

Aperta
#1,210 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
playwright, typescript

Direzione di ricerca

Start by tracing playwrightIngest and the bootstrap KV skip-marker behavior described in the issue, then inspect the D1 batch insert path and its tests. Reproduce the overload with a large ingest batch and check how retries, batch sizing, and bootstrap behavior affect D1 usage. Done means ingest bursts no longer cause repeated bootstrap work or 500 responses on admin routes; the issue does not identify specific files or tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Summary

/admin/content (and likely all admin routes) return {"error":"Internal Server Error","status":500} on stats.sonicjs.com due to D1 resource exhaustion caused by high-volume CI playwright ingest traffic.

Root cause (confirmed from wrangler tail)

The stats D1 hits Cloudflare's per-10-second resource limit when the CI ships large playwright test batches. Every cold-start isolate re-runs bootstrap (because bootstrap never caches its KV skip-marker when D1 is overloaded), creating a feedback loop:

D1_ERROR: D1 DB is overloaded. Too many resources consumed within a 10 second window.

Errors seen on:

  • [MigrationService] auth_two_factor lockout column repair failed
  • [Bootstrap] Error registering document types
  • [Bootstrap] Error auto-registering collection document types
  • [PluginBootstrap] Error ensuring plugin Authentication System / Media Manager / Lexical Editor
  • [playwright] /v1/playwright-runs insert failed

When a user hits /admin/content during a burst, any D1 query inside the route handler throws the same overload error. It escapes the route's try/catch (RBAC middleware runs before the route handler), hits the global app.onError → JSON 500.

Trigger

CI is processing multiple playwright runs in parallel (logs show simultaneous POST /v1/playwright-runs/{id}/tests from run 44587182379-1). Each batch inserts ~750 test rows via D1 batch(). At peak parallelism this saturates the stats D1's 10s resource window.

Secondary issue: JWT_SECRET not set

Every cold-start logs:

[SonicJS Security] JWT_SECRET is not set — using hardcoded fallback.

The stats worker is running without a JWT_SECRET Wrangler secret — admin sessions are signed with the public fallback key (security risk).

Fixes

Immediate:

  • Set JWT_SECRET on the stats worker: wrangler secret put JWT_SECRET --name sonicjs-stats

Short-term (D1 overload):

  • Throttle playwright ingest: add a small delay/retry between batch inserts in playwrightIngest
  • Cap per-request D1 batch size below 500 statements to stay under D1's row-write budget per 10s window
  • Reduce CI playwright ingest parallelism

Long-term:

  • Bootstrap KV skip-marker never writes when D1 is overloaded → every cold-start isolate retries full bootstrap during the same burst. Add in-memory backoff or circuit-breaker to avoid compounding the overload.
  • Consider upgrading the stats D1 from the free tier to a paid plan for higher resource limits.

Steps to reproduce

Trigger a large CI playwright run while tailing wrangler tail sonicjs-stats --format=pretty, then visit /admin/content during the ingest burst.

Lingua principale
TypeScript
Stelle
1.8k
Fork
219
Merge medio
2g 21h
PR unite (30g)
67

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SonicJs-Org/sonicjs

Tutte le issue di SonicJs-Org/sonicjs

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.