stats.sonicjs.com: /admin/content 500 — D1 overloaded by CI playwright ingest bursts

Open
#1,210 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
playwright, typescript

Research direction

Start by tracing playwrightIngest and the bootstrap KV skip-marker behavior described in the issue, then inspect the D1 batch insert path and its tests. Reproduce the overload with a large ingest batch and check how retries, batch sizing, and bootstrap behavior affect D1 usage. Done means ingest bursts no longer cause repeated bootstrap work or 500 responses on admin routes; the issue does not identify specific files or tests.

Written by the indexing model from the issue text.

Description

bug

Summary

/admin/content (and likely all admin routes) return {"error":"Internal Server Error","status":500} on stats.sonicjs.com due to D1 resource exhaustion caused by high-volume CI playwright ingest traffic.

Root cause (confirmed from wrangler tail)

The stats D1 hits Cloudflare's per-10-second resource limit when the CI ships large playwright test batches. Every cold-start isolate re-runs bootstrap (because bootstrap never caches its KV skip-marker when D1 is overloaded), creating a feedback loop:

D1_ERROR: D1 DB is overloaded. Too many resources consumed within a 10 second window.

Errors seen on:

  • [MigrationService] auth_two_factor lockout column repair failed
  • [Bootstrap] Error registering document types
  • [Bootstrap] Error auto-registering collection document types
  • [PluginBootstrap] Error ensuring plugin Authentication System / Media Manager / Lexical Editor
  • [playwright] /v1/playwright-runs insert failed

When a user hits /admin/content during a burst, any D1 query inside the route handler throws the same overload error. It escapes the route's try/catch (RBAC middleware runs before the route handler), hits the global app.onError → JSON 500.

Trigger

CI is processing multiple playwright runs in parallel (logs show simultaneous POST /v1/playwright-runs/{id}/tests from run 44587182379-1). Each batch inserts ~750 test rows via D1 batch(). At peak parallelism this saturates the stats D1's 10s resource window.

Secondary issue: JWT_SECRET not set

Every cold-start logs:

[SonicJS Security] JWT_SECRET is not set — using hardcoded fallback.

The stats worker is running without a JWT_SECRET Wrangler secret — admin sessions are signed with the public fallback key (security risk).

Fixes

Immediate:

  • Set JWT_SECRET on the stats worker: wrangler secret put JWT_SECRET --name sonicjs-stats

Short-term (D1 overload):

  • Throttle playwright ingest: add a small delay/retry between batch inserts in playwrightIngest
  • Cap per-request D1 batch size below 500 statements to stay under D1's row-write budget per 10s window
  • Reduce CI playwright ingest parallelism

Long-term:

  • Bootstrap KV skip-marker never writes when D1 is overloaded → every cold-start isolate retries full bootstrap during the same burst. Add in-memory backoff or circuit-breaker to avoid compounding the overload.
  • Consider upgrading the stats D1 from the free tier to a paid plan for higher resource limits.

Steps to reproduce

Trigger a large CI playwright run while tailing wrangler tail sonicjs-stats --format=pretty, then visit /admin/content during the ingest burst.

Dominant language
TypeScript
Stars
1.8k
Forks
219
Avg merge
2d 21h
Merged PRs (30d)
67

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SonicJs-Org/sonicjs

All issues in SonicJs-Org/sonicjs

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.