Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503

Abierto
  • ¿Algo no está bien?
#35,995 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
node.js, sqlite, typescript
Área
backend, databases

Línea de trabajo

Read apps/server/src/httpResponseErrorGuard.ts and inspect ProjectionStore.ts for the request guard and the migration-82 index dependency; then trace how startup migrations run and report progress. The issue suggests either making indexes run after readiness or in batches, or exposing progress in startup 503 responses, but notes a query fallback may be needed. Done means users can understand migration progress and the server’s startup behavior is addressed without breaking reads.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

via-triage
What happened

After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.

Diagnosis

#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:

  1. Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
  2. The 503 carries no information. rejectRequestsUntilServing (apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with 503, Retry-After: 1 and an empty body. The progress lines exist only in boot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress.
  3. Each release adds another expensive startup migration. On this build migration 86 (86_QueuedTurnInputIndex, a partial index on json_extract(payload_json, '$.inputIntent')) took as long as 82. #35421 only covered 82/83, and more full-table index builds keep shipping.
  4. The root cause keeps growing. statev2.sqlite grows without bound (#26591, #17845). A long-lived install hits this on every update that adds an index.

Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.

Steps to reproduce
  1. Standalone systemd service with a ~15 GB statev2.sqlite, on 0.1.8-nightly.20261108.4353.
  2. Update to 0.1.10-nightly.20261113.4616 (migrations 81–87 pending); service restarts.
  3. Open the app or curl -i http://127.0.0.1:3773/.well-known/t3/environment.
  4. Empty-body 503 for ~7 min; progress only in boot-service.log.
Version

Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.

Environment

Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.

Evidence
`boot-service.log` (the per-migration progress added by #35486):


[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText   1 of 7
[05:53:45.644] 82_TurnItemConversationIndex     2 of 7
[05:56:16.942] 83_CommandReadIndexes            3 of 7
[05:57:34.330] 84_LastExecutedRunIndex          4 of 7
[05:57:35.645] 85_TurnItemCounts                5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex          6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773



`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.
Related issues
  • #35421 / #35486: same symptom; fix removed the hang, not the blocking or missing progress.
  • #35313: desktop loading screen during migrations; web/standalone has no equivalent.
  • #23372: desktop windowless 8+ min during startup recovery.
  • #26591, #17845: unbounded statev2.sqlite growth.

Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.

Fix applied or workaround

None. Waited it out; nothing changed on the install or database.

Filed by

Claude Code (Claude Opus 5.5), via t3 triage.

Lenguaje dominante
TypeScript
Estrellas
24.8k
Forks
6.4k
Merge medio
59 min
PR fusionados (30 d)
418

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pingdotgg/t3code

Todos los issues de pingdotgg/t3code

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.