Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503

Aberta
  • Algo errado?
#35,995 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
node.js, sqlite, typescript
Domínio
backend, databases

Direção de pesquisa

Read apps/server/src/httpResponseErrorGuard.ts and inspect ProjectionStore.ts for the request guard and the migration-82 index dependency; then trace how startup migrations run and report progress. The issue suggests either making indexes run after readiness or in batches, or exposing progress in startup 503 responses, but notes a query fallback may be needed. Done means users can understand migration progress and the server’s startup behavior is addressed without breaking reads.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

via-triage
What happened

After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.

Diagnosis

#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:

  1. Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
  2. The 503 carries no information. rejectRequestsUntilServing (apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with 503, Retry-After: 1 and an empty body. The progress lines exist only in boot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress.
  3. Each release adds another expensive startup migration. On this build migration 86 (86_QueuedTurnInputIndex, a partial index on json_extract(payload_json, '$.inputIntent')) took as long as 82. #35421 only covered 82/83, and more full-table index builds keep shipping.
  4. The root cause keeps growing. statev2.sqlite grows without bound (#26591, #17845). A long-lived install hits this on every update that adds an index.

Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.

Steps to reproduce
  1. Standalone systemd service with a ~15 GB statev2.sqlite, on 0.1.8-nightly.20261108.4353.
  2. Update to 0.1.10-nightly.20261113.4616 (migrations 81–87 pending); service restarts.
  3. Open the app or curl -i http://127.0.0.1:3773/.well-known/t3/environment.
  4. Empty-body 503 for ~7 min; progress only in boot-service.log.
Version

Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.

Environment

Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.

Evidence
`boot-service.log` (the per-migration progress added by #35486):


[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText   1 of 7
[05:53:45.644] 82_TurnItemConversationIndex     2 of 7
[05:56:16.942] 83_CommandReadIndexes            3 of 7
[05:57:34.330] 84_LastExecutedRunIndex          4 of 7
[05:57:35.645] 85_TurnItemCounts                5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex          6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773



`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.
Related issues
  • #35421 / #35486: same symptom; fix removed the hang, not the blocking or missing progress.
  • #35313: desktop loading screen during migrations; web/standalone has no equivalent.
  • #23372: desktop windowless 8+ min during startup recovery.
  • #26591, #17845: unbounded statev2.sqlite growth.

Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.

Fix applied or workaround

None. Waited it out; nothing changed on the install or database.

Filed by

Claude Code (Claude Opus 5.5), via t3 triage.

Linguagem predominante
TypeScript
Estrelas
24.8k
Forks
6.4k
Merge médio
59min
PRs com merge (30d)
418

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de pingdotgg/t3code

Todas as issues de pingdotgg/t3code

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.