Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503

Đang mở
  • Có gì không ổn?
#35,995 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, sqlite, typescript
Lĩnh vực
backend, databases

Hướng nghiên cứu

Read apps/server/src/httpResponseErrorGuard.ts and inspect ProjectionStore.ts for the request guard and the migration-82 index dependency; then trace how startup migrations run and report progress. The issue suggests either making indexes run after readiness or in batches, or exposing progress in startup 503 responses, but notes a query fallback may be needed. Done means users can understand migration progress and the server’s startup behavior is addressed without breaking reads.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

via-triage
What happened

After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.

Diagnosis

#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:

  1. Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
  2. The 503 carries no information. rejectRequestsUntilServing (apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with 503, Retry-After: 1 and an empty body. The progress lines exist only in boot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress.
  3. Each release adds another expensive startup migration. On this build migration 86 (86_QueuedTurnInputIndex, a partial index on json_extract(payload_json, '$.inputIntent')) took as long as 82. #35421 only covered 82/83, and more full-table index builds keep shipping.
  4. The root cause keeps growing. statev2.sqlite grows without bound (#26591, #17845). A long-lived install hits this on every update that adds an index.

Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.

Steps to reproduce
  1. Standalone systemd service with a ~15 GB statev2.sqlite, on 0.1.8-nightly.20261108.4353.
  2. Update to 0.1.10-nightly.20261113.4616 (migrations 81–87 pending); service restarts.
  3. Open the app or curl -i http://127.0.0.1:3773/.well-known/t3/environment.
  4. Empty-body 503 for ~7 min; progress only in boot-service.log.
Version

Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.

Environment

Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.

Evidence
`boot-service.log` (the per-migration progress added by #35486):


[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText   1 of 7
[05:53:45.644] 82_TurnItemConversationIndex     2 of 7
[05:56:16.942] 83_CommandReadIndexes            3 of 7
[05:57:34.330] 84_LastExecutedRunIndex          4 of 7
[05:57:35.645] 85_TurnItemCounts                5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex          6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773



`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.
Related issues
  • #35421 / #35486: same symptom; fix removed the hang, not the blocking or missing progress.
  • #35313: desktop loading screen during migrations; web/standalone has no equivalent.
  • #23372: desktop windowless 8+ min during startup recovery.
  • #26591, #17845: unbounded statev2.sqlite growth.

Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.

Fix applied or workaround

None. Waited it out; nothing changed on the install or database.

Filed by

Claude Code (Claude Opus 5.5), via t3 triage.

Ngôn ngữ chính
TypeScript
Star
24.8k
Fork
6.4k
Merge trung bình
59 phút
Pull request đã merge (30 ngày)
418

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pingdotgg/t3code

Tất cả issue của pingdotgg/t3code

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.