Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug] routing-history index rebuild runs synchronously on the serving event loop (487 s proxy-wide freeze)

Open
  • Is something wrong?
#6,990 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@lidge-jun is already working on this.

Since Oct 12, 2026.

  • #6999 by @lidge-jun — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
bun, sqlite, typescript
Domain
backend, performance

Research direction

Start with src/routing/health.ts:236 and trace the sync and async entry points into refreshLockedSync() and fullRebuild() in the request-history index. Check how rebuilds and PRAGMA quick_check affect the serving event loop, then run the relevant existing tests. Done means large index rebuilds no longer freeze requests and the specific rebuild reason remains visible; the issue does not name test files or show a patch.

Written by the indexing model from the issue text.

Description

bug proxy
Client or integration

Codex App

Area

Proxy and routing

Summary

The routing-history index rebuild runs synchronously on the serving event loop. On a large ledger one rebuild froze the whole proxy for 487 s: no request on any provider completed and Codex clients showed "Connection failed: error sending request". I expected the rebuild to run in the background without blocking requests.

One call into the request-history index froze the proxy for 487 s. No request completed on any provider.
Codex clients then showed "Connection failed: error sending request". The ledger usage.jsonl was 2.34 GB (1.12 M rows).
The index file routing-history.sqlite was rebuilt to 4.93 GB.

Reproduction
  1. Let usage.jsonl grow to about 2 GB (no usageLedgerMaxBytes).
  2. Trigger a schema/identity mismatch or delete routing-history.sqlite, then send any request (routing calls openRequestHistoryIndexSync() from src/routing/health.ts:236).
  3. Observe: the event loop is blocked for the whole rebuild (scratch measurements below).
Version

2.83.0 (same code in 2.81.0)

Operating system

macOS (Bun 1.4.2)

Provider and model

All providers (proxy-wide)

Logs or error output
rebuilt after identity/schema mismatch
routing-history.sqlite rebuilt to 4.93 GB from usage.jsonl 2.34 GB (1,119,276 rows); proxy unresponsive 487 s
Source-level analysis (v2.83.0)
  1. refreshLockedSync() (line 432) is synchronous. ensureSchemaAndIdentity() may call destroyAndRecreate(), then
    fullRebuild() (line 412) reads the whole ledger and inserts every row in the same call stack. No await, no time slice.
  2. The sync entry openRequestHistoryIndexSync() is called at routing time from src/routing/health.ts:236.
    The async entry openRequestHistoryIndex() wraps the same sync function in Promise.resolve(...), so it blocks too.
  3. isHealthy() runs PRAGMA quick_check on every refresh (line 394). On a 4.8 GB file it takes 2.3 s per call, and 12 s on a cold cache.
  4. Line 438 overwrites the specific reason that destroyAndRecreate() stored (schema version, integrity, identity, unreadable)
    with the text "rebuilt after identity/schema mismatch". The operator cannot tell which check fired.
Ledger Rows Cold rebuild, event loop blocked quick_check per warm call
490 MB 228 k 17 s 0.54 s
1.1 GB 511 k 45 s 1.0 s
2.4 GB 1.09 M 92 s 2.3 s

Rebuild time grows faster than the row count (random primary-key inserts). The live freeze took 487 s on a loaded machine.

Proposed change
  • Make the ingest a generator that yields after each read chunk. Drive it with a time slice (25 ms) and setImmediate for async callers.
  • Give the sync entry a small budget (50 ms). When work remains, hand the generator to a background flight and throw a typed "busy" error.
    health.ts already treats a throw as "evidence unknown".
  • Build a full rebuild in a private sidecar file (routing-history.sqlite.building.<pid>) and publish it with one atomic rename.
    No reader, including another process, sees a partial index, and no table is dropped under a reader.
  • Commit rows, indexed_rows and indexed_offset in the same transaction for each batch, so an aborted ingest resumes exactly.
  • Run no quick_check on the refresh path. Flag the index when any statement returns a corruption error and rebuild on the next refresh.
  • Keep the specific rebuild reason.
  • Keep one refresh in flight across closeRequestHistoryIndex().

Result on the 2.4 GB ledger: the sync call returns in 0.1 s, the largest event-loop stall is 0.2 s, the build takes 98 s in the background,
and a warm call costs 0 ms. A reference patch and 25 bun tests are available on request.

Also consider a default for usageLedgerMaxBytes. With no limit the ledger and the index grow without bound, and every rebuild is proportional to that size.

Checks
  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.
Dominant language
TypeScript
Stars
16.9k
Forks
1.3k
Avg merge
5h 22m
Merged PRs (30d)
627

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lidge-jun/opencodex

All issues in lidge-jun/opencodex

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.