Codex cost scan never completes when the 256 MiB per-file cap falls inside a multi-MB `compacted` line; Codex model breakdown unavailable

Aperta
#5,132 5 commenti 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
sqlite, swift
Ambito
backend, databases

Direzione di ricerca

Start by tracing the Codex cost scanner’s per-file byte limit, resume state handling, and SQLite persistence; the issue does not name source files or tests. Reproduce the scan with a large compacted record crossing the 256 MiB boundary, then check whether later refreshes advance the saved offset and complete the file. Done means the scan progresses or the report provides partial data with a coverage warning.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

clawsweeper:needs-info clawsweeper:needs-maintainer-review clawsweeper:no-new-fix-pr impact:ux-friction issue-rating: 🦐 gold shrimp P2
Environment
  • CodexBar 0.84.0 (177), installed via Homebrew cask
  • Codex CLI 0.170.0
  • macOS 27.0.1, Apple Silicon
  • Local token cost usage enabled, refresh every 5 minutes
Summary

One active Codex rollout is about 772 MB (809,825,875 bytes). The bounded scanner reads exactly maxFileBytes (268,435,456) of it and then never makes progress. While this file stays incomplete, every Codex cost refresh fails, so the Codex model and cost breakdown never appears. Claude and Antigravity cost data load normally.

Observed behavior

The app logs this about every 5 minutes, for at least 6 hours in a row:

cost usage failed provider=codex duration=1.3s error=Local token history is unavailable or incomplete.

Each pass reads the same number of bytes, so it is not resuming:

Codex cost scan applied work limits (bytesConsumed=412369328 deferredByBudget=0 deferredByTime=0 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=2)
Codex cost scan applied work limits (bytesConsumed=421551265 deferredByBudget=0 deferredByTime=0 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=2)

These two lines repeat with identical values.

This error also repeats several times per refresh, about 500 times in 6 hours:

cost-usage store operation failed; keeping database: sqlite(10)

There is 64 GB of free disk space.

Cache state (cost-usage.sqlite)
  • scan_metadata: catchUpPending = true, completedFiles = 86, totalFiles = 88
  • The stuck row in files: size = 809825875, parsed_bytes = 268435456, anchor_indexed_bytes = 268435456, scan_complete = 0
  • Its scan_state is about 2.5 MB. The decoded resumePayload holds a truncated partial line:
    • offset = 268435456
    • lineStartOffset = 267753485
    • lineBytes = 681971
    • truncated = true
    • jsonTailState.insideString = true, containerDepth = 6
  • The token snapshots for this file cover only the first week of the session. Nothing after the cut-off is counted.
Likely trigger

The line containing the 256 MiB boundary is one "type":"compacted" record (with a replacement_history payload) that is about 5.0 MB long. It starts about 0.68 MB before the cap and ends about 4.3 MB after it. No line in the file is larger than 20 MB, and the lines around the boundary are ordinary.

My guess is one of these:

  1. Resuming in the middle of a truncated JSON line fails or is thrown away on each pass, so the file restarts from the same resume point every time.
  2. The large resume state (scan_state about 2.5 MB, detailsPayload about 2 MB) fails to save. The repeating sqlite(10) errors fit this. Then the next pass starts from the old state.
Expected behavior
  • The scanner should get past the boundary line on a later pass and finish the file.
  • Failing that, the app should show partial Codex data with a coverage warning. It should not fail the whole Codex cost report.
Related
  • #2815: a large active rollout stays incomplete
  • #3203: deferred large files starve
  • #4140: a stalled catch-up blocks model breakdowns

These are closed. This looks like a remaining case where the 256 MiB cut-off splits a large line.

I can share more cache fields or a byte-range description of the boundary line on request. I can't share the session contents.

Lingua principale
Swift
Stelle
23.3k
Fork
2.2k
Merge medio
1g 18h
PR unite (30g)
231

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di steipete/CodexBar

Tutte le issue di steipete/CodexBar

Issue simili

Altre issue su Swift

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.