Codex cost scan never completes when the 256 MiB per-file cap falls inside a multi-MB `compacted` line; Codex model breakdown unavailable

Open
#5,132 5 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
sqlite, swift
Domain
backend, databases

Research direction

Start by tracing the Codex cost scanner’s per-file byte limit, resume state handling, and SQLite persistence; the issue does not name source files or tests. Reproduce the scan with a large compacted record crossing the 256 MiB boundary, then check whether later refreshes advance the saved offset and complete the file. Done means the scan progresses or the report provides partial data with a coverage warning.

Written by the indexing model from the issue text.

Description

clawsweeper:needs-info clawsweeper:needs-maintainer-review clawsweeper:no-new-fix-pr impact:ux-friction issue-rating: 🦐 gold shrimp P2
Environment
  • CodexBar 0.84.0 (177), installed via Homebrew cask
  • Codex CLI 0.170.0
  • macOS 27.0.1, Apple Silicon
  • Local token cost usage enabled, refresh every 5 minutes
Summary

One active Codex rollout is about 772 MB (809,825,875 bytes). The bounded scanner reads exactly maxFileBytes (268,435,456) of it and then never makes progress. While this file stays incomplete, every Codex cost refresh fails, so the Codex model and cost breakdown never appears. Claude and Antigravity cost data load normally.

Observed behavior

The app logs this about every 5 minutes, for at least 6 hours in a row:

cost usage failed provider=codex duration=1.3s error=Local token history is unavailable or incomplete.

Each pass reads the same number of bytes, so it is not resuming:

Codex cost scan applied work limits (bytesConsumed=412369328 deferredByBudget=0 deferredByTime=0 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=2)
Codex cost scan applied work limits (bytesConsumed=421551265 deferredByBudget=0 deferredByTime=0 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=2)

These two lines repeat with identical values.

This error also repeats several times per refresh, about 500 times in 6 hours:

cost-usage store operation failed; keeping database: sqlite(10)

There is 64 GB of free disk space.

Cache state (cost-usage.sqlite)
  • scan_metadata: catchUpPending = true, completedFiles = 86, totalFiles = 88
  • The stuck row in files: size = 809825875, parsed_bytes = 268435456, anchor_indexed_bytes = 268435456, scan_complete = 0
  • Its scan_state is about 2.5 MB. The decoded resumePayload holds a truncated partial line:
    • offset = 268435456
    • lineStartOffset = 267753485
    • lineBytes = 681971
    • truncated = true
    • jsonTailState.insideString = true, containerDepth = 6
  • The token snapshots for this file cover only the first week of the session. Nothing after the cut-off is counted.
Likely trigger

The line containing the 256 MiB boundary is one "type":"compacted" record (with a replacement_history payload) that is about 5.0 MB long. It starts about 0.68 MB before the cap and ends about 4.3 MB after it. No line in the file is larger than 20 MB, and the lines around the boundary are ordinary.

My guess is one of these:

  1. Resuming in the middle of a truncated JSON line fails or is thrown away on each pass, so the file restarts from the same resume point every time.
  2. The large resume state (scan_state about 2.5 MB, detailsPayload about 2 MB) fails to save. The repeating sqlite(10) errors fit this. Then the next pass starts from the old state.
Expected behavior
  • The scanner should get past the boundary line on a later pass and finish the file.
  • Failing that, the app should show partial Codex data with a coverage warning. It should not fail the whole Codex cost report.
Related
  • #2815: a large active rollout stays incomplete
  • #3203: deferred large files starve
  • #4140: a stalled catch-up blocks model breakdowns

These are closed. This looks like a remaining case where the 256 MiB cut-off splits a large line.

I can share more cache fields or a byte-range description of the boundary line on request. I can't share the session contents.

Dominant language
Swift
Stars
23.3k
Forks
2.2k
Avg merge
1d 18h
Merged PRs (30d)
231

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from steipete/CodexBar

All issues in steipete/CodexBar

Similar issues

More Swift issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.