Recover compressed and archived Codex usage with a bounded preview

Aperta
  • Qualcosa non va?
#15,022 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
javascript, node.js, react
Ambito
backend, databases

Direzione di ricerca

Start with the decomposed issues #15029, #15030 and #15031, then read server/services/usageReconciler.js, usageInteractiveScan.js, usageBackfillWorker.js, usageBackfill.js and usage.js alongside the existing usage routes and Usage page controls. Trace the current readers, workers and ledger repair structures, and review the tests for duplicate sources, interval boundaries, interrupted scans and live completion overlap. Done means bounded previews report coverage and ambiguity, while explicit apply repairs only provable differences idempotently; the parent issue spans several coordinated changes and requires accounting compatibility expertise.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:devtools bug decomposed effort:high epic model:heavy plan

Problem

The Codex usage readers in server/services/usageReconciler.js and usageInteractiveScan.js enumerate active .jsonl rollouts only. Native Codex also retains older sessions as .jsonl.zst and moves sessions under archived_sessions. Read-only investigation on macOS verified standard Zstandard files and successful integrity checks; the files were not deleted. An empty raw-file date range is therefore not proof that source history is absent. Session filenames encode start dates, so older-started sessions can also contain events within the requested interval.

Historical backfill cannot currently recover those sources. Blindly adding their cumulative totals would be unsafe because earlier run accounting and standalone scans may already have recorded some of them.

Chosen approach

Extend the shared Codex source discovery/parser to stream raw and native Zstandard rollouts from active and archived locations, extracting only the existing accounting metadata. Identify a logical session independently of path/compression form and deduplicate raw/compressed or moved copies. Use bounded worker concurrency and a small incremental file inventory keyed by identity, size and modification time; do not decompress all history on every live completion.

Extend the existing explicit historical-reconciliation action with a date-bounded preview. It must show source coverage, provable ledger differences and ambiguous overlap separately. Apply only corrections whose prior attribution and replacement can be established, with existing idempotency conventions. Preserve existing counts where prior billing is ambiguous, rather than assuming missing source classifications mean zero usage. Keep malformed/unreadable/unsupported sources visible. Do not invoke providers or restore full plaintext archives.

This is distinct from #8846, which concerns PortOS-owned CoS artifact retention rather than reading native Codex rollouts.

Affected areas

server/services/usageReconciler.js, usageInteractiveScan.js, usageBackfillWorker.js, usageBackfill.js, usage.js, the existing usage routes and Usage page reconciliation controls. Reuse existing readers, workers and ledger repair structures rather than introducing a parallel accounting store.

Acceptance criteria

  • Synthetic raw, compressed, archived, moved and duplicate-copy sessions produce identical per-model counters exactly once.
  • A session starting outside a selected date range but emitting inside it is included correctly.
  • Malformed/truncated compressed input reports incomplete coverage and does not silently replace trustworthy ledger totals.
  • Preview performs no historical writes, provider calls or full-history synchronous work; ordinary completion does not acquire a full-history decompression bottleneck.
  • Apply is explicit, idempotent and limited to provable adjustments; unknown prior attribution remains unresolved and visible.
  • Tests cover interrupted scans, source growth and a live completion overlapping a preview/apply without double counting.
  • No migration automatically rewrites live history.

hardwareRequirements: []

Dispatch: model:heavy for source identity and accounting compatibility; effort:high for duplicate-source, interval and replay tests. Planner attribution is omitted because no verified planner-model identifier was supplied.

Decomposed into

  • #15029 — Preview bounded Codex usage from raw, compressed and archived sources
  • #15030 — Use shared Codex source discovery in live and historical accounting readers
  • #15031 — Add explicit bounded Codex reconciliation preview and provable ledger repair
Lingua principale
JavaScript
Stelle
38
Fork
32
Merge medio
35m
PR unite (30g)
989

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di atomantic/PortOS

Tutte le issue di atomantic/PortOS

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.