Assay tune silently skips every caller whose labels live in the LabelStore

Aperta
#2,869 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
bun, typescript

Direzione di ricerca

Start at LIFEOS/ASSAY/Report.ts:835 and trace how outcomeLabels at Report.ts:240 supplies labels for agreement; check the ledger shapes in LIFEOS/ASSAY/Registry.ts:24 and the label records in LIFEOS/TOOLS/lib/LabelStore.ts. Use the provided tune-probe.ts reproduction to confirm the empty result, then verify that demo.is_x gets a sweep and recommended cut, or that skipped questions explain why.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bun LIFEOS/ASSAY/Assay.ts tune --caller <name> prints nothing, and --json returns "questions": [] with exit 0, for every caller whose ground_truth rows carry no ledger. That is the shape both registration paths write: registryRow (LIFEOS/ASSAY/QuestionSystem.ts:2201) and registerQuestion (LIFEOS/ASSAY/Assay.ts:115) set { source, join_key } only. The result is the same when the label store holds principal right/wrong labels for that caller and question and the shadow log holds the engine's answers to the same turns. Nothing says the question was skipped, so the output reads as an empty window.

Where

LIFEOS/ASSAY/Report.ts:835

if (cfg.corrective[question] || !truth?.ledger) continue;
Repro

On main (94bd8f25), from LifeOS/install, save this as tune-probe.ts and run bun tune-probe.ts. It builds a throwaway fixture in the OS temp dir and points Assay at it through LIFEOS_GLANCE_CALLERS, LIFEOS_GLANCE_LOG and LIFEOS_LABEL_STORE.

import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const dir = mkdtempSync(join(tmpdir(), "tune-"));
const q = "is_x";
writeFileSync(join(dir, "callers.json"), JSON.stringify({
  demo: {
    mode: "shadow",
    ground_truth: { [q]: { source: "principal right/wrong labels in the label store", join_key: "session + turn" } },
    corrective: { [q]: false },
    fail: "open",
    budget: { calls_per_day: 100, tokens_per_day: 100000 },
    thresholds: { [q]: { value: 0.7, calibrated: null, model: null, samples: 0, agreement: null, balanced_accuracy: null, mode: "shadow" } },
    enforce_evidence: null,
  },
}));
const log: string[] = [], labels: string[] = [];
for (let i = 0; i < 40; i++) {
  const ts = new Date(Date.now() - i * 60_000).toISOString();
  log.push(JSON.stringify({ ts, caller: "demo", session: "s", turn: String(i), model: "m", answers: { [q]: { type: "noul", noul: i % 2 ? 0.9 : 0.2 } } }));
  labels.push(JSON.stringify({ v: 1, id: `l${i}`, ts, source: "principal", kind: "check", caller: "demo", question: q, verdict: i % 5 ? "right" : "wrong", truth: i % 2 ? "pass" : "fail", by: "principal", engine: "m" }));
}
writeFileSync(join(dir, "assay.jsonl"), log.join("\n") + "\n");
writeFileSync(join(dir, "labels.jsonl"), labels.join("\n") + "\n");

const env = { ...process.env, LIFEOS_GLANCE_CALLERS: join(dir, "callers.json"), LIFEOS_GLANCE_LOG: join(dir, "assay.jsonl"), LIFEOS_LABEL_STORE: join(dir, "labels.jsonl") };
const tune = (...a: string[]) => Bun.spawnSync(["bun", "LIFEOS/ASSAY/Assay.ts", "tune", "--caller", "demo", ...a], { env });
const text = tune(), json = tune("--json");
rmSync(dir, { recursive: true });
console.log(`tune (text): [${text.stdout.toString().trim()}] exit ${text.exitCode}`);
console.log(`tune --json: ${json.stdout.toString().trim()} exit ${json.exitCode}`);

Output:

tune (text): [] exit 0
tune --json: {"caller":"demo","days":7,"questions":[]} exit 0

Expected: a sweep and a recommended cut for demo.is_x, or a line saying why the question was skipped. The empty text run also shows the caller loaded, since an unregistered caller prints "not registered".

Fix

tune takes ground truth only from the four comparison ledgers GroundTruth.ledger enumerates (LIFEOS/ASSAY/Registry.ts:24, written by the old engine-vs-word-list comparison) and never from the LabelStore that calls itself the one place a human verdict lands (LIFEOS/TOOLS/lib/LabelStore.ts:2), so its input took the shape of its first use instead of staying somewhat general-purpose (Ousterhout, A Philosophy of Software Design, ch. 6), and the bare continue at Report.ts:835 skips the question silently where it should fail loudly. Reading the principal's labels through outcomeLabels (Report.ts:240, already used for agreement) whenever a question has no ledger, and printing why any question is skipped, lets tune cut every store-labelled caller from the principal's own labels, and an empty result can no longer pass for an empty window.

Lingua principale
TypeScript
Stelle
19.9k
Fork
2.5k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di danielmiessler/LifeOS

Tutte le issue di danielmiessler/LifeOS

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.