Assay tune silently skips every caller whose labels live in the LabelStore
Los mantenedores suelen responder en 5 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 54/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- bun, typescript
- Área
- backend-api-design, data
Línea de trabajo
Start at LIFEOS/ASSAY/Report.ts:835 and trace how outcomeLabels at Report.ts:240 supplies labels for agreement; check the ledger shapes in LIFEOS/ASSAY/Registry.ts:24 and the label records in LIFEOS/TOOLS/lib/LabelStore.ts. Use the provided tune-probe.ts reproduction to confirm the empty result, then verify that demo.is_x gets a sweep and recommended cut, or that skipped questions explain why.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
bun LIFEOS/ASSAY/Assay.ts tune --caller <name> prints nothing, and --json returns "questions": [] with exit 0, for every caller whose ground_truth rows carry no ledger. That is the shape both registration paths write: registryRow (LIFEOS/ASSAY/QuestionSystem.ts:2201) and registerQuestion (LIFEOS/ASSAY/Assay.ts:115) set { source, join_key } only. The result is the same when the label store holds principal right/wrong labels for that caller and question and the shadow log holds the engine's answers to the same turns. Nothing says the question was skipped, so the output reads as an empty window.
Where
LIFEOS/ASSAY/Report.ts:835
if (cfg.corrective[question] || !truth?.ledger) continue;
Repro
On main (94bd8f25), from LifeOS/install, save this as tune-probe.ts and run bun tune-probe.ts. It builds a throwaway fixture in the OS temp dir and points Assay at it through LIFEOS_GLANCE_CALLERS, LIFEOS_GLANCE_LOG and LIFEOS_LABEL_STORE.
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
const dir = mkdtempSync(join(tmpdir(), "tune-"));
const q = "is_x";
writeFileSync(join(dir, "callers.json"), JSON.stringify({
demo: {
mode: "shadow",
ground_truth: { [q]: { source: "principal right/wrong labels in the label store", join_key: "session + turn" } },
corrective: { [q]: false },
fail: "open",
budget: { calls_per_day: 100, tokens_per_day: 100000 },
thresholds: { [q]: { value: 0.7, calibrated: null, model: null, samples: 0, agreement: null, balanced_accuracy: null, mode: "shadow" } },
enforce_evidence: null,
},
}));
const log: string[] = [], labels: string[] = [];
for (let i = 0; i < 40; i++) {
const ts = new Date(Date.now() - i * 60_000).toISOString();
log.push(JSON.stringify({ ts, caller: "demo", session: "s", turn: String(i), model: "m", answers: { [q]: { type: "noul", noul: i % 2 ? 0.9 : 0.2 } } }));
labels.push(JSON.stringify({ v: 1, id: `l${i}`, ts, source: "principal", kind: "check", caller: "demo", question: q, verdict: i % 5 ? "right" : "wrong", truth: i % 2 ? "pass" : "fail", by: "principal", engine: "m" }));
}
writeFileSync(join(dir, "assay.jsonl"), log.join("\n") + "\n");
writeFileSync(join(dir, "labels.jsonl"), labels.join("\n") + "\n");
const env = { ...process.env, LIFEOS_GLANCE_CALLERS: join(dir, "callers.json"), LIFEOS_GLANCE_LOG: join(dir, "assay.jsonl"), LIFEOS_LABEL_STORE: join(dir, "labels.jsonl") };
const tune = (...a: string[]) => Bun.spawnSync(["bun", "LIFEOS/ASSAY/Assay.ts", "tune", "--caller", "demo", ...a], { env });
const text = tune(), json = tune("--json");
rmSync(dir, { recursive: true });
console.log(`tune (text): [${text.stdout.toString().trim()}] exit ${text.exitCode}`);
console.log(`tune --json: ${json.stdout.toString().trim()} exit ${json.exitCode}`);
Output:
tune (text): [] exit 0
tune --json: {"caller":"demo","days":7,"questions":[]} exit 0
Expected: a sweep and a recommended cut for demo.is_x, or a line saying why the question was skipped. The empty text run also shows the caller loaded, since an unregistered caller prints "not registered".
Fix
tune takes ground truth only from the four comparison ledgers GroundTruth.ledger enumerates (LIFEOS/ASSAY/Registry.ts:24, written by the old engine-vs-word-list comparison) and never from the LabelStore that calls itself the one place a human verdict lands (LIFEOS/TOOLS/lib/LabelStore.ts:2), so its input took the shape of its first use instead of staying somewhat general-purpose (Ousterhout, A Philosophy of Software Design, ch. 6), and the bare continue at Report.ts:835 skips the question silently where it should fail loudly. Reading the principal's labels through outcomeLabels (Report.ts:240, already used for agreement) whenever a question has no ledger, and printing why any question is skipped, lets tune cut every store-labelled caller from the principal's own labels, and an empty result can no longer pass for an empty window.
- Lenguaje dominante
- TypeScript
- Estrellas
- 19.9k
- Forks
- 2.5k
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de danielmiessler/LifeOS
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 67/100
danielmiessler/LifeOS#2894 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
danielmiessler/LifeOS#2891 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
danielmiessler/LifeOS#2888 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
danielmiessler/LifeOS#2886 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 68/100
danielmiessler/LifeOS#2848 ·
Los mantenedores suelen responder en 5 días
Todos los issues de danielmiessler/LifeOS
Issues similares
-
CLI OAuth login fails behind an authenticating proxy: GET /api/v1/user omits Proxy-AuthorizationAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 85/100
lightdash/lightdash#33086 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[功能请求] 设置选项语言调整Abierto功能请求
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
antithesishq/bombadil#473 ·
Los mantenedores suelen responder en 2 días
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
ardia-browser/ardia#3291 ·
Los mantenedores suelen responder en 1 día
-
The structured output reference promises null results for refusals that carry validation detailsAbiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día