Assay tune silently skips every caller whose labels live in the LabelStore

Abierto
#2,869 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 5 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
54/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
bun, typescript

Línea de trabajo

Start at LIFEOS/ASSAY/Report.ts:835 and trace how outcomeLabels at Report.ts:240 supplies labels for agreement; check the ledger shapes in LIFEOS/ASSAY/Registry.ts:24 and the label records in LIFEOS/TOOLS/lib/LabelStore.ts. Use the provided tune-probe.ts reproduction to confirm the empty result, then verify that demo.is_x gets a sweep and recommended cut, or that skipped questions explain why.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bun LIFEOS/ASSAY/Assay.ts tune --caller <name> prints nothing, and --json returns "questions": [] with exit 0, for every caller whose ground_truth rows carry no ledger. That is the shape both registration paths write: registryRow (LIFEOS/ASSAY/QuestionSystem.ts:2201) and registerQuestion (LIFEOS/ASSAY/Assay.ts:115) set { source, join_key } only. The result is the same when the label store holds principal right/wrong labels for that caller and question and the shadow log holds the engine's answers to the same turns. Nothing says the question was skipped, so the output reads as an empty window.

Where

LIFEOS/ASSAY/Report.ts:835

if (cfg.corrective[question] || !truth?.ledger) continue;
Repro

On main (94bd8f25), from LifeOS/install, save this as tune-probe.ts and run bun tune-probe.ts. It builds a throwaway fixture in the OS temp dir and points Assay at it through LIFEOS_GLANCE_CALLERS, LIFEOS_GLANCE_LOG and LIFEOS_LABEL_STORE.

import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const dir = mkdtempSync(join(tmpdir(), "tune-"));
const q = "is_x";
writeFileSync(join(dir, "callers.json"), JSON.stringify({
  demo: {
    mode: "shadow",
    ground_truth: { [q]: { source: "principal right/wrong labels in the label store", join_key: "session + turn" } },
    corrective: { [q]: false },
    fail: "open",
    budget: { calls_per_day: 100, tokens_per_day: 100000 },
    thresholds: { [q]: { value: 0.7, calibrated: null, model: null, samples: 0, agreement: null, balanced_accuracy: null, mode: "shadow" } },
    enforce_evidence: null,
  },
}));
const log: string[] = [], labels: string[] = [];
for (let i = 0; i < 40; i++) {
  const ts = new Date(Date.now() - i * 60_000).toISOString();
  log.push(JSON.stringify({ ts, caller: "demo", session: "s", turn: String(i), model: "m", answers: { [q]: { type: "noul", noul: i % 2 ? 0.9 : 0.2 } } }));
  labels.push(JSON.stringify({ v: 1, id: `l${i}`, ts, source: "principal", kind: "check", caller: "demo", question: q, verdict: i % 5 ? "right" : "wrong", truth: i % 2 ? "pass" : "fail", by: "principal", engine: "m" }));
}
writeFileSync(join(dir, "assay.jsonl"), log.join("\n") + "\n");
writeFileSync(join(dir, "labels.jsonl"), labels.join("\n") + "\n");

const env = { ...process.env, LIFEOS_GLANCE_CALLERS: join(dir, "callers.json"), LIFEOS_GLANCE_LOG: join(dir, "assay.jsonl"), LIFEOS_LABEL_STORE: join(dir, "labels.jsonl") };
const tune = (...a: string[]) => Bun.spawnSync(["bun", "LIFEOS/ASSAY/Assay.ts", "tune", "--caller", "demo", ...a], { env });
const text = tune(), json = tune("--json");
rmSync(dir, { recursive: true });
console.log(`tune (text): [${text.stdout.toString().trim()}] exit ${text.exitCode}`);
console.log(`tune --json: ${json.stdout.toString().trim()} exit ${json.exitCode}`);

Output:

tune (text): [] exit 0
tune --json: {"caller":"demo","days":7,"questions":[]} exit 0

Expected: a sweep and a recommended cut for demo.is_x, or a line saying why the question was skipped. The empty text run also shows the caller loaded, since an unregistered caller prints "not registered".

Fix

tune takes ground truth only from the four comparison ledgers GroundTruth.ledger enumerates (LIFEOS/ASSAY/Registry.ts:24, written by the old engine-vs-word-list comparison) and never from the LabelStore that calls itself the one place a human verdict lands (LIFEOS/TOOLS/lib/LabelStore.ts:2), so its input took the shape of its first use instead of staying somewhat general-purpose (Ousterhout, A Philosophy of Software Design, ch. 6), and the bare continue at Report.ts:835 skips the question silently where it should fail loudly. Reading the principal's labels through outcomeLabels (Report.ts:240, already used for agreement) whenever a question has no ledger, and printing why any question is skipped, lets tune cut every store-labelled caller from the principal's own labels, and an empty result can no longer pass for an empty window.

Lenguaje dominante
TypeScript
Estrellas
19.9k
Forks
2.5k
Métricas de merge de PR
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de danielmiessler/LifeOS

Todos los issues de danielmiessler/LifeOS

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.