Fix behavioral eval cases that fail on unrelated PRs

Aperta
  • Qualcosa non va?
#2,413 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
ai, testing

Direzione di ricerca

Start with the four named eval cases and policies/evals.md, then inspect the eval runner’s 60s idle timeout and relevant product behavior. For each case, determine whether the product, eval expectation, or timeout is at fault and make the corresponding justified change; done means all four cases are addressed without skipping or removing any, and the eval summary check reflects the fixes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Some behavioral eval cases fail or error on most PRs, including PRs that do not touch product code. The evals summary check goes red when any case fails or errors, so these cases make the check red on almost every PR. Reviewers then learn to ignore it.

Seen on #2371, #2374, #2376, #2377 and #2410:

  • evals/google/scheduling.eval.ts: "when a recurring slot is for two people and the requester's afternoons are holds, check both and propose a time". It errors with STACK_TRACE_ERROR on almost every run, and sometimes hits the 60s idle timeout. #2375 fixed a related findMeetingTimes bug, but not this error.
  • evals/agent/delegation.eval.ts: "when a request splits into independent research tracks, start a child agent for each track". It often hits the 60s "agent not idle" timeout.
  • evals/google/scheduling.eval.ts: "when the requester corrects a one-sided answer, check the other attendee without asking again". It sometimes hits the 60s timeout.
  • evals/coding/subscriptions.eval.ts: "looks up and subscribes to an exact deployment before GitHub creates it". It fails on some runs.

For each case, find out whether:

  • the product has a bug (fix the product),
  • the eval checks wording or needs the model to do something wrong (fix the eval, following policies/evals.md; do not rewrite the user's message to make it pass), or
  • the 60s idle limit is too short for a task that needs several tool calls (change the limit for that case, with a reason).

Do not skip or remove a case only to make the check green.

via David Cramer.

--

View Session [Sentry]

Lingua principale
TypeScript
Stelle
385
Fork
42
Merge medio
10h 4m
PR unite (30g)
361

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di getsentry/junior

Tutte le issue di getsentry/junior

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.