Plan B Task 5 follow-ups: measurement driver polish + matplotlib collection caveat

Aperta
  • Qualcosa non va?
#2 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
matplotlib, python, shell
Ambito
devops, performance, testing-qa

Direzione di ricerca

Read the four follow-ups and inspect deploy/knative/measure-swebench-runtimes.sh and scripts/gen_swebench_deck.py; identify which item you plan to address before changing code. For the driver items, check the existing CI shellcheck coverage and how samples are aggregated; for the deck item, note that the list branch is currently unexercised. Done means addressing one follow-up without changing the stated default behavior or gold test command.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up items from Plan B Task 5 (#129, SWE-bench runtime measurement). All non-blocking; documented for later.

1. MEASURE_SAMPLES>1 status-label aggregation (deploy/knative/measure-swebench-runtimes.sh)

With more than one sample per instance, the per-sample status label reflects only the last sample, and median could mix a real runtime with the timeout-cap value. Purely cosmetic at the default MEASURE_SAMPLES=1 (the only exercised path). If multi-sample is ever used, treat any timeout sample as dominating and aggregate ms only within a single status class.

2. CI shellcheck for the measurement driver

Add deploy/knative/measure-swebench-runtimes.sh to the CI shellcheck coverage (it was shellcheck-clean at authoring, but isn't gated in CI).

3. matplotlib collection-as-error under newer setuptools

matplotlib__matplotlib-24177's measured runtime is import+collection only: its pytest collection errors on a DeprecationWarning-as-error at base_commit (newer setuptools in the baked env), so 0 tests are collected. The recorded time is faithful to the gold command but undervalues the instance. If a truer matplotlib number is wanted, relax the warning filter for the measurement run (e.g. PYTHONWARNINGS=default) — note this deviates from the exact gold test_cmd.

4. test_cmd-as-list defensive branch (scripts/gen_swebench_deck.py)

The list→string join branch is unexercised with swebench 4.1.0 (returns str); kept for forward-compat. Add a unit case if a future swebench version returns a list.

Assisted-By: Claude Code

Lingua principale
Shell
Stelle
0
Fork
1
Merge medio
6h 16m
PR unite (30g)
7

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di rossoctl/moca-experiments

Tutte le issue di rossoctl/moca-experiments

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.