Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Same unconfined `predictions/<item_id>` path pattern remains in four benchmark rollouts

Aperta
  • Qualcosa non va?
#306 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

@adongwanai ci sta già lavorando.

Dal 8/10/2026.

  • #309 di @adongwanai — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
72/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
security

Direzione di ricerca

Start with the merged reference implementation in skillopt/envs/spreadsheetbench/rollout.py (_is_safe_task_id(), _confined_task_out_dir()), then decide whether to lift them into a shared helper. Apply the pattern to the listed call sites in skillopt/envs/docvqa/rollout.py:173,:232, livemathematicianbench/rollout.py:144, searchqa/rollout.py:204, and officeqa/rollout.py:530, keeping in mind livemathematicianbench's colon-shaped ids need an id-to-safe-dirname mapping rather than validation-and-reject. Any mapping must be applied on the read side too: skillopt/optimizer/slow_update.py:113 reads predictions/<task_id>/conversation.json. Done when ids like ../../x cannot escape out_root and existing rollouts plus the reader still resolve their own outputs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up from #264 (now merged). That PR validated the SpreadsheetBench task identifier and confined its persistent output directory. The same unconfined pattern remains in four other benchmark rollouts:

  • skillopt/envs/docvqa/rollout.py:173 and :232
  • skillopt/envs/livemathematicianbench/rollout.py:144
  • skillopt/envs/searchqa/rollout.py:204
  • skillopt/envs/officeqa/rollout.py:530

Each is os.path.join(out_root, "predictions", item_id), where item_id comes straight from the dataset item — no validation of the identifier and no containment check on the destination. An id such as ../../x therefore derives a destination outside out_root and processing continues into the model / code-execution path. skillopt/envs/spreadsheetbench/rollout.py now has _is_safe_task_id() and _confined_task_out_dir(), which could be lifted into a shared helper.

Two caveats are why this is filed rather than sent as a direct port:

  1. A charset validator is not sufficient for livemathematicianbench. Its ids are colon-shaped (202602:12), so all 177 ids across the shipped splits would be rejected by the SpreadsheetBench rule. Those runs need confinement by construction (map an id to a safe directory name) rather than validate-and-reject. Separately, an id containing : cannot be a Windows directory name at all, so those runs are already broken on Windows for an unrelated reason.

  2. Renaming the directory is not free. Readers look the task up by its raw id — skillopt/optimizer/slow_update.py:113 reads predictions/<task_id>/conversation.json — so any id-to-dirname mapping has to be applied on the read side too.

docvqa (63180), searchqa (hex) and officeqa (UID0003) use ids that are already safe shapes, so for those three the SpreadsheetBench approach can be applied as-is.

Filed separately because it is outside #264's reviewed scope.

Lingua principale
Python
Stelle
18.2k
Fork
1.7k
Merge medio
6g 18h
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/SkillOpt

Tutte le issue di microsoft/SkillOpt

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.