Question: independent Slai Agent Runtime evaluation and leaderboard submission

Aperta
  • Qualcosa non va?
#30 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 4 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
1/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
25/100
Tipo di issue
Documentazione
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
docker, kubernetes, python
Ambito
machine-learning

Direzione di ricerca

Read the EdgeBench documentation for leaderboard eligibility, result comparability, custom agent scaffolds, and submission channels; the issue does not name specific files or tests. Clarify the workflow for independent runs on the public task subset and the recommended path for private suite access. Done means the documentation answers the five questions raised here.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Thank you for releasing EdgeBench and SForge.

Slai Agent Runtime can provision and operate an evaluation environment from a Windows host, including Docker, Python, task data, job configuration, execution, and preservation of reproducibility artifacts. Task execution would use the documented Linux Docker, E2B, or Kubernetes environment.

We have participated in Terminal-Bench 2.1 and newer Terminal-Bench test sets, with 155 published Harbor job records:

https://hub.harborframework.com/users/huong8373tt-beep/jobs

The documentation already makes clear that third-party models and custom SForge agents are supported. Before investing in a full EdgeBench run, could you please clarify the reporting and leaderboard workflow for an independently operated run?

  1. Are independently run results on the public 51-task subset eligible for consideration on the EdgeBench leaderboard or project website?

  2. If so, what evidence is required for a result to be considered comparable? For example: the exact SForge and task-definition revisions, official experiment YAML, agent/scaffold source and revision, model and endpoint configuration, hardware/backend details, run logs, trajectories, and per-task result artifacts.

  3. Is a custom Slai Agent Runtime scaffold acceptable for a reported result, provided that it uses the official task definitions and leaderboard settings unchanged apart from the documented model/environment fields?

  4. Is there a preferred submission channel for an independently run result? For example, a GitHub issue, pull request, form, or direct email.

  5. For the private 134-task suite, should an independent team first demonstrate a reproducible public-subset run before contacting the project by email?

We will clearly label any independently operated result as third-party and will not present it as an official comparable leaderboard entry without confirmation.

Thank you.

Lingua principale
Python
Stelle
471
Fork
20
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ByteDance-Seed/EdgeBench

Tutte le issue di ByteDance-Seed/EdgeBench

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.