Question: independent Slai Agent Runtime evaluation and leaderboard submission

Abierto
  • ¿Algo no está bien?
#30 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
1/5
Tiempo estimado
Menos de una hora
Aptitud para principiantes
25/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
docker, kubernetes, python
Área
machine-learning

Línea de trabajo

Read the EdgeBench documentation for leaderboard eligibility, result comparability, custom agent scaffolds, and submission channels; the issue does not name specific files or tests. Clarify the workflow for independent runs on the public task subset and the recommended path for private suite access. Done means the documentation answers the five questions raised here.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Thank you for releasing EdgeBench and SForge.

Slai Agent Runtime can provision and operate an evaluation environment from a Windows host, including Docker, Python, task data, job configuration, execution, and preservation of reproducibility artifacts. Task execution would use the documented Linux Docker, E2B, or Kubernetes environment.

We have participated in Terminal-Bench 2.1 and newer Terminal-Bench test sets, with 155 published Harbor job records:

https://hub.harborframework.com/users/huong8373tt-beep/jobs

The documentation already makes clear that third-party models and custom SForge agents are supported. Before investing in a full EdgeBench run, could you please clarify the reporting and leaderboard workflow for an independently operated run?

  1. Are independently run results on the public 51-task subset eligible for consideration on the EdgeBench leaderboard or project website?

  2. If so, what evidence is required for a result to be considered comparable? For example: the exact SForge and task-definition revisions, official experiment YAML, agent/scaffold source and revision, model and endpoint configuration, hardware/backend details, run logs, trajectories, and per-task result artifacts.

  3. Is a custom Slai Agent Runtime scaffold acceptable for a reported result, provided that it uses the official task definitions and leaderboard settings unchanged apart from the documented model/environment fields?

  4. Is there a preferred submission channel for an independently run result? For example, a GitHub issue, pull request, form, or direct email.

  5. For the private 134-task suite, should an independent team first demonstrate a reproducible public-subset run before contacting the project by email?

We will clearly label any independently operated result as third-party and will not present it as an official comparable leaderboard entry without confirmation.

Thank you.

Lenguaje dominante
Python
Estrellas
474
Forks
20
Merge medio
1 d 7 h
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ByteDance-Seed/EdgeBench

Todos los issues de ByteDance-Seed/EdgeBench

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.