Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Optional EvalPort adapter for WorkflowRunResult (portable eval-result interchange)

Abierto
  • ¿Algo no está bien?
#676 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
testing-qa, tooling

Línea de trabajo

Start with runtime/sdk.py to inspect WorkflowRunResult and run_workflow(), then read EvalPort's SPEC.md to verify the Result, ResultSet, Suite, and TestCase mappings. Confirm with maintainers whether the adapter belongs in ChatDev or EvalPort and how an optional dependency should be handled. Done means the chosen adapter scope and portable task-to-workflow mapping are agreed and implemented without changing the executor.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

I've been reading runtime/sdk.py — run_workflow() returning a WorkflowRunResult(final_message: Message, meta_info: WorkflowMetaInfo), where WorkflowMetaInfo carries session_name, log_id, outputs, token_usage, and output_dir. That's a clean, structured shape for "what happened when this workflow graph ran" — session identity, token accounting, and the final output message all in one place.

I maintain EvalPort, a JSON-Schema-based interchange spec (TestCase/Suite/Grader/Result/ResultSet) so eval data — a run's outcome plus whatever graded it — doesn't stay locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).

WorkflowRunResult maps fairly directly onto an EvalPort Result: final_message as the graded output, token_usage as EvalPort's usage metadata, session_name/log_id as run identifiers. Running the same YAML workflow over a batch of task prompts and collecting the WorkflowRunResults would give a natural ResultSet. What's missing is a way to get that out in a form another framework's grading tooling could read, or to drive run_workflow() from a portable task suite instead of one-off task_prompt strings.

I'd propose a standalone, optional adapter: WorkflowRunResult → EvalPort Result/ResultSet, plus a helper that maps an EvalPort Suite's TestCase.inputs onto run_workflow(..., task_prompt=...) calls. No required dependency, no change to runtime/sdk.py or the graph executor.

Happy to build this as a PR into ChatDev (e.g. a small module beside runtime/sdk.py), or as a standalone package in EvalPort's own adapters/ directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

— Sahi, independent contributor (not affiliated with ChatDev/OpenBMB)

Lenguaje dominante
Python
Estrellas
34.5k
Forks
4.3k
Sin PR fusionados en 30 d

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de OpenBMB/ChatDev

Todos los issues de OpenBMB/ChatDev

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.