Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Optional EvalPort adapter for WorkflowRunResult (portable eval-result interchange)

Aperta
#676 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Start with runtime/sdk.py to inspect WorkflowRunResult and run_workflow(), then read EvalPort's SPEC.md to verify the Result, ResultSet, Suite, and TestCase mappings. Confirm with maintainers whether the adapter belongs in ChatDev or EvalPort and how an optional dependency should be handled. Done means the chosen adapter scope and portable task-to-workflow mapping are agreed and implemented without changing the executor.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I've been reading runtime/sdk.pyrun_workflow() returning a WorkflowRunResult(final_message: Message, meta_info: WorkflowMetaInfo), where WorkflowMetaInfo carries session_name, log_id, outputs, token_usage, and output_dir. That's a clean, structured shape for "what happened when this workflow graph ran" — session identity, token accounting, and the final output message all in one place.

I maintain EvalPort, a JSON-Schema-based interchange spec (TestCase/Suite/Grader/Result/ResultSet) so eval data — a run's outcome plus whatever graded it — doesn't stay locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).

WorkflowRunResult maps fairly directly onto an EvalPort Result: final_message as the graded output, token_usage as EvalPort's usage metadata, session_name/log_id as run identifiers. Running the same YAML workflow over a batch of task prompts and collecting the WorkflowRunResults would give a natural ResultSet. What's missing is a way to get that out in a form another framework's grading tooling could read, or to drive run_workflow() from a portable task suite instead of one-off task_prompt strings.

I'd propose a standalone, optional adapter: WorkflowRunResult → EvalPort Result/ResultSet, plus a helper that maps an EvalPort Suite's TestCase.inputs onto run_workflow(..., task_prompt=...) calls. No required dependency, no change to runtime/sdk.py or the graph executor.

Happy to build this as a PR into ChatDev (e.g. a small module beside runtime/sdk.py), or as a standalone package in EvalPort's own adapters/ directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

— Sahi, independent contributor (not affiliated with ChatDev/OpenBMB)

Lingua principale
Python
Stelle
34.3k
Fork
4.3k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenBMB/ChatDev

Tutte le issue di OpenBMB/ChatDev

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.