Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Optional EvalPort adapter for WorkflowRunResult (portable eval-result interchange)

Open
#676 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start with runtime/sdk.py to inspect WorkflowRunResult and run_workflow(), then read EvalPort's SPEC.md to verify the Result, ResultSet, Suite, and TestCase mappings. Confirm with maintainers whether the adapter belongs in ChatDev or EvalPort and how an optional dependency should be handled. Done means the chosen adapter scope and portable task-to-workflow mapping are agreed and implemented without changing the executor.

Written by the indexing model from the issue text.

Description

I've been reading runtime/sdk.py — run_workflow() returning a WorkflowRunResult(final_message: Message, meta_info: WorkflowMetaInfo), where WorkflowMetaInfo carries session_name, log_id, outputs, token_usage, and output_dir. That's a clean, structured shape for "what happened when this workflow graph ran" — session identity, token accounting, and the final output message all in one place.

I maintain EvalPort, a JSON-Schema-based interchange spec (TestCase/Suite/Grader/Result/ResultSet) so eval data — a run's outcome plus whatever graded it — doesn't stay locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).

WorkflowRunResult maps fairly directly onto an EvalPort Result: final_message as the graded output, token_usage as EvalPort's usage metadata, session_name/log_id as run identifiers. Running the same YAML workflow over a batch of task prompts and collecting the WorkflowRunResults would give a natural ResultSet. What's missing is a way to get that out in a form another framework's grading tooling could read, or to drive run_workflow() from a portable task suite instead of one-off task_prompt strings.

I'd propose a standalone, optional adapter: WorkflowRunResult → EvalPort Result/ResultSet, plus a helper that maps an EvalPort Suite's TestCase.inputs onto run_workflow(..., task_prompt=...) calls. No required dependency, no change to runtime/sdk.py or the graph executor.

Happy to build this as a PR into ChatDev (e.g. a small module beside runtime/sdk.py), or as a standalone package in EvalPort's own adapters/ directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

— Sahi, independent contributor (not affiliated with ChatDev/OpenBMB)

Dominant language
Python
Stars
34.3k
Forks
4.3k
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenBMB/ChatDev

All issues in OpenBMB/ChatDev

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.