Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Feat: Add new eval type 'match_output' for stdout grading

Open
  • Is something wrong?
#3 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Stale
Tech stack
elixir
Domain
ai, testing-qa

Research direction

Start in evals.ex by tracing Evals.from_yml/1, Evals.grade/3, and Evals.write_and_eval/3 to understand the existing eval flow and stdout capture. Implement the new YAML structure and grading path so trimmed expected and captured output produce 1 on a match and 0 otherwise, then verify both parsing and grading behavior.

Written by the indexing model from the issue text.

Description

enhancement
Is your feature request related to a problem? Please describe.

Labels: feature, eval-engine, grader

Description:
As an eval author, I want to create evaluations that test the standard output (stdout) of a script. This is crucial for testing command-line behavior, simple script outputs, or any task where the primary result is printed to the console rather than returned from a function.

This involves creating a new eval type named match_output and implementing the corresponding grading logic.

Describe the solution you'd like

Acceptance Criteria:

  • The framework supports a new eval type: "match_output" in the YAML definition.
  • The YAML for this type must support a new key: eval.expected_output (string).
  • The Evals.from_yml/1 function must correctly parse this new structure.
  • A new function clause must be added to Evals.grade/3 to handle evals where type is :match_output.
  • This new grader should:
    1. Execute the model-generated code using Evals.write_and_eval/3.
    2. Capture the stdout from the execution.
    3. Compare the captured stdout with the expected_output from the eval's definition.
    4. The comparison should be robust against leading/trailing whitespace on both the actual and expected output.
    5. A successful match results in a grade of 1; a mismatch results in 0.

Implementation Notes:

  • The main logic will be a new function head for Evals.grade/3 in evals.ex.
  • The Evals.write_and_eval/3 function already captures stdout, so it can be used directly.
  • Use String.trim/1 on both the captured output and the expected output before comparison.

Example YAML:

type: match_output
messages:
  - type: user
    text: "Write a script that prints 'Hello, World!' to the console."
eval:
  expected_output: |
    Hello, World!
Describe alternatives you've considered

No response

Additional context

No response

Dominant language
Elixir
Stars
51
Forks
6
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ash-project/evals

All issues in ash-project/evals

Similar issues

More Elixir issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.