Feat: Implement `:tagged_summary` report format
@SubWal is already working on this.
Since Oct 7, 2026.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
Research direction
Start in evals/formatter.ex at Evals.Formatter.format_report/3 and compare the existing :full and :summary formats. Implement the :tagged_summary option using completed eval results, then verify unique tags are alphabetically sorted, averages are human-readable, untagged results are omitted gracefully, and existing formats remain unchanged.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
Labels: feature, reporting, formatter
Dependencies: Blocks Ticket #1 (YAML Metadata Parsing)
Description:
As a model evaluator, I want a report format that groups evaluation results by their tags. This will allow for a more granular analysis of a model's performance, helping to identify strengths and weaknesses in specific areas like otp, ecto, or pattern-matching.
This requires creating a new report format option, :tagged_summary.
Describe the solution you'd like
Acceptance Criteria:
- The
Evals.Formatter.format_report/3function accepts a new value for the:formatoption::tagged_summary. - When
format: :tagged_summaryis used, the report output should:- List each unique tag found across all completed evals.
- For each tag, display the average score of all evals that include that tag.
- The list of tags should be sorted alphabetically for consistent output.
- The final report must be clean and human-readable.
- The implementation must not break the existing
:fulland:summaryreport formats. - If no evals have tags, this section of the report should be gracefully omitted.
Implementation Notes:
- This work will be done in
evals/formatter.ex. - A new private function, such as
format_tagged_summary(results), will likely be needed. - The logic will involve:
- Reducing the
resultslist into a map where keys are tags and values are lists of scores (e.g.,%{otp: [1.0, 0.5], elixir_core: [1.0, 1.0, 0.5], ...}). - Transforming this map to calculate the average score for each tag.
- Sorting the results by tag name before formatting them into strings.
- Reducing the
Example Output:
... (Overall Summary) ...
TAGGED SUMMARY:
----------------------------------------
elixir-core | 83.3%
ecosystem | 100.0%
otp | 75.0%
pattern-matching | 100.0%
security | 0.0%
... (Detailed Results or end of report) ...
Describe alternatives you've considered
No response
Additional context
No response
- Dominant language
- Elixir
- Stars
- 51
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ash-project/evals
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
ash-project/evals#5 · 11 comments ·
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 55/100
ash-project/evals#3 · 1 reaction ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 50/100
ash-project/evals#1 · 1 reaction ·
All issues in ash-project/evals
Similar issues
-
feat: signup pageOpend: medium
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
L: javascript
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
dependabot/dependabot-core#16744 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 Under an hour Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
elixir-ecto/tds#191 ·