Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Interest check: example for exporting chat.chat() results as portable eval fixtures (EvalPort)?

Open
#262 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
58/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
documentation

Research direction

Review the existing examples/ directory and the streaming helpers to see how chat.chat() usage is currently demonstrated. Assess whether a short EvalPort export example fits the library’s scope; done means adding a self-contained example only if it matches the project’s conventions and maintainers approve the direction.

Written by the indexing model from the issue text.

Description

Hi! I maintain EvalPort, a small open spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets meant to move between different eval tools instead of each one inventing its own file format.

This SDK's client.chat.chat(messages=..., model="palmyra-x5") response already carries what's needed for a lightweight eval record — the input messages, model, choices[0].message.content, and id. Something like:

from evalport import TestResult

def to_evalport(messages, model, chat_completion) -> TestResult:
    return TestResult(
        test_case_id=chat_completion.id,
        input={"messages": messages},
        actual_output=chat_completion.choices[0].message.content,
        metadata={"model": model},
    )

I know this SDK is Stainless-generated, so I'm not proposing a change to the generated client — more asking whether a short snippet like this would be worth having in examples/ alongside the streaming helpers, or whether it's not a good fit for a library this thin. Genuinely fine either way, just wanted to check before doing anything.

Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure — feel free to close this if it's not useful.

Dominant language
Python
Stars
11
Forks
2
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from writer/writer-python

All issues in writer/writer-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.