Interest check: example for exporting chat.chat() results as portable eval fixtures (EvalPort)?
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 58/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- documentation
Research direction
Review the existing examples/ directory and the streaming helpers to see how chat.chat() usage is currently demonstrated. Assess whether a short EvalPort export example fits the library’s scope; done means adding a self-contained example only if it matches the project’s conventions and maintainers approve the direction.
Written by the indexing model from the issue text.
Description
Hi! I maintain EvalPort, a small open spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets meant to move between different eval tools instead of each one inventing its own file format.
This SDK's client.chat.chat(messages=..., model="palmyra-x5") response already carries what's needed for a lightweight eval record — the input messages, model, choices[0].message.content, and id. Something like:
from evalport import TestResult
def to_evalport(messages, model, chat_completion) -> TestResult:
return TestResult(
test_case_id=chat_completion.id,
input={"messages": messages},
actual_output=chat_completion.choices[0].message.content,
metadata={"model": model},
)
I know this SDK is Stainless-generated, so I'm not proposing a change to the generated client — more asking whether a short snippet like this would be worth having in examples/ alongside the streaming helpers, or whether it's not a good fit for a library this thin. Genuinely fine either way, just wanted to check before doing anything.
Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — feel free to close this if it's not useful.
- Dominant language
- Python
- Stars
- 11
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from writer/writer-python
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
writer/writer-python#156 ·
All issues in writer/writer-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
learningequality/ricecooker#747 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
run-llama/llama_index#23199 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/glTF-Blender-IO#2769 ·