Interest check: example for exporting chat.chat() results as portable eval fixtures (EvalPort)?
还没有人认领这个 Issue。
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 58/100
- Issue 类型
- 文档
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
Review the existing examples/ directory and the streaming helpers to see how chat.chat() usage is currently demonstrated. Assess whether a short EvalPort export example fits the library’s scope; done means adding a self-contained example only if it matches the project’s conventions and maintainers approve the direction.
由索引模型根据 Issue 内容生成。
描述
Hi! I maintain EvalPort, a small open spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets meant to move between different eval tools instead of each one inventing its own file format.
This SDK's client.chat.chat(messages=..., model="palmyra-x5") response already carries what's needed for a lightweight eval record — the input messages, model, choices[0].message.content, and id. Something like:
from evalport import TestResult
def to_evalport(messages, model, chat_completion) -> TestResult:
return TestResult(
test_case_id=chat_completion.id,
input={"messages": messages},
actual_output=chat_completion.choices[0].message.content,
metadata={"model": model},
)
I know this SDK is Stainless-generated, so I'm not proposing a change to the generated client — more asking whether a short snippet like this would be worth having in examples/ alongside the streaming helpers, or whether it's not a good fit for a library this thin. Genuinely fine either way, just wanted to check before doing anything.
Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — feel free to close this if it's not useful.
- 主要语言
- Python
- 星标
- 11
- 派生
- 2
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
writer/writer-python 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 35/100
writer/writer-python#156 ·
查看 writer/writer-python 的全部 Issue
相似的 Issue
-
bug status/needs-triage
难度 2/5 1-3 小时 新手友好度 86/100
prowler-cloud/prowler#12887 · 1 条评论 ·
维护者通常 1 天内回复
-
area: desktop platform: macos priority: p3 status: ready type: enhancement
难度 1/5 1 小时以内 新手友好度 92/100
use-agent-os/agent-os#3484 ·
维护者通常 2 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 86/100
open-telemetry/opentelemetry-python-contrib#5113 · 2 条评论 · 2 个 reaction ·
维护者通常 1 天内回复
-
external
难度 2/5 1-3 小时 新手友好度 68/100
langchain-ai/docs#6255 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复