chatcops test: calls-tool and no-tool-call assertions
维护者通常 1 天内回复
评估
这个 Issue 还没有评估数据。
描述
Lets an eval check that the bot called the right tool with the right input, or that it called none.
How assertions work
chatcops test (#394, with replays from #395 and the judge from #400) checks each turn's reply with the assertions in the eval file. They're registered by name in ASSERTIONS (packages/cli/src/test/assertions/index.ts:11). Each one is an AssertionCheck (packages/cli/src/test/types.ts): it gets the value from the YAML, the turn's TurnResult (text, toolCalls, sources, leads, usage, costUsd, latencyMs), the conversation so far, and a context with the eval file's path, and returns { pass, reason }. Throw for a bad value (see asStrings in assertions/values.ts), and the runner turns it into a failure with your message. The not- prefix works for every name without extra code.
contains in assertions/text.ts is the reference: one file per kind of assertion, added to the map. Write reasons a person can act on, such as the reply doesn't contain "$19".
What to add (new assertions/tools.ts)
calls-tool: a tool name (calls-tool: lookup_order), or{ name, args }.argsis a partial deep match against the call'sinput: every key inargshas to be there with an equal value, and extra keys in the input are fine. Nested objects match the same way, and arrays have to be equal.no-tool-call:truemeans the turn called no tool at all. A tool name means it didn't call that one.
The calls are in turn.toolCalls as { name, input }. The reason for a failure should list the calls that did happen, so people see what went wrong.
Acceptance criteria
- Unit tests: name only, partial args, nested args, a mismatch that lists the actual calls,
no-tool-callboth ways, bad values - An end to end test with a tool mock (see "answers tools from mocks" in
test-command.test.ts) - Documented in
website/src/content/docs/core/testing.mdx, with a changeset for@chatcops/cli(minor) -
pnpm test:coverageandpnpm lintpass
- 主要语言
- TypeScript
- 星标
- 13
- 派生
- 7
- 平均合并
- 3 小时 39 分钟
- 30 天内合并 PR
- 275
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
codercops/chatcops 的其他 Issue
-
Widget: fetch signed-in visitor tokens when a session is requested可能已有人在做 @piyusshhjangid 今天认领。 未关闭enhancement
难度 3/5 1-2 天 新手友好度 63/100
codercops/chatcops#461 · 2 条评论 ·
维护者通常 1 天内回复
-
Widget: support CSP nonces for injected styles可能已有人在做 @piyusshhjangid 今天认领。 未关闭enhancement
难度 4/5 3-5 天 新手友好度 55/100
codercops/chatcops#460 · 2 条评论 ·
维护者通常 1 天内回复
-
Server: validate `rateLimit.windowMs` before creating the limiter可能已有人在做 @ashutoshpalhare 于 1 天前认领。 未关闭
codercops/chatcops#452 · 1 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Widget: a "New reply" button when the visitor has scrolled up可能已有人在做 @srujankaleru2007 于 1 天前认领。 未关闭enhancement hacktoberfest widget
codercops/chatcops#420 · 3 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
chatcops test: --url to test a deployed server可能已有人在做 @srujankaleru2007 于 1 天前认领。 未关闭enhancement hacktoberfest testing
codercops/chatcops#419 · 3 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
查看 codercops/chatcops 的全部 Issue
相似的 Issue
-
bug
难度 1/5 1 小时以内 新手友好度 86/100
awslabs/visual-asset-management-system#937 ·
维护者通常 1 天内回复
-
t0ggles
难度 1/5 1 小时以内 新手友好度 82/100
nolimits4web/swiper#8255 · 1 条评论 ·
维护者通常 15 天内回复
-
render: --variables-file (and --variables <path>) fails with "io.readFile is not a function" since 0.8.212可能已有人在做 @miguel-heygen 今天认领。 未关闭bug triage/needs-triage
难度 2/5 1-3 小时 新手友好度 76/100
heygen-com/hyperframes#6830 ·
维护者通常 1 天内回复
-
bug launch local use
难度 2/5 1-3 小时 新手友好度 65/100
维护者通常 1 天内回复
-
account: profile bug
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 5 天内回复