chatcops test: calls-tool and no-tool-call assertions
維護者通常 1 天內回覆
評估
這個 Issue 還沒有評估資料。
描述
Lets an eval check that the bot called the right tool with the right input, or that it called none.
How assertions work
chatcops test (#394, with replays from #395 and the judge from #400) checks each turn's reply with the assertions in the eval file. They're registered by name in ASSERTIONS (packages/cli/src/test/assertions/index.ts:11). Each one is an AssertionCheck (packages/cli/src/test/types.ts): it gets the value from the YAML, the turn's TurnResult (text, toolCalls, sources, leads, usage, costUsd, latencyMs), the conversation so far, and a context with the eval file's path, and returns { pass, reason }. Throw for a bad value (see asStrings in assertions/values.ts), and the runner turns it into a failure with your message. The not- prefix works for every name without extra code.
contains in assertions/text.ts is the reference: one file per kind of assertion, added to the map. Write reasons a person can act on, such as the reply doesn't contain "$19".
What to add (new assertions/tools.ts)
calls-tool: a tool name (calls-tool: lookup_order), or{ name, args }.argsis a partial deep match against the call'sinput: every key inargshas to be there with an equal value, and extra keys in the input are fine. Nested objects match the same way, and arrays have to be equal.no-tool-call:truemeans the turn called no tool at all. A tool name means it didn't call that one.
The calls are in turn.toolCalls as { name, input }. The reason for a failure should list the calls that did happen, so people see what went wrong.
Acceptance criteria
- Unit tests: name only, partial args, nested args, a mismatch that lists the actual calls,
no-tool-callboth ways, bad values - An end to end test with a tool mock (see "answers tools from mocks" in
test-command.test.ts) - Documented in
website/src/content/docs/core/testing.mdx, with a changeset for@chatcops/cli(minor) -
pnpm test:coverageandpnpm lintpass
- 主要語言
- TypeScript
- 星號
- 13
- 分支
- 7
- 平均合併
- 3 小時 28 分鐘
- 30 天內合併 PR
- 296
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
codercops/chatcops 的其他 Issue
-
Core README lists old default models可能已有人在做 @prakashNalla 今天認領。 未關閉
難度 1/5 1 小時以內 新手友好度 90/100
codercops/chatcops#503 · 1 則留言 ·
維護者通常 1 天內回覆
-
Widget: theme.tokens to set any color or size token, and the full token table in the docs可能已有人在做 @Sreehita-here 今天認領。 未關閉hacktoberfest
codercops/chatcops#500 · 2 則留言 · 已指派 1 人 ·
維護者通常 1 天內回覆
-
Widget: a header menu with New conversation, Download transcript and a privacy link可能已有人在做 @anurag2787 今天認領。 未關閉hacktoberfest
codercops/chatcops#496 · 4 則留言 · 已指派 1 人 ·
維護者通常 1 天內回覆
-
Widget: keep a message sent while offline, and send it when the connection is back可能已有人在做 @Saifjafri212213 今天認領。 未關閉hacktoberfest
codercops/chatcops#495 · 3 則留言 · 已指派 1 人 ·
維護者通常 1 天內回覆
-
Widget: copy a completed assistant reply可能已有人在做 @piyusshhjangid 今天認領。 未關閉hacktoberfest
codercops/chatcops#477 · 2 則留言 · 已指派 1 人 ·
維護者通常 1 天內回覆
查看 codercops/chatcops 的全部 Issue
相似的 Issue
-
Replace discontinued crypto-js in the theme可能已有人在做 @aelmanaa 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 85/100
PaloAltoNetworks/docusaurus-openapi-docs#1689 ·
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 70/100
solana-foundation/solana-web3.js#4015 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 85/100
維護者通常 1 天內回覆
-
documentation
難度 1/5 1-3 小時 新手友好度 90/100
ivopogace/opencode-courier#422 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 68/100
chapel-lang/chapel-vscode#158 ·
維護者通常 1 天內回覆