Injection and tool-use benchmark on local models
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 20/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
- 领域
- security, testing-qa
调研方向
Start by reading docs/VERIFICATION.md for the existing manual real-model checks and sentient/cli.py for where a new script entry would go. The harness goes in a new bench/ or tests/bench/ folder that the default pytest run must skip; the issue leaves the choice of injection and tool-chain cases open, so settle that first. Done means one command against a configured local model prints attack and task success rates, and the qwen3:8b results with and without spotlighting are recorded in docs/VERIFICATION.md.
由索引模型根据 Issue 内容生成。
描述
No published evaluation shows how CaMeL-style or spotlighting defenses do on 7 to 8B models. Sentient should know before it advertises SAFE-2 and SAFE-12, and it can fill that gap publicly.
Idea: A local harness modelled on AgentDojo and ClawBench that runs injection attempts and multi-step tool chains on qwen3:8b before each release and measures the effect of SAFE-12 and SAFE-2. It is a manual or scheduled job, never part of the no-network CI.
Status in Sentient today: New. docs/VERIFICATION.md records manual real-model checks.
Where: A new bench/ or tests/bench/ folder excluded from the default pytest run, a script entry in sentient/cli.py, and results recorded in docs/VERIFICATION.md.
Done when:
- one command runs the suite against a configured local model and prints attack success and task success rates
- results for qwen3:8b with and without spotlighting are recorded
Sources:
Roadmap ID: LOCAL-10 · Priority: P2 · Size: M
- 主要语言
- Python
- 星标
- 702
- 派生
- 113
- 平均合并
- 5 小时 8 分钟
- 30 天内合并 PR
- 171
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
existence-master/Sentient 的其他 Issue
-
area: channels enhancement good first issue priority: P2 size: S
难度 2/5 1-3 小时 新手友好度 78/100
existence-master/Sentient#457 ·
维护者通常 1 天内回复
-
area: desktop bug good first issue priority: P2 size: S
难度 2/5 1-3 小时 新手友好度 82/100
existence-master/Sentient#454 ·
维护者通常 1 天内回复
-
area: desktop bug priority: P2 size: S
难度 2/5 1-3 小时 新手友好度 82/100
existence-master/Sentient#440 ·
维护者通常 1 天内回复
-
area: desktop bug
难度 2/5 1-3 小时 新手友好度 78/100
existence-master/Sentient#426 ·
维护者通常 1 天内回复
-
area: memory bug flaky
难度 2/5 1-3 小时 新手友好度 76/100
existence-master/Sentient#422 ·
维护者通常 1 天内回复
查看 existence-master/Sentient 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
google-ai-edge/litert-torch#1346 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
geneontology/gocam-py#272 ·
-
pushed_authorization_request sends list authorization_details as a Python repr可能已有人在做 @RaphaelFakhri 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
auth0/auth0-python#903 ·
维护者通常 1 天内回复