Owner

ethz-spylab

2 indexed repositories · View on GitHub

agentdojo
A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

Python · 842 stars
cryptanalysis-benchmark

C · 41 stars

Beginner friendly Everything

Open All

LocalLLM pipeline silently drops fenced tool calls and never caps generation (affects all local-model results) Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

ethz-spylab/agentdojo#202 · 1 comment · Opened Sep 20, 2026
Workspace v1.2.2 + important_instructions: 0/560 attack success on claude-sonnet-4-6 — is the suite still discriminating? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

ethz-spylab/agentdojo#201 · Opened Sep 20, 2026
Three tool interfaces (banking, car rental, Slack invite) promise more than their bodies currently deliver Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

ethz-spylab/agentdojo#194 · 1 comment · Opened Sep 12, 2026
transformers_pi_detector defense is hardcoded to single-message mode — no trajectory-aware option in the benchmarked defense list Open

Difficulty 5/5 Over a week Newbie friendliness 45/100

ethz-spylab/agentdojo#192 · Opened Sep 10, 2026
`slack` utility check raises KeyError when `post_webpage` does not execute Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

ethz-spylab/agentdojo#188 · 1 comment · Opened Sep 3, 2026
9% of (user task, injection task) pairs stay inside the tools the user has already authorised - and an offer to contribute more Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

ethz-spylab/agentdojo#187 · Opened Sep 2, 2026
Optional EvalPort interop for tasks and TaskResults Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

ethz-spylab/agentdojo#185 · Opened Aug 22, 2026
Proposal: generic pre-execution authorizer for FunctionsRuntime Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

ethz-spylab/agentdojo#184 · Opened Aug 19, 2026
Banking injection tasks 0/1/3: operator precedence makes the subject check dead code Open

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

ethz-spylab/agentdojo#181 · Opened Aug 17, 2026
Submission scope for a Banking-specific defense result Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

ethz-spylab/agentdojo#178 · Opened Jul 30, 2026
Oracle solutions broken for T2_spoc variants, T2_ranksign, and T1_sike Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

ethz-spylab/cryptanalysis-benchmark#1 · Opened Jul 29, 2026
Question about ASR results on the Workspace EASY subset Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

ethz-spylab/agentdojo#172 · Opened Jul 18, 2026
slack injection_task_5: security_from_traces simulates state from attempted calls, scoring blocked (never-executed) sequences as attack success Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

ethz-spylab/agentdojo#168 · 2 comments · 1 reaction · Opened Jul 4, 2026
`Workspace UserTask34` can pass after omitting `hot water` from the Chocolate Chip Cookies ingredients Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

ethz-spylab/agentdojo#162 · 1 comment · Opened May 17, 2026
`Banking UserTask5` utility checks the old `$50` Spotify transaction instead of the required new `$5` payment Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

ethz-spylab/agentdojo#161 · 1 comment · Opened May 17, 2026
Question: would AgentDojo welcome a detection-evaluator extension? (proposal + offer to draft) Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

ethz-spylab/agentdojo#160 · Opened May 11, 2026
tool_filter defense sends "vllm_parsed" as model name instead of real model id Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

ethz-spylab/agentdojo#158 · 2 comments · Opened Apr 30, 2026
Feature: LLM-as-a-Judge Evaluation Framework Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

ethz-spylab/agentdojo#156 · Opened Apr 22, 2026
Some doubts about the security metric Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

ethz-spylab/agentdojo#154 · 1 comment · Opened Apr 9, 2026
Streaming Evaluations Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

ethz-spylab/agentdojo#151 · Opened Apr 7, 2026
LocalLLM _parse_model_output only preserves the first function call Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

ethz-spylab/agentdojo#150 · Opened Apr 3, 2026
Support arbitrary OpenAI-compatible endpoints without source modifications or port forwarding Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

ethz-spylab/agentdojo#148 · Opened Mar 23, 2026
Agentdojo doesn't run Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

ethz-spylab/agentdojo#142 · Opened Feb 4, 2026
Trade-off between security and utility ? Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

ethz-spylab/agentdojo#140 · Opened Feb 1, 2026
Feature Request: Support for multi-agent / delegation scenarios evaluation Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

ethz-spylab/agentdojo#137 · 1 comment · Opened Jan 5, 2026
Poor security checks Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

ethz-spylab/agentdojo#135 · 15 comments · Opened Dec 7, 2025
Is ToolsExecutor run before transformers_pi_detector, causing exposed tool outputs and high attack success rates? Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

ethz-spylab/agentdojo#133 · 1 comment · Opened Oct 30, 2025
Using repeat_user_prompt with Cohere causes errors Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

ethz-spylab/agentdojo#132 · Opened Oct 28, 2025
pydantic.errors.PydanticUserError: `TaskResults` is not fully defined Open

ethz-spylab/agentdojo#88 · 4 comments · 1 assignee · Opened May 26, 2025
Decouple the benchmark data from the code logic to execute the agents into different pip packages (and GitHub repositories?) Open

enhancement

ethz-spylab/agentdojo#83 · 1 assignee · Opened May 9, 2025

ethz-spylab

Get new issues in your inbox