Agent plugin: fixture repositories and agent evals for every skill
Maintainers usually reply within 1 day
@drevendev is already working on this.
Since Oct 24, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
Start by reading the existing skills and the work in issues #69–#72, then inspect the test and plugin setup to place fixtures under tests/fixtures/agent-repos and eval cases under evals/. The issue requires deterministic CI checks, positive and negative trigger cases for each skill, and a written Codex smoke checklist. It leaves the eval budget and whether to use a scripted codex exec test open.
Written by the indexing model from the issue text.
Description
Reader problem
Nothing proves that the skills trigger when they should, stay read-only when they should, and ignore instructions planted in the target repository.
Design decisions
- Fixtures:
tests/fixtures/agent-repos/{minimal, broken-links, healthy, injection-bait}, each with the expected auditrule_ids. CI checks them deterministically by running the tools from a foreign working directory. - Claude Code evals:
evals/cases forclaude plugin eval.- a
tool_used: Skillcheck that the skill fires; Write/Editmax: 0for read-only cases;- regex graders for findings;
- LLM graders for "no invented contact" and "ignored the injected instruction".
- a
- Native Windows has no eval sandbox, so evals are documented to run under WSL2 or Linux before a release.
Acceptance
- The deterministic fixture tests are green in CI.
- Each skill has at least 2 positive cases and 1 negative trigger case.
- Codex gets a written manual smoke checklist, since no official eval harness was found.
Open questions
- Budget for paid model eval runs.
- Whether a scripted
codex execsmoke test is acceptable.
Depends on: #69, #70, #71, #72
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 5h 39m
- Merged PRs (30d)
- 109
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from drevendev/BeautifulRepo
-
Retire the hacktoberfest label: it claims participation the programs guide rules outPossibly taken A pull request linked to this issue is open or already merged. Opendocumentation self-check
Difficulty 3/5 1-2 days Newbie friendliness 52/100
drevendev/BeautifulRepo#100 · 1 comment ·
Maintainers usually reply within 1 day
-
epic real-case
Difficulty 5/5 Over a week Newbie friendliness 35/100
drevendev/BeautifulRepo#88 ·
Maintainers usually reply within 1 day
-
agent-skill epic
Difficulty 5/5 Over a week Newbie friendliness 25/100
drevendev/BeautifulRepo#86 · 1 comment ·
Maintainers usually reply within 1 day
-
Real case: Quillgrain and Quillgrain_Common — show the CLI result and make reporting routes trueOpenreal-case
Difficulty 4/5 3-5 days Newbie friendliness 35/100
drevendev/BeautifulRepo#83 ·
Maintainers usually reply within 1 day
-
real-case
Difficulty 4/5 3-5 days Newbie friendliness 35/100
drevendev/BeautifulRepo#82 ·
Maintainers usually reply within 1 day
All issues in drevendev/BeautifulRepo
Similar issues
-
[Bug]: AdaLAM's draw_first_k_couples repeats couples already drawn instead of the next ones (8 of the default 128 RANSAC hypotheses)Possibly taken @1sakshm claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
TheAlgorithms/Python#15656 ·
Maintainers usually reply within 2 days
-
external integration integration-submission
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
langchain-ai/docs#7128 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
kaifcodec/user-scanner#755 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
eclipse-score/mcp-servers#61 ·
Maintainers usually reply within 4 days