Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feat(developer-experience): claude plugin eval suite in GitHub Actions

Open
  • Is something wrong?
#6,649 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
github-actions, shell
Domain
developer-experience, devops, testing-qa

Research direction

Start with the linked upstream issues (anthropics/claude-code#100681, melodic-software/provisioning#741) and the plugin-evals docs to see what claude plugin eval needs in a sandboxed Linux runner. Then check the hand-rolled harness and fixtures from the first developer-experience PR (#6302) to see which scenarios to port. Done means a workflow_dispatch job runs the suite and posts the with/without delta per case, and the CLAUDE_CODE_OAUTH_TOKEN question is answered from that run's logs.

Written by the indexing model from the issue text.

Description

needs-human needs-triage

Refs: #6302

Summary

The developer-experience plugin's first PR measures its skills with a hand-rolled harness: nested claude -p runs in three arms (no plugin, team conventions in AGENTS.md, plugin) on fixture copies outside this repository. This issue tracks the follow-up: a claude plugin eval suite for the plugin that runs in GitHub Actions, so the measurement repeats on every change instead of by hand.

Why it is a separate issue

  • claude plugin eval refuses Bash-granting cases on the maintainer's WSL host (socat missing; Docker Desktop's ~/.docker/contexts symlink into /mnt/c, which the eval pre-flight refuses; seen on Claude Code 2.1.289): melodic-software/provisioning#741.
  • workflow_dispatch only triggers a workflow whose file is on the default branch (docs.github.com, "Manually running a workflow"), so the CI job can only run once this lands on main.
  • claude plugin eval loads no project CLAUDE.md, .claude/ or .mcp.json from the workspace (code.claude.com plugin-evals page), so the conventions-only arm stays in the hand-rolled harness; plugin eval covers the with/without-plugin comparison and the presence and firing cases.

Proposed shape

  • A workflow_dispatch job on ubuntu-24.04 that installs bubblewrap and socat and lifts the Ubuntu 24.04 AppArmor user-namespace restriction for bwrap (sysctl kernel.apparmor_restrict_unprivileged_userns=0), the step Anthropic's sandbox-runtime CI runs. MEDIUM confidence; the first dispatch settles it.
  • One GitHub environment per Claude account, each holding its own CLAUDE_CODE_OAUTH_TOKEN secret, so eval spend is split from the review account.
  • First run checks whether CLAUDE_CODE_OAUTH_TOKEN reaches the eval run: the plugin-evals page says most of the shell environment is withheld (an allowlist plus EVAL_* variables), and its CI example names only ANTHROPIC_API_KEY.
  • Cases ported from the PR's scenarios (S1-S7, T1-T4, N1, stays-quiet), deterministic graders where possible and llm graders checked against labeled samples.
  • The provisioning#741 eval distro mirrors the job locally for pre-push runs.

Done when

  • The workflow runs the suite on a dispatch and posts the with/without delta per case.
  • The OAuth-secret question above is answered with the run's evidence.
Dominant language
Shell
Stars
22
Forks
2
Avg merge
5h 18m
Merged PRs (30d)
869

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from melodic-software/claude-code-plugins

All issues in melodic-software/claude-code-plugins

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.