Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

feat(developer-experience): claude plugin eval suite in GitHub Actions

Aperta
  • Qualcosa non va?
#6,649 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github-actions, shell
Ambito
developer-experience, devops, testing-qa

Direzione di ricerca

Start with the linked upstream issues (anthropics/claude-code#100681, melodic-software/provisioning#741) and the plugin-evals docs to see what claude plugin eval needs in a sandboxed Linux runner. Then check the hand-rolled harness and fixtures from the first developer-experience PR (#6302) to see which scenarios to port. Done means a workflow_dispatch job runs the suite and posts the with/without delta per case, and the CLAUDE_CODE_OAUTH_TOKEN question is answered from that run's logs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

needs-human needs-triage

Refs: #6302

Summary

The developer-experience plugin's first PR measures its skills with a hand-rolled harness: nested claude -p runs in three arms (no plugin, team conventions in AGENTS.md, plugin) on fixture copies outside this repository. This issue tracks the follow-up: a claude plugin eval suite for the plugin that runs in GitHub Actions, so the measurement repeats on every change instead of by hand.

Why it is a separate issue

  • claude plugin eval refuses Bash-granting cases on the maintainer's WSL host (socat missing; Docker Desktop's ~/.docker/contexts symlink into /mnt/c, which the eval pre-flight refuses; seen on Claude Code 2.1.289): melodic-software/provisioning#741.
  • workflow_dispatch only triggers a workflow whose file is on the default branch (docs.github.com, "Manually running a workflow"), so the CI job can only run once this lands on main.
  • claude plugin eval loads no project CLAUDE.md, .claude/ or .mcp.json from the workspace (code.claude.com plugin-evals page), so the conventions-only arm stays in the hand-rolled harness; plugin eval covers the with/without-plugin comparison and the presence and firing cases.

Proposed shape

  • A workflow_dispatch job on ubuntu-24.04 that installs bubblewrap and socat and lifts the Ubuntu 24.04 AppArmor user-namespace restriction for bwrap (sysctl kernel.apparmor_restrict_unprivileged_userns=0), the step Anthropic's sandbox-runtime CI runs. MEDIUM confidence; the first dispatch settles it.
  • One GitHub environment per Claude account, each holding its own CLAUDE_CODE_OAUTH_TOKEN secret, so eval spend is split from the review account.
  • First run checks whether CLAUDE_CODE_OAUTH_TOKEN reaches the eval run: the plugin-evals page says most of the shell environment is withheld (an allowlist plus EVAL_* variables), and its CI example names only ANTHROPIC_API_KEY.
  • Cases ported from the PR's scenarios (S1-S7, T1-T4, N1, stays-quiet), deterministic graders where possible and llm graders checked against labeled samples.
  • The provisioning#741 eval distro mirrors the job locally for pre-push runs.

Done when

  • The workflow runs the suite on a dispatch and posts the with/without delta per case.
  • The OAuth-secret question above is answered with the run's evidence.
Lingua principale
Shell
Stelle
22
Fork
2
Merge medio
6h 42m
PR unite (30g)
902

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di melodic-software/claude-code-plugins

Tutte le issue di melodic-software/claude-code-plugins

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.