Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

feat(developer-experience): claude plugin eval suite in GitHub Actions

未关闭
  • 哪里不对？
#6,649 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
22/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
github-actions, shell
领域
developer-experience, devops, testing-qa

调研方向

Start with the linked upstream issues (anthropics/claude-code#100681, melodic-software/provisioning#741) and the plugin-evals docs to see what claude plugin eval needs in a sandboxed Linux runner. Then check the hand-rolled harness and fixtures from the first developer-experience PR (#6302) to see which scenarios to port. Done means a workflow_dispatch job runs the suite and posts the with/without delta per case, and the CLAUDE_CODE_OAUTH_TOKEN question is answered from that run's logs.

由索引模型根据 Issue 内容生成。

描述

needs-human needs-triage

Refs: #6302

Summary

The developer-experience plugin's first PR measures its skills with a hand-rolled harness: nested claude -p runs in three arms (no plugin, team conventions in AGENTS.md, plugin) on fixture copies outside this repository. This issue tracks the follow-up: a claude plugin eval suite for the plugin that runs in GitHub Actions, so the measurement repeats on every change instead of by hand.

Why it is a separate issue

  • claude plugin eval refuses Bash-granting cases on the maintainer's WSL host (socat missing; Docker Desktop's ~/.docker/contexts symlink into /mnt/c, which the eval pre-flight refuses; seen on Claude Code 2.1.289): melodic-software/provisioning#741.
  • workflow_dispatch only triggers a workflow whose file is on the default branch (docs.github.com, "Manually running a workflow"), so the CI job can only run once this lands on main.
  • claude plugin eval loads no project CLAUDE.md, .claude/ or .mcp.json from the workspace (code.claude.com plugin-evals page), so the conventions-only arm stays in the hand-rolled harness; plugin eval covers the with/without-plugin comparison and the presence and firing cases.

Proposed shape

  • A workflow_dispatch job on ubuntu-24.04 that installs bubblewrap and socat and lifts the Ubuntu 24.04 AppArmor user-namespace restriction for bwrap (sysctl kernel.apparmor_restrict_unprivileged_userns=0), the step Anthropic's sandbox-runtime CI runs. MEDIUM confidence; the first dispatch settles it.
  • One GitHub environment per Claude account, each holding its own CLAUDE_CODE_OAUTH_TOKEN secret, so eval spend is split from the review account.
  • First run checks whether CLAUDE_CODE_OAUTH_TOKEN reaches the eval run: the plugin-evals page says most of the shell environment is withheld (an allowlist plus EVAL_* variables), and its CI example names only ANTHROPIC_API_KEY.
  • Cases ported from the PR's scenarios (S1-S7, T1-T4, N1, stays-quiet), deterministic graders where possible and llm graders checked against labeled samples.
  • The provisioning#741 eval distro mirrors the job locally for pre-push runs.

Done when

  • The workflow runs the suite on a dispatch and posts the with/without delta per case.
  • The OAuth-secret question above is answered with the run's evidence.
主要语言
Shell
星标
22
派生
2
平均合并
8 小时 44 分钟
30 天内合并 PR
926

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

melodic-software/claude-code-plugins 的其他 Issue

查看 melodic-software/claude-code-plugins 的全部 Issue

相似的 Issue

更多 Shell/Bash Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。