Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

feat(developer-experience): claude plugin eval suite in GitHub Actions

オープン
  • 何か問題がありますか？
#6,649 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
22/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
github-actions, shell
領域
developer-experience, devops, testing-qa

調査の方向性

Start with the linked upstream issues (anthropics/claude-code#100681, melodic-software/provisioning#741) and the plugin-evals docs to see what claude plugin eval needs in a sandboxed Linux runner. Then check the hand-rolled harness and fixtures from the first developer-experience PR (#6302) to see which scenarios to port. Done means a workflow_dispatch job runs the suite and posts the with/without delta per case, and the CLAUDE_CODE_OAUTH_TOKEN question is answered from that run's logs.

索引モデルが issue の本文から書いたものです。

説明

needs-human needs-triage

Refs: #6302

Summary

The developer-experience plugin's first PR measures its skills with a hand-rolled harness: nested claude -p runs in three arms (no plugin, team conventions in AGENTS.md, plugin) on fixture copies outside this repository. This issue tracks the follow-up: a claude plugin eval suite for the plugin that runs in GitHub Actions, so the measurement repeats on every change instead of by hand.

Why it is a separate issue

  • claude plugin eval refuses Bash-granting cases on the maintainer's WSL host (socat missing; Docker Desktop's ~/.docker/contexts symlink): anthropics/claude-code#100681, melodic-software/provisioning#741.
  • workflow_dispatch only triggers a workflow whose file is on the default branch (docs.github.com, "Manually running a workflow"), so the CI job can only run once this lands on main.
  • claude plugin eval loads no project CLAUDE.md, .claude/ or .mcp.json from the workspace (code.claude.com plugin-evals page), so the conventions-only arm stays in the hand-rolled harness; plugin eval covers the with/without-plugin comparison and the presence and firing cases.

Proposed shape

  • A workflow_dispatch job on ubuntu-24.04 that installs bubblewrap and socat and lifts the Ubuntu 24.04 AppArmor user-namespace restriction for bwrap (sysctl kernel.apparmor_restrict_unprivileged_userns=0), the step Anthropic's sandbox-runtime CI runs. MEDIUM confidence; the first dispatch settles it.
  • One GitHub environment per Claude account, each holding its own CLAUDE_CODE_OAUTH_TOKEN secret, so eval spend is split from the review account.
  • First run checks whether CLAUDE_CODE_OAUTH_TOKEN reaches the eval run: the plugin-evals page says most of the shell environment is withheld (an allowlist plus EVAL_* variables), and its CI example names only ANTHROPIC_API_KEY.
  • Cases ported from the PR's scenarios (S1-S7, T1-T4, N1, stays-quiet), deterministic graders where possible and llm graders checked against labeled samples.
  • The provisioning#741 eval distro mirrors the job locally for pre-push runs.

Done when

  • The workflow runs the suite on a dispatch and posts the with/without delta per case.
  • The OAuth-secret question above is answered with the run's evidence.
主要言語
Shell
スター
22
フォーク
2
平均マージ
5時間 15分
マージ済み PR（30日）
833

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

melodic-software/claude-code-plugins のほかの issue

melodic-software/claude-code-plugins の issue をすべて見る

似ている issue

Shell/Bash の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。