fix(aidd-vcs): commit and merge request rules in project memory are rarely applied

オープン
#1,258 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
github, typescript

調査の方向性

Start with plugins/aidd-vcs/skills/01-commit/SKILL.md, plugins/aidd-vcs/skills/02-pull-request/SKILL.md, and plugins/aidd-context/skills/02-project-memory/references/capabilities.md; then trace ALWAYS_WRITTEN and accepted names in scripts/project-memory.cjs. Check the existing trigger grid and project memory tests before changing descriptions or file names. Done means the acceptance criteria pass, including make check, the legacy vcs.md check and refresh behavior, and the reported four-request measurements.

索引モデルが issue の本文から書いたものです。

説明

Problem

A project writes its commit format, its branch naming and its merge rules in aidd_docs/memory/vcs.md. On a host that receives memory as links (the AGENTS.md block), an agent applies them only if it opens that file or loads a skill that sends it there. Most of the time it does neither.

Measured on a fixture whose rules exist nowhere else (commit subject <area>/<verb>: <summary>, merge requests into staging, squash merge): about 4 200 headless runs with bare links, on Claude Code, OpenCode, Kiro CLI, Codex and VS Code Copilot, with Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.2, GPT-6.1 Sol, GPT-6 Luna, GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3 and MAI-Code-1.1-Flash.

  • Four requests (write a commit message, commit, give the command that creates the merge request, write its description): rules applied in 97 of 222 runs today, on the 19 host and model pairs of the table below.
  • A loaded skill does the job: rules applied in 164 of 168 command-line runs where 01-commit or 02-pull-request loaded (a commit skill that read the rule and stopped for approval counts as applied). Their descriptions only claim "commit changes" and "open a pull or merge request", so on the ten command-line pairs measured first a skill loaded on a plain "commit" (21 of 28 runs) and never on the three other requests (0 of 108).
  • No skill covers a merge. Asked to merge an approved branch, 3 of 47 runs merged into main against the project's rule.
  • Models split in two groups. OpenAI models reach the rules through a skill, never through the file name: on the commit message they opened the file in 1 of 112 runs, whichever of 12 names it had. Haiku 5.5 and Kimi K3 do the opposite: they seldom load a skill and open a file whose name matches the request. On the commit message, Anthropic models applied the rule in 5 of 30 runs with vcs.md and 27 of 30 with commit-rules.md. A name about commits then loses the request (merge request command, Opus 5.5 in OpenCode and Haiku 5.5 in Kiro: commit-rules.md 2/10, vcs.md 14/19), so no single name covers both.
Scope

Two changes, one per group of models. Neither adds text to the memory block.

1. Skill descriptions, aidd-vcs. The descriptions claim the partial requests and the merge, and keep the routings of today. 398 always-on characters against 439 today, each shorter than the one it replaces (192 against 232, 206 against 207).

 plugins/aidd-vcs/skills/01-commit/SKILL.md
-description: Create atomic conventional commits; safely retry scoped hook fixes; optionally push. Use when the user wants to commit changes, optionally pushing the branch. Not for amending, rebasing, opening a pull request, or tagging a release.
+description: Commit changes or write a commit message following the project's commit rules. Use when the user wants to commit or asks for a commit message. Not for amending, rebasing, or tagging a release.

 plugins/aidd-vcs/skills/02-pull-request/SKILL.md
-description: Create a draft pull or merge request from the current branch, in whatever VCS tool the project uses. Use when the user wants to open a pull or merge request. Not for committing, pushing, or merging a branch.
+description: Open or merge a pull or merge request per the project's request rules. Use when the user wants to open a request, write its title, description or command, or merge a branch. Not for committing or conflicts.

Length and the lean contract: since #1249 (make the lean skill contract hold on small models in skill-generate), check-skill.cjs caps a description at 150 characters in one sentence. The cap did not exist when these two were measured, and neither the shipped descriptions nor these meet it. Shorter wordings were tried on the trigger grid (330 runs): they lost requests on GPT-6 Luna whenever they dropped the partial requests or the project's rules (297 characters for both skills; 194 and 145 for 02-pull-request). How to fit 150 characters without losing that is open for discussion here.

  • 02-pull-request gets a merge action (actions/04-merge.md: read the target branch and the merge method from memory, merge, report), and its rule "Never commit, push, or branch here" becomes "Never push or branch here; commit only to finish a merge".
  • No step of the existing actions changes.

2. Memory file names, aidd-context:02-project-memory. vcs.md becomes two always-written files: commit-rules.md (platform, commits, what the agent may do) and pull-request-rules.md (branches, pull requests, merging).

  • The template, capabilities.md, ALWAYS_WRITTEN, the two skills above, the docs and the tests follow.
  • vcs.md stays an accepted name: a bank that holds it keeps passing check, and a refresh offers the split.

Measured effect. Share of runs that apply the rules, the four requests pooled (commit message, commit, merge request command, merge request description), memory as bare links. Short: the request alone. Long: the same request after about 340 000 tokens of unrelated material (Python standard library sources; 192 000 in Claude Code). "Change 1": the two descriptions above, with vcs.md kept.

Context Today Change 1 (descriptions) Changes 1 + 2 (descriptions + two files)
Short 57% 69% 86%
Long 18% 55% 73%

Pairs where all three variants ran: 6 short (Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3, MAI-Code in OpenCode, GPT-5.6 Luna in Codex), 7 long. On every pair, today against both changes: 44% and 90% short (19 pairs), 31% and 74% long (13 pairs).

Short context, by vendor:

Vendor Models Today Change 1 Changes 1 + 2
Anthropic Haiku 5.5 31% not run 89%
Anthropic Sonnet 5.5, Opus 5.5, Fable 5.2 49% not run 100%
OpenAI GPT-6 Luna, GPT-5.6 Luna 30% not run 92%
OpenAI GPT-6.1 Sol, GPT-5.6 Terra 31% not run 80%
Other vendors Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3 77% 85% 98%

Long context, by vendor:

Vendor Models Today Change 1 Changes 1 + 2
Anthropic Haiku 5.5 0% 25% (OpenCode only) 56%
Anthropic Sonnet 5.5, Opus 5.5 15% 69% (OpenCode only) 75%
OpenAI GPT-6 Luna, GPT-5.6 Luna 25% 63% 69%
OpenAI GPT-6.1 Sol 38% 100% 100%
Other vendors Kimi K3 0% 0% 75%
Other vendors Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7 83% not run 96%
Per host and model
Host, model Short: today Short: change 1 Short: changes 1 + 2 Long: today Long: change 1 Long: changes 1 + 2
Claude Code, Haiku 5.5 0% not run 100% 0% not run 63%
Claude Code, Sonnet 5.5 25% not run 100% 0% not run 75%
Claude Code, Fable 5.2 100% not run 100% not run not run not run
OpenCode, Haiku 5.5 25% not run 100% 0% 25% 50%
OpenCode, Sonnet 5.5 42% not run 100% 38% 63% 63%
OpenCode, Opus 5.5 52% not run 100% 0% 75% 88%
OpenCode, GPT-6.1 Sol 45% not run 83% 38% 100% 100%
OpenCode, GPT-6 Luna 27% not run 100% 25% 88% 75%
OpenCode, GPT-5.6 Luna 33% not run 83% 25% 38% 63%
OpenCode, GPT-5.6 Terra 33% not run 67% not run not run not run
OpenCode, Gemini 4 Argon 100% 100% 100% 100% not run 100%
OpenCode, Gemini 3.8 Flash 92% 100% 100% 75% not run 88%
OpenCode, Grok 4.7 92% 100% 100% 88% not run 100%
OpenCode, Kimi K3 25% 42% 92% 0% 0% 75%
OpenCode, MAI-Code-1.1-Flash 8% 25% 50% 0% (a) not run 25% (a)
Codex, GPT-5.6 Luna 20% 53% 70% not run not run not run
Kiro CLI, Haiku 5.5 71% not run 100% not run not run not run
VS Code, Sonnet 5.5 38% not run 100% not run not run not run
VS Code, GPT-6.1 Sol (b) 0% not run 100% not run not run not run
VS Code, Haiku 5.5 0% not run 38% not run not run not run

2 to 3 runs per request, so 4 to 21 runs per cell. (a) OpenCode compacted the context before every answer: MAI-Code has 128 000 tokens of input. (b) Without commit: its commit skill proposes a conforming subject and waits for an approval nobody gives.

  • Change 1 lifts the models that load skills: GPT-6.1 Sol, GPT-6 Luna, and Opus 5.5 under load (0% to 75%).
  • Change 2 lifts the models that open files by name: Kimi K3 goes from 42% with change 1 to 92% with both, and from 0% to 75% under load; Haiku 5.5 from 25% to 50% under load.
  • With an earlier, 232-character wording of change 1 alone, Haiku 5.5 in Kiro got worse on the merge request command (9/12 today, 3/8); with both changes the cell is 9/9.
  • Merge: with the merge action, 146 of 149 runs squash into staging (short, long and at 500 000 tokens); two did not merge and one merged into staging without squashing (MAI-Code under load); none into main.
  • Under load the gain holds and shrinks. What breaks is the commit message: Haiku 5.5, Sonnet 5.5 and Opus 5.5 answer it from habit with no call (1 of 6); the GPT-6 models keep loading the skill (4 of 4).
  • Models that already apply the rules (Gemini 4 Argon, Grok 4.7, Fable 5.2, Opus 5.5 in Claude Code) keep doing so with either change. Claude Code with the plugin's default wiring, which loads the bank at session start, gives 5 of 5 before and after.
  • Reasoning effort: each host ran at its own default, with no effort flag set. Codex used each model's default from its catalogue (low for GPT-6 Astra, medium for GPT-5.6 Luna and Terra). Claude Code inherited the effort of the session that launched it (xhigh). OpenCode, Kiro CLI and VS Code Copilot used their own defaults, which were not recorded.
  • Limits: one fixture, English prompts, 2 to 3 runs per cell. Most windows are about 1 000 000 tokens, so the long context fills a third of them; Grok 4.7 is filled to 0.87 of its 372 000 tokens of input. The Claude Code figures come from one user configuration (68 skills listed, memory wired as links).
Acceptance criteria
  • The two descriptions read as above, or a wording agreed in this issue that fits the lean contract and scores as well on the trigger grid and the four requests.
  • On the 15-prompt trigger grid (6 current uses, 5 exclusions, 4 new uses), every prompt that loads the right skill today still does, and "write the commit message", "give the command that creates the merge request", "write its description" and "merge this approved branch" load 01-commit or 02-pull-request on GPT-6 Luna and GPT-6.1 Sol.
  • 02-pull-request routes a merge to actions/04-merge.md, and on the fixture no run merges into the production branch.
  • A new bank holds commit-rules.md and pull-request-rules.md, no vcs.md, and the memory block lists them as bare links.
  • A bank that holds vcs.md passes check unchanged; a refresh offers the split and writes nothing unapproved.
  • grep -rn "vcs\.md" plugins docs returns only the accepted-name entries.
  • make check passes.
  • The PR reports the four-request score today and with both changes, near-empty and loaded, on Claude Code and on one link-loading host, with a small and a large model from two vendors.
Prior art in this repo
  • aidd_docs/tasks/2026_11/2026_11_06_project-memory-challenge/phase-1.md:86: "On Codex, Cursor, OpenCode and Copilot the memory reaches the model as links, so the file name alone decides whether a file is opened." That slice renamed the capability files; vcs.md was not reviewed.
  • plugins/aidd-context/skills/02-project-memory/references/capabilities.md:10 (the vcs.md row) and scripts/project-memory.cjs:18-25 (ALWAYS_WRITTEN, and ACCEPTED_NAMES, the existing way to keep an older name).
  • plugins/aidd-context/skills/04-skill-generate/references/skill-authoring.md (R2: the description is the only always-on text) and scripts/check-skill.cjs (150 characters, one sentence).
  • plugins/aidd-vcs/skills/01-commit/SKILL.md and 02-pull-request/SKILL.md: once loaded, both skills already send the agent to vcs.md.
  • #1164 (apply the new skill template to every skill): the two aidd-vcs descriptions are not converted yet; this change can land with or before it.
  • #1079 (test what a skill makes an agent do) and #1237 (portable multi-host skill evaluation): the fixture and the trigger grid of this measure are a case for them.
  • Method, figures per host and model, and the bench kit: they come with the draft PR (aidd_docs/tasks/2026_11/2026_11_16_vcs-rules/measure.md), and can be shared before on request.
Out of scope
  • A line after each link that says when to read the file, and loading the bank at session start. Both were measured and score higher (two files with such lines: 34 of 36 on Anthropic models; bank loaded at start: every run). Both add text to every session, so they are not proposed here.
  • Renaming any other memory file.
  • A separate merge skill. A prototype was measured (20 of 21); the action does as well without always-on text.
  • The steps of the two skills: once loaded they already apply the rule.
  • Small models that answer with no tool call at all (MAI-Code-1.1-Flash, GPT-5.6 at times, Haiku 5.5 in VS Code on text-only requests): no file name or description reaches them.
主要言語
TypeScript
スター
481
フォーク
45
平均マージ
1日 13時間
マージ済み PR(30日)
91

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ai-driven-dev/framework のほかの issue

ai-driven-dev/framework の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。