fix(aidd-vcs): commit and merge request rules in project memory are rarely applied

Ouverte
#1,258 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Fonctionnalité
Clarté
Clairement spécifiée
Activité
Active
Stack technique
github, typescript

Piste de recherche

Start with plugins/aidd-vcs/skills/01-commit/SKILL.md, plugins/aidd-vcs/skills/02-pull-request/SKILL.md, and plugins/aidd-context/skills/02-project-memory/references/capabilities.md; then trace ALWAYS_WRITTEN and accepted names in scripts/project-memory.cjs. Check the existing trigger grid and project memory tests before changing descriptions or file names. Done means the acceptance criteria pass, including make check, the legacy vcs.md check and refresh behavior, and the reported four-request measurements.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Problem

A project writes its commit format, its branch naming and its merge rules in aidd_docs/memory/vcs.md. On a host that receives memory as links (the AGENTS.md block), an agent applies them only if it opens that file or loads a skill that sends it there. Most of the time it does neither.

Measured on a fixture whose rules exist nowhere else (commit subject <area>/<verb>: <summary>, merge requests into staging, squash merge): about 4 200 headless runs with bare links, on Claude Code, OpenCode, Kiro CLI, Codex and VS Code Copilot, with Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.2, GPT-6.1 Sol, GPT-6 Luna, GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3 and MAI-Code-1.1-Flash.

  • Four requests (write a commit message, commit, give the command that creates the merge request, write its description): rules applied in 97 of 222 runs today, on the 19 host and model pairs of the table below.
  • A loaded skill does the job: rules applied in 164 of 168 command-line runs where 01-commit or 02-pull-request loaded (a commit skill that read the rule and stopped for approval counts as applied). Their descriptions only claim "commit changes" and "open a pull or merge request", so on the ten command-line pairs measured first a skill loaded on a plain "commit" (21 of 28 runs) and never on the three other requests (0 of 108).
  • No skill covers a merge. Asked to merge an approved branch, 3 of 47 runs merged into main against the project's rule.
  • Models split in two groups. OpenAI models reach the rules through a skill, never through the file name: on the commit message they opened the file in 1 of 112 runs, whichever of 12 names it had. Haiku 5.5 and Kimi K3 do the opposite: they seldom load a skill and open a file whose name matches the request. On the commit message, Anthropic models applied the rule in 5 of 30 runs with vcs.md and 27 of 30 with commit-rules.md. A name about commits then loses the request (merge request command, Opus 5.5 in OpenCode and Haiku 5.5 in Kiro: commit-rules.md 2/10, vcs.md 14/19), so no single name covers both.
Scope

Two changes, one per group of models. Neither adds text to the memory block.

1. Skill descriptions, aidd-vcs. The descriptions claim the partial requests and the merge, and keep the routings of today. 398 always-on characters against 439 today, each shorter than the one it replaces (192 against 232, 206 against 207).

 plugins/aidd-vcs/skills/01-commit/SKILL.md
-description: Create atomic conventional commits; safely retry scoped hook fixes; optionally push. Use when the user wants to commit changes, optionally pushing the branch. Not for amending, rebasing, opening a pull request, or tagging a release.
+description: Commit changes or write a commit message following the project's commit rules. Use when the user wants to commit or asks for a commit message. Not for amending, rebasing, or tagging a release.

 plugins/aidd-vcs/skills/02-pull-request/SKILL.md
-description: Create a draft pull or merge request from the current branch, in whatever VCS tool the project uses. Use when the user wants to open a pull or merge request. Not for committing, pushing, or merging a branch.
+description: Open or merge a pull or merge request per the project's request rules. Use when the user wants to open a request, write its title, description or command, or merge a branch. Not for committing or conflicts.

Length and the lean contract: since #1249 (make the lean skill contract hold on small models in skill-generate), check-skill.cjs caps a description at 150 characters in one sentence. The cap did not exist when these two were measured, and neither the shipped descriptions nor these meet it. Shorter wordings were tried on the trigger grid (330 runs): they lost requests on GPT-6 Luna whenever they dropped the partial requests or the project's rules (297 characters for both skills; 194 and 145 for 02-pull-request). How to fit 150 characters without losing that is open for discussion here.

  • 02-pull-request gets a merge action (actions/04-merge.md: read the target branch and the merge method from memory, merge, report), and its rule "Never commit, push, or branch here" becomes "Never push or branch here; commit only to finish a merge".
  • No step of the existing actions changes.

2. Memory file names, aidd-context:02-project-memory. vcs.md becomes two always-written files: commit-rules.md (platform, commits, what the agent may do) and pull-request-rules.md (branches, pull requests, merging).

  • The template, capabilities.md, ALWAYS_WRITTEN, the two skills above, the docs and the tests follow.
  • vcs.md stays an accepted name: a bank that holds it keeps passing check, and a refresh offers the split.

Measured effect. Share of runs that apply the rules, the four requests pooled (commit message, commit, merge request command, merge request description), memory as bare links. Short: the request alone. Long: the same request after about 340 000 tokens of unrelated material (Python standard library sources; 192 000 in Claude Code). "Change 1": the two descriptions above, with vcs.md kept.

Context Today Change 1 (descriptions) Changes 1 + 2 (descriptions + two files)
Short 57% 69% 86%
Long 18% 55% 73%

Pairs where all three variants ran: 6 short (Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3, MAI-Code in OpenCode, GPT-5.6 Luna in Codex), 7 long. On every pair, today against both changes: 44% and 90% short (19 pairs), 31% and 74% long (13 pairs).

Short context, by vendor:

Vendor Models Today Change 1 Changes 1 + 2
Anthropic Haiku 5.5 31% not run 89%
Anthropic Sonnet 5.5, Opus 5.5, Fable 5.2 49% not run 100%
OpenAI GPT-6 Luna, GPT-5.6 Luna 30% not run 92%
OpenAI GPT-6.1 Sol, GPT-5.6 Terra 31% not run 80%
Other vendors Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3 77% 85% 98%

Long context, by vendor:

Vendor Models Today Change 1 Changes 1 + 2
Anthropic Haiku 5.5 0% 25% (OpenCode only) 56%
Anthropic Sonnet 5.5, Opus 5.5 15% 69% (OpenCode only) 75%
OpenAI GPT-6 Luna, GPT-5.6 Luna 25% 63% 69%
OpenAI GPT-6.1 Sol 38% 100% 100%
Other vendors Kimi K3 0% 0% 75%
Other vendors Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7 83% not run 96%
Per host and model
Host, model Short: today Short: change 1 Short: changes 1 + 2 Long: today Long: change 1 Long: changes 1 + 2
Claude Code, Haiku 5.5 0% not run 100% 0% not run 63%
Claude Code, Sonnet 5.5 25% not run 100% 0% not run 75%
Claude Code, Fable 5.2 100% not run 100% not run not run not run
OpenCode, Haiku 5.5 25% not run 100% 0% 25% 50%
OpenCode, Sonnet 5.5 42% not run 100% 38% 63% 63%
OpenCode, Opus 5.5 52% not run 100% 0% 75% 88%
OpenCode, GPT-6.1 Sol 45% not run 83% 38% 100% 100%
OpenCode, GPT-6 Luna 27% not run 100% 25% 88% 75%
OpenCode, GPT-5.6 Luna 33% not run 83% 25% 38% 63%
OpenCode, GPT-5.6 Terra 33% not run 67% not run not run not run
OpenCode, Gemini 4 Argon 100% 100% 100% 100% not run 100%
OpenCode, Gemini 3.8 Flash 92% 100% 100% 75% not run 88%
OpenCode, Grok 4.7 92% 100% 100% 88% not run 100%
OpenCode, Kimi K3 25% 42% 92% 0% 0% 75%
OpenCode, MAI-Code-1.1-Flash 8% 25% 50% 0% (a) not run 25% (a)
Codex, GPT-5.6 Luna 20% 53% 70% not run not run not run
Kiro CLI, Haiku 5.5 71% not run 100% not run not run not run
VS Code, Sonnet 5.5 38% not run 100% not run not run not run
VS Code, GPT-6.1 Sol (b) 0% not run 100% not run not run not run
VS Code, Haiku 5.5 0% not run 38% not run not run not run

2 to 3 runs per request, so 4 to 21 runs per cell. (a) OpenCode compacted the context before every answer: MAI-Code has 128 000 tokens of input. (b) Without commit: its commit skill proposes a conforming subject and waits for an approval nobody gives.

  • Change 1 lifts the models that load skills: GPT-6.1 Sol, GPT-6 Luna, and Opus 5.5 under load (0% to 75%).
  • Change 2 lifts the models that open files by name: Kimi K3 goes from 42% with change 1 to 92% with both, and from 0% to 75% under load; Haiku 5.5 from 25% to 50% under load.
  • With an earlier, 232-character wording of change 1 alone, Haiku 5.5 in Kiro got worse on the merge request command (9/12 today, 3/8); with both changes the cell is 9/9.
  • Merge: with the merge action, 146 of 149 runs squash into staging (short, long and at 500 000 tokens); two did not merge and one merged into staging without squashing (MAI-Code under load); none into main.
  • Under load the gain holds and shrinks. What breaks is the commit message: Haiku 5.5, Sonnet 5.5 and Opus 5.5 answer it from habit with no call (1 of 6); the GPT-6 models keep loading the skill (4 of 4).
  • Models that already apply the rules (Gemini 4 Argon, Grok 4.7, Fable 5.2, Opus 5.5 in Claude Code) keep doing so with either change. Claude Code with the plugin's default wiring, which loads the bank at session start, gives 5 of 5 before and after.
  • Reasoning effort: each host ran at its own default, with no effort flag set. Codex used each model's default from its catalogue (low for GPT-6 Astra, medium for GPT-5.6 Luna and Terra). Claude Code inherited the effort of the session that launched it (xhigh). OpenCode, Kiro CLI and VS Code Copilot used their own defaults, which were not recorded.
  • Limits: one fixture, English prompts, 2 to 3 runs per cell. Most windows are about 1 000 000 tokens, so the long context fills a third of them; Grok 4.7 is filled to 0.87 of its 372 000 tokens of input. The Claude Code figures come from one user configuration (68 skills listed, memory wired as links).
Acceptance criteria
  • The two descriptions read as above, or a wording agreed in this issue that fits the lean contract and scores as well on the trigger grid and the four requests.
  • On the 15-prompt trigger grid (6 current uses, 5 exclusions, 4 new uses), every prompt that loads the right skill today still does, and "write the commit message", "give the command that creates the merge request", "write its description" and "merge this approved branch" load 01-commit or 02-pull-request on GPT-6 Luna and GPT-6.1 Sol.
  • 02-pull-request routes a merge to actions/04-merge.md, and on the fixture no run merges into the production branch.
  • A new bank holds commit-rules.md and pull-request-rules.md, no vcs.md, and the memory block lists them as bare links.
  • A bank that holds vcs.md passes check unchanged; a refresh offers the split and writes nothing unapproved.
  • grep -rn "vcs\.md" plugins docs returns only the accepted-name entries.
  • make check passes.
  • The PR reports the four-request score today and with both changes, near-empty and loaded, on Claude Code and on one link-loading host, with a small and a large model from two vendors.
Prior art in this repo
  • aidd_docs/tasks/2026_11/2026_11_06_project-memory-challenge/phase-1.md:86: "On Codex, Cursor, OpenCode and Copilot the memory reaches the model as links, so the file name alone decides whether a file is opened." That slice renamed the capability files; vcs.md was not reviewed.
  • plugins/aidd-context/skills/02-project-memory/references/capabilities.md:10 (the vcs.md row) and scripts/project-memory.cjs:18-25 (ALWAYS_WRITTEN, and ACCEPTED_NAMES, the existing way to keep an older name).
  • plugins/aidd-context/skills/04-skill-generate/references/skill-authoring.md (R2: the description is the only always-on text) and scripts/check-skill.cjs (150 characters, one sentence).
  • plugins/aidd-vcs/skills/01-commit/SKILL.md and 02-pull-request/SKILL.md: once loaded, both skills already send the agent to vcs.md.
  • #1164 (apply the new skill template to every skill): the two aidd-vcs descriptions are not converted yet; this change can land with or before it.
  • #1079 (test what a skill makes an agent do) and #1237 (portable multi-host skill evaluation): the fixture and the trigger grid of this measure are a case for them.
  • Method, figures per host and model, and the bench kit: they come with the draft PR (aidd_docs/tasks/2026_11/2026_11_16_vcs-rules/measure.md), and can be shared before on request.
Out of scope
  • A line after each link that says when to read the file, and loading the bank at session start. Both were measured and score higher (two files with such lines: 34 of 36 on Anthropic models; bank loaded at start: every run). Both add text to every session, so they are not proposed here.
  • Renaming any other memory file.
  • A separate merge skill. A prototype was measured (20 of 21); the action does as well without always-on text.
  • The steps of the two skills: once loaded they already apply the rule.
  • Small models that answer with no tool call at all (MAI-Code-1.1-Flash, GPT-5.6 at times, Haiku 5.5 in VS Code on text-only requests): no file name or description reaches them.
Langage dominant
TypeScript
Étoiles
481
Forks
45
Merge moyen
1 j 13 h
PR mergées (30 j)
91

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ai-driven-dev/framework

Toutes les issues de ai-driven-dev/framework

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.