Proposal: ce-docs-audit — keep a repo's entry-point docs true (README, AGENTS.md, CLAUDE.md)

Abierto
#1,503 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
28/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
python

Línea de trabajo

Start by reading the existing ce-compound, ce-compound-refresh, ce-doc-review, and ce-handoff skills plus issue #866, then inspect validate-doc-claims.py for reusable claim checking. Use README.md, AGENTS.md, and CLAUDE.md in test repositories to define the audit checks and safe command policy. Done means maintainers have chosen the skill boundary and agreed how reporting, consent, and --report-only mode should work before implementation under tests/ and docs/skills/ begins.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

The gap

The plugin compounds knowledge about problems exceptionally well. It has no
skill that maintains the front door — README.md, AGENTS.md, CLAUDE.md —
against the current tree.

I checked the existing skills before writing this, and each is adjacent rather
than overlapping:

Skill Surface Why it does not cover this
ce-compound docs/solutions/ Captures one solved problem. Its Discoverability Check does touch AGENTS.md, but only to add a pointer to the knowledge store — it never checks whether anything already in those files is still true
ce-compound-refresh docs/solutions/ Refreshes solution docs for staleness. Same idea, different surface
ce-doc-review A requirements or plan document A document under construction, reviewed with role lenses. Entry-point docs are shipped artifacts with different failure modes
ce-handoff Session context Ephemeral by design, and correctly so. This is about what belongs in the repo permanently
ce-explain A teaching artifact Explicitly "for learning, not repo docs or verdicts"

So the gap is real and narrow: nobody audits the docs a newcomer actually reads
first.

Why it matters, with the case that prompted it

Entry-point docs rot differently from other docs, because the people who would
notice already know the answers. The cost lands entirely on the next person.

Yesterday I finished a week-long spike and wrote its README carefully. It contained
this setup step:

sprite file push -s <name> sprite/bootstrap.sh /tmp/bootstrap.sh

The second time I provisioned a machine, that command failed: a fresh Fly Sprite
has no /tmp, and the CLI refuses the path with directory /tmp does not exist
even though ls shows the directory. My own setup instructions were wrong, in a
document I had written two hours earlier, and no existing skill would have caught
it
— because it is not a stale reference or a missing section, it is a claim
about the world that only fails when someone follows it.

That is the highest-value check this skill would run, and I have not seen it
anywhere: treat every command in the docs as an assertion, and verify the ones
you safely can.

Proposed shape

Four checks, all reported even when they pass, because a silent pass is
indistinguishable from a skipped check.

1. Executable claims. Extract every command from the entry-point docs. Run the
cheap, safe, side-effect-free ones (--help, --version, --dry-run,
typechecks). For the rest, verify statically — does the referenced file exist, does
the flag appear in --help, does the script name exist in package.json — and say
plainly which could not be verified rather than implying they passed. A failing
command is the most valuable output.

2. State. Do the docs say what is deployed, what is half-done, and what to pick
up first with why it matters? State drifts faster than anything else in a README,
so check it against recent commits, open PRs, and whether the "coming soon" thing
shipped months ago.

3. Load-bearing decisions. Which decisions would a competent newcomer undo,
reasonably, and be wrong? The retry that looks paranoid, the async hop that could
obviously be an await, the function that looks dead. Then check whether the
reason is written where the person about to undo it would look. A reason that
lives only in a commit message is not documented
— nobody reads git log before
refactoring. This is the strongest argument for a repo having an AGENTS.md at
all, so proposing one is a legitimate finding.

4. Orientation. Is there a plain first sentence, a stated reading order, and do
the entry-point files point at each other rather than each assuming it is the only
one read? If both AGENTS.md and CLAUDE.md exist, is one a pointer rather than a
second copy that will drift?

Design choices worth arguing about before any code

  • Report before editing, and lead with what is wrong rather than what is
    missing.
    A broken command outranks an absent section. Absence is easy to see
    and easy to fix; the newcomer who hits a failing setup command in the first five
    minutes never reaches the missing section.
  • Smallest edit, never a rewrite. The job is a true front door, not a good
    one. A rewrite loses the author's voice, invites review nobody asked for, and
    buries the two lines that mattered.
  • Blocking consent before touching entry-point docs, matching
    ce-compound's treatment of instruction files. These are the repo's public face
    and frequently reviewed.
  • --report-only for CI, which is where I would expect most of the value over
    time: catch a broken setup command on the PR that broke it.

Relationship to #866

#866
(promote durable lessons onto enforced surfaces) is the closest existing issue and
it is solving a different half of the same problem, so worth naming rather than
letting a maintainer spot the overlap.

  • #866 asks whether a lesson is enforced. Its concern is that a captured lesson
    stays retrieval-dependent - it only changes behaviour if an agent happens to
    read it.
  • This asks whether what is written is still true. Its concern is a document
    that is read, confidently, and is wrong.

They compose rather than compete. Check 3 below - are load-bearing decisions
documented where someone about to undo them would look - is the weaker,
verification-side cousin of #866's promotion ladder. If #866 lands, this skill is
what later catches the moment a promoted rule stops matching the tree.

Open questions for maintainers

  1. Is this a skill or a phase of an existing one? A case exists for folding it
    into ce-compound-refresh as a second surface — it is the same "verify against
    the tree" motion. I lean separate, because the trigger is different (handing
    over or onboarding, not documenting a learning) and because --report-only in
    CI wants its own entry point. Happy to be wrong.
  2. How far should command execution go? Running commands from a document is
    real risk. My instinct is a conservative allowlist — no writes, no network
    mutations, no deploys — and static verification for everything else. There may
    be an existing convention in the repo I should follow instead.
  3. Naming. ce-docs-audit reads clearly; ce-docs is terser but sounds like
    it generates docs, which is the opposite of what it does.
  4. Overlap to resolve deliberately: ce-compound's Discoverability Check also
    edits AGENTS.md. If both exist, one of them should own instruction-file edits.
    I would give it to this skill and have ce-compound keep only its pointer
    insertion, but that is a maintainer call.

I ran it, and it caught this

Not a design sketch. I built the local version, then pointed it at the same
repository whose docs I had written the previous day, and it found four false
commands out of nine.

Executable claims: 9 checked, 4 fail

FAIL  sprite file push -s <name> sprite/bootstrap.sh /tmp/bootstrap.sh
FAIL  sprite file push -s <name> sprite/agent-server.mjs /tmp/agent-server.mjs
FAIL  sprite exec -s <name> --env GITHUB_TOKEN=... -- bash /tmp/bootstrap.sh
      Error: directory /tmp does not exist

FAIL  cd worker && npm install && npx wrangler deploy
      Wrangler requires at least Node.js v22.0.0. You are using v20.20.0.

The other three checks passed - the state section was current, all five
load-bearing decisions were documented with the cost of reverting them, and the
reading order was clear. Thirteen of thirteen referenced files and doc links
resolved.

The cause is the argument for the skill. AGENTS.md in that same repository
recorded the /tmp trap correctly, on the same day, in a section literally titled
"three things that will waste your time otherwise". The README still told you to do
the broken thing. Two documents, one truth, and nothing keeping them agreed.

The Node 22 failure has the same shape: the constraint was written in one file and
the command that violates it was in the other.

Neither is a stale link, a missing section, or a dead reference - the classes of
problem existing tooling finds. Both are commands that only fail when a newcomer
runs them, which is exactly when it is most expensive.

What I have already

A working local version, used on two repositories, and the concrete case above. If
this is worth having I am happy to do the work to the repo's standards — tests
under tests/, a doc under docs/skills/, the headless/lightweight mode split,
and the validate-doc-claims.py mechanical check reused rather than reimplemented.

If the answer is "fold it into ce-compound-refresh" or "not for this plugin",
that is a useful answer too and I would rather hear it before writing anything.

Lenguaje dominante
TypeScript
Estrellas
25.6k
Forks
2.1k
Merge medio
3 h 57 min
PR fusionados (30 d)
170

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de EveryInc/compound-engineering-plugin

Todos los issues de EveryInc/compound-engineering-plugin

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.