Add experiment-specific review guidance to AGENTS.md

Aperta
  • Qualcosa non va?
#45 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
55/100
Tipo di issue
Documentazione
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
documentation

Direzione di ricerca

The change belongs in AGENTS.md at the root of fullsend-ai/experiments, next to the existing experiment conventions (numbering, frontmatter, linting). The issue supplies a ready-made markdown block to add as a review-priorities section; context is in PR #30 and the linked review run log. Done means the section is added with the four priorities and the deprioritized items, and the next review runs on experiment PRs stop flagging naming or missing linked issues while raising at least one methodology or hypothesis observation. Check the open question about whether this guidance is still needed alongside fullsend-ai/fullsend#1722 before editing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature triaged

What happened

On PR #30, the review agent applied generic code-quality criteria to proof-of-concept experiment code. It produced some findings that were noise for this context (naming convention for _example suffix, "no linked issue" warning) while missing the most impactful concerns (methodology soundness of scorers, statistical validity of baselines). The review run log shows the agent was given no experiment-specific review guidance — the repo's AGENTS.md covers experiment conventions (numbering, frontmatter, linting) but contains no guidance about what to prioritize during code review of experiment/PoC code.

What could go better

The review agent treated this PoC experiment code the same as production code, spending attention on naming conventions and production-grade error handling while missing design-level methodology concerns that matter most for experiments. Experiment repos have different review priorities than production repos: methodology soundness, hypothesis validity, and reproducibility matter more than production-grade error handling or naming conventions. The existing AGENTS.md in fullsend-ai/experiments does not communicate these priorities to the review agent. Confidence is moderate: the current AGENTS.md may not have existed at the time of the June 9 review (the run log noted injecting an org-level AGENTS.md), but it exists now and still lacks review-specific guidance.

Proposed change

Add a review-specific section to fullsend-ai/experiments/AGENTS.md that guides the review agent's priorities for experiment code. Suggested content:

### Review priorities for experiment code

Experiments are proof-of-concept code. When reviewing, prioritize:

1. **Methodology soundness** — Do metrics, scorers, and evaluation criteria measure what they claim? Are statistical assumptions valid?
2. **Hypothesis alignment** — Does the implementation actually test the stated hypothesis?
3. **Reproducibility** — Can someone else run this experiment from the README instructions?
4. **Security** — No hardcoded secrets, credentials via env vars, sensitive data excluded via .gitignore.

Deprioritize for experiments:
- Production-grade error handling (try/except on every call)
- Naming conventions and code style beyond readability
- Missing linked issues (experiments may not use issue-driven workflows)

This is complementary to fullsend-ai/fullsend#1722 (repo-level review profiles), which is a platform feature. This proposal uses the existing AGENTS.md mechanism available today.

Validation criteria

After updating AGENTS.md, the next 2 review agent runs on experiment PRs in this repo should: (1) not flag naming conventions or missing linked issues as findings, and (2) include at least one methodology-level or hypothesis-alignment observation in their review output. Verify by comparing the review output against the updated guidance priorities.

Generated by retro agent from https://github.com/fullsend-ai/experiments/pull/30

Lingua principale
Python
Stelle
1
Fork
9
Merge medio
1g 13h
PR unite (30g)
7

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fullsend-ai/experiments

Tutte le issue di fullsend-ai/experiments

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.