Add unit tests for lint scripts in hack/

Aperta
  • Qualcosa non va?
#38 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
github-actions, python
Ambito
testing

Direzione di ricerca

Start with extract_h1 and validate_experiment in hack/lint-experiment-frontmatter, then skim hack/lint-experiment-numbers and hack/lint-experiment-index. Run the frontmatter script on a sample experiment file that has a # YAML comment inside its frontmatter. Done means a pytest file such as hack/test_lint_scripts.py covers the listed cases, includes a regression test for the frontmatter-skipping fix in commit 05bd450e, and runs in CI.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement feature ready-for-triage triaged

What happened

On PR #36, the review agent caught a bug in the newly added extract_h1() function in hack/lint-experiment-frontmatter: the function did not skip YAML frontmatter when searching for H1 headings, meaning a YAML comment starting with # inside the frontmatter block could be misidentified as the document's H1 heading. The author fixed this in commit 05bd450e by adding frontmatter-skipping logic. The three lint scripts (lint-experiment-frontmatter, lint-experiment-numbers, lint-experiment-index) have zero test coverage — there are no test files in hack/ or elsewhere for these scripts.

What could go better

The frontmatter-skipping bug was only caught because the review agent reasoned about the code's behavior. A unit test with a frontmatter block containing a comment line (e.g., # this is a YAML comment) would have caught this deterministically during development, before the PR was even opened. As these lint scripts enforce core repo conventions and are modified when conventions evolve (as happened in this PR), they are high-value targets for test coverage. Without tests, future modifications to these scripts rely entirely on code review — human or agent — to catch regressions.

Proposed change

Add a test file (e.g., hack/test_lint_scripts.py) with pytest-based unit tests covering at minimum: (1) extract_h1 correctly skips YAML frontmatter and does not match comments inside it, (2) extract_h1 handles files without frontmatter, (3) H1/title mismatch detection in validate_experiment, (4) missing required frontmatter fields, (5) invalid status values, (6) topics field type validation. Wire the tests into CI via a GitHub Actions step or pre-commit hook. The lint-experiment-frontmatter script already uses standard Python with pyyaml as the only dependency, so tests should be straightforward.

Validation criteria

Tests exist for all three lint scripts covering at least the cases listed above. Tests run in CI on every PR. The specific frontmatter-skipping regression (YAML comment matching as H1) has a dedicated test case that would fail if the extract_h1 fix from commit 05bd450e were reverted.

Generated by retro agent from https://github.com/fullsend-ai/experiments/pull/36

Lingua principale
Python
Stelle
1
Fork
9
Merge medio
1g 13h
PR unite (30g)
7

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fullsend-ai/experiments

Tutte le issue di fullsend-ai/experiments

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.