[curriculum-eval] curriculum_assessment.py: checkpoint_quality — regex misses :white_check_mark: headings, false 0.0 scores
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 83/100
Research direction
Start with .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py and inspect CHECKPOINT_RE plus the checklist-item counter used for checkpoint_quality. Compare their behavior with the :white_check_mark: headings in the named workshop pages. Done when both heading styles are recognized, checklist items in the section are counted, and the five listed pages no longer receive missing_checkpoint flags.
Written by the indexing model from the issue text.
Description
File: .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py
Overall Score: 4.9 / 10.0 for the worst finding (corpus mean: 6.09)
Flagged Dimensions:
| Dimension | Score | Benchmark | Delta |
|---|---|---|---|
| checkpoint_quality | 0.0 | ≥ 4 items → 10 | −10.0 |
Root Cause (≤ 2 sentences):
CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint") only matches the emoji, while workshop files use the ## :white_check_mark: Checkpoint shortcode (required by AGENTS.md). All 5 worst-scored pages (e.g. 04-github-actions-intro.md, side-quest-01-02-environment-reference.md) have real checkpoints with 5–7 items but score 0.0, so missing_checkpoint flags are false positives.
Evidence (quoted from the file):
✅ Checkpoint
- You can identify
on,jobs, andstepsin a workflow file
Learning Science Rationale:
Retrieval-practice research (Roediger & Karpicke) supports end-of-step checkpoints, and the rubric should reward them. A metric that misses them mis-ranks content and sends remediation effort at pages that already follow the recommended pattern, undermining the evaluator's validity.
Improvement Prompt (for an agent):
In .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py, update CHECKPOINT_RE so it matches both "## ✅ Checkpoint" and "## :white_check_mark: Checkpoint" (case-insensitive). Also confirm the checklist-item counter for checkpoint_quality counts items inside that section. Re-run the assessment over workshop/*.md and confirm missing_checkpoint is no longer flagged for 04-github-actions-intro.md, 05-agentic-workflows-intro.md, side-quest-01-02-environment-reference.md, side-quest-11-06-anthropic-key.md and side-quest-17-07-repo-poisoning.md. Report the new corpus mean and the remaining lowest-scoring pages.
Expected Score After Fix: ~6.9 / 10.0 for the worst pages (checkpoint_quality 0 → 10 adds about +2.0 overall); corpus mean rises by a similar amount.
Generated by 🔬 Curriculum Quality Evaluator · copilot · auto · 21.2 AIC · ⌖ 0.741 AIC · ⊞ 10.4K · ◷
- expires on Oct 22, 2026, 4:43 AM UTC
- Dominant language
- JavaScript
- Stars
- 55
- Forks
- 36
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 18
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from githubnext/gh-aw-workshop
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 84/100
githubnext/gh-aw-workshop#4803 ·
Maintainers usually reply within 1 day
-
curriculum documentation quality
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
githubnext/gh-aw-workshop#4802 ·
Maintainers usually reply within 1 day
-
curriculum documentation quality
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
githubnext/gh-aw-workshop#4800 ·
Maintainers usually reply within 1 day
-
documentation guidelines
Difficulty 1/5 Under an hour Newbie friendliness 90/100
githubnext/gh-aw-workshop#4670 ·
Maintainers usually reply within 1 day
-
[guidelines] side-quest-11-05-event-triggers.md: [!IMPORTANT] used for non-blocking informationOpendocumentation guidelines
Difficulty 1/5 Under an hour Newbie friendliness 83/100
githubnext/gh-aw-workshop#4669 ·
Maintainers usually reply within 1 day
All issues in githubnext/gh-aw-workshop
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
bluerobotics/Cockpit-docs#48 ·
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PhyberApex/slds-json-schema-renderer#372 ·
Maintainers usually reply within 1 day
-
usage: a custom provider's balance meters ("…%" in the balance path) ignore the Left/Used switchOpen
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
[BUG] rag: sentence content can cross the reported sentence boundariesPossibly taken @LJK-ikun claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day