[curriculum-eval] curriculum_assessment.py: checkpoint_quality — regex misses :white_check_mark: and :clipboard: headings, penalising all pages

未关闭 适合新手
  • 哪里不对？
#5,437 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
77/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
github-actions, python
领域
testing, tooling

调研方向

从 .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py 开始，尤其检查 CHECKPOINT_RE 以及它用于检测先决条件和 activity:checkpoint 的逻辑；查看现有的单元测试。更新匹配模式，使其能够识别包含字面 emoji 和 GitHub shortcode 的标题，同时保留 ## Prerequisites，然后对 workshop/*.md 运行评估。完成标准是检测到匹配的标题并清除错误的 missing_checkpoint 标记；不要编辑 workshop 内容。

由索引模型根据 Issue 内容生成。

描述

curriculum documentation quality

File: .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py
Overall Score: corpus mean 6.06 / 10.0 (threshold 5.52)

Flagged Dimensions:

Dimension Score Benchmark Delta
checkpoint_quality 0.0 on 92 of 94 pages ≥ 4 items → 10 −10
scaffolding 5.0 on 94 of 94 pages 10 when "Before You Start" present −5

Root Cause (≤ 2 sentences):
CHECKPOINT_RE = r"##\s+✅\s*Checkpoint" and the prereq regex ##\s+📋\s*Before You Start match only literal emoji, but workshop files use GitHub shortcodes (## :white_check_mark: Checkpoint, ## :clipboard: Before You Start). Every page loses about 2.0 × 10 (checkpoint) and 1.5 × 5 (scaffolding) weighted points, so scores cluster at 5–7 and the "missing_checkpoint" flag is meaningless.

Evidence (quoted from the file):

CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE) (the page workshop/side-quest-11-06-anthropic-key.md has ## :white_check_mark: Checkpoint with 5 checklist items, yet has_checkpoint is false)

Learning Science Rationale:
Valid measurement is a precondition for acting on Bloom's-aligned assessment and cognitive-load findings: retrieval-practice checkpoints are a key scaffolding feature (Roediger and Karpicke's testing effect), and a rubric that scores them as absent on every page cannot discriminate good from poor steps. The false negatives also hide real gaps and produce spurious trend swings (for example the −2.00 "declines" on side-quest-11-09-agent-session-phases.md and side-quest-12-01-iterate-agent-output.md).

Improvement Prompt (for an agent):

In .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py, update CHECKPOINT_RE and the has_prereq_section regexes so they accept both the literal emoji and the GitHub shortcode forms: "## ✅ Checkpoint" / "## :white_check_mark: Checkpoint", and "## 📋 Before You Start" / "## :clipboard: Before You Start" (also keep "## Prerequisites"). Also update the Bloom "activity:checkpoint" detection that uses CHECKPOINT_RE. Add or update unit tests if present, rerun the assessment over workshop/*.md, and confirm has_checkpoint is true for pages with a Checkpoint heading and that the corpus mean rises accordingly. Do not edit workshop content.

Expected Score After Fix: ~7.5 / 10.0 corpus mean (checkpoint_quality and scaffolding recovered on most pages)

Generated by 🔬 Curriculum Quality Evaluator · copilot · auto · ~46 AIC · ⌖ ~1 AIC · ⊞ 5K · ◷

  • expires on Nov 8, 2026, 3:30 AM UTC
主要语言
JavaScript
星标
58
派生
40
平均合并
4 天 4 小时
30 天内合并 PR
8

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

githubnext/gh-aw-workshop 的其他 Issue

查看 githubnext/gh-aw-workshop 的全部 Issue

相似的 Issue

更多 JavaScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。