[workshop-sim] Workshop Simulation Report — 2026-11-21 (Run #252000, 1000×Monte Carlo)

Aperta
#5,975 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
20/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github, github-actions

Direzione di ricerca

This is a generated simulation report, not a task. The lesson pages it cites are 07-your-first-workflow.md, 04-github-actions-intro.md, 05-agentic-workflows-intro.md and 08-run-your-workflow.md, and the checkpoint rubric it blames is in the same repo. Before editing anything, check whether the ## Checkpoint sections are really missing or the rubric is mismatched, since the report itself flags that as unverified. Done would mean a maintainer has confirmed which repair applies and the change is scoped to specific lesson files.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feedback simulation workshop
Overview
  • Date: 2026-11-21
  • Students simulated: 46 × 1000 Monte Carlo runs
  • Workshop steps available: 32/32
  • Overall success rate: 23.1% (95% Monte Carlo interval: 22.8%–23.5%)
  • Highest-dropout step: 07-first-workflow (22.7% conditional dropout among 22,198 at-risk runs; 95% Monte Carlo interval: 22.2%–23.3%)
  • Lowest curriculum quality step: 04-github-actions-intro.md (overall score 5.39/10; tied with 05-agentic-workflows-intro.md)
  • Learning KPI index: 2.9/10 (active_learning 4.2 · checkpoint_quality 0.0 · scaffolding 5.0)
  • Model: 2026-07-survival-model-v2 / 2026-07-assumption-model-v2 (parameter hash 2024391902)
  • Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Note: the baseline replay showed 08-run-your-workflow at 100% dropout (workflow-not-compiled). After semantic evaluation of Step 7/7d (source, compile, push, and both billing paths are covered with evidence), that artifact disappeared; Step 8 is no longer a top hotspot. Selected steps by semantic risk: 08-run-your-workflow, 07-first-workflow, 05-agentic-intro (all insights verified, hashes match).

Part Summary
Part Files Mean Score Std Dev
Part 1 — core path (lessons 00–14) 15 6.70 / 10.0 ±1.74
Part 2 — advanced (lessons 15+) 17 6.03 / 10.0 ±0.35
Overall corpus 32 6.35 / 10.0 —

No pages are classified as other.

Critical Findings
  1. Part 1 early-concept steps dominate dropout: 04-actions-intro (16.4%), 05-agentic-intro (18.6%), 05c (12.7%), 05b (9.9%) — mostly beginner and github-basic learners (beginner mean success 0.5%, github-basic 15.4%).
  2. Step 7 (22.7%) is the top single-step dropout; 3,897 of 5,048 failures are copilot-access-missing. Part 1 repair.
  3. Learning quality health is weak: KPI index 2.9/10, driven by checkpoint_quality = 0.0 on every step (the rubric detects no checkpoints even where pages have ## Checkpoint sections — likely a rubric/format mismatch worth verifying) and scaffolding flat at 5.0.
  4. UI-preferred learners succeed far less (11.8% vs 33.6%); tool: CCA learners 8.4% vs CLI 35.5% — the Codespace terminal transition is the largest preference-driven gap.
Top Repairs to Prioritize

Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index.

  1. Step 7: unify the Copilot access check into a single gate and make the billing decision a guided choice (completion ↑ · learning KPI ↑)
  2. Steps 4–5: split concept density with prediction/practice checkpoints (completion ↑ · learning KPI ↑)
  3. Step 5: remove the "open any .lock.yml" assumption before a lock file exists; add real checkpoints (completion ↑ · learning KPI ↑)
Dropout by step
Step At-risk runs Dropouts Rate 95% interval Mode Top reason
07-first-workflow 22,198 5,048 22.7% 22.2–23.3% access barrier Copilot model access missing
05-agentic-intro 36,948 6,864 18.6% 18.2–19.0% learning barrier Agentic concept gap
04-actions-intro 44,197 7,249 16.4% 16.1–16.7% learning barrier Concept overload
05c-agentic-practice 30,084 3,815 12.7% 12.3–13.1% learning barrier Cannot classify agentic vs standard tasks
05b-agentic-security 26,269 2,604 9.9% 9.6–10.3% learning barrier Security concept gap
06-install-gh-aw 23,665 1,467 6.2% 5.9–6.5% access barrier Extension install friction
17-add-mcp-tools 14,753 694 4.7% 4.4–5.1% learning barrier MCP tooling friction
18-share-and-reuse 14,059 563 4.0% 3.7–4.3% learning barrier Workflow reuse friction
Curriculum quality and learning KPIs
File Overall Active Checkpoint Scaffold KPI Lowest dim Priority
04-github-actions-intro.md 5.39 3.9 0.0 5.0 2.8 checkpoint_quality High
05-agentic-workflows-intro.md 5.39 2.3 0.0 5.0 2.3 checkpoint_quality High
05b-agentic-workflows-security.md 5.75 2.5 0.0 5.0 2.3 checkpoint_quality Medium
05c-agentic-workflows-practice.md 6.22 6.2 0.0 5.0 3.6 checkpoint_quality Medium
07-your-first-workflow.md 6.25 6.3 0.0 5.0 3.7 checkpoint_quality High
07d-confirm-model-access.md 6.51 6.3 0.0 5.0 3.7 checkpoint_quality High
08-run-your-workflow.md 5.67 3.0 0.0 5.0 2.4 checkpoint_quality Medium
06-install-gh-aw.md 10.0 3.5 0.0 5.0 2.6 checkpoint_quality Low
Cohort mean 6.35 4.2 0.0 5.0 2.9 checkpoint_quality —
Segment breakdowns
Level Mean success n
beginner 0.5% 11
github-basic 15.4% 19
actions-user 47.2% 11
advanced 49.5% 5
Personality Mean success n
impatient 26.6% 6
methodical 26.2% 12
skeptical 22.8% 7
confused 21.1% 6
curious 20.2% 15

UI preferred: true 11.8% (22) vs false 33.6% (24). Tool: cli 35.5%, vscode 30.1%, CCA 8.4%.

Notable student journeys (3)
  • Surprising success: Learner 004 (actions-user, backend-dev, vscode, UI-preferred) reaches 62% despite UI preference — Actions familiarity offsets the terminal transition; its main failure is still Step 7.
  • Unexpected dropout: Learner 024 (advanced, devops) tops out at 59.6% and fails at Step 7 — expertise doesn't remove model-access/billing friction.
  • Content gap: Learner 001 (github-basic, UI-preferred, vscode) 0% — drops at Step 4 from concept overload, with no checkpoint to consolidate before Step 5.

Generated by 🔬 Workshop Student Simulator · copilot · auto · ~38 AIC · ⌖ ~1 AIC · ⊞ 15.4K · ◷

  • expires on Nov 22, 2026, 11:03 AM UTC
Lingua principale
JavaScript
Stelle
61
Fork
43
Merge medio
10h 38m
PR unite (30g)
12

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di githubnext/gh-aw-workshop

Tutte le issue di githubnext/gh-aw-workshop

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.