Chapter 8: Hallucination example and explanation conflates two techniques - incorrectly attributes it to the wrong technique
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- jupyter-notebook
- Domain
- ai, documentation
Research direction
Start with the Chapter 8 lesson and inspect the hallucination example and its two prompt modifications. Compare scratchpad-only, metacognitive-only, and combined versions, then revise the explanation so the techniques are tested independently or their separate contributions are clearly acknowledged.
Written by the indexing model from the issue text.
Description
Issue Summary
Chapter 8's main example conflates two distinct prompt modifications and incorrectly attributes hallucination reduction to the scratchpad technique, when the metacognitive instruction appears to be the primary (or sole) effective component.
The Problem
The lesson states:
"a great way to reduce hallucinations on long documents is to make Claude gather evidence first. In this case, we tell Claude to first extract relevant quotes, then base its answer on those quotes."
However, the "improved" prompt makes two simultaneous changes:
Adding scratchpad for evidence gathering
Adding explicit metacognitive instruction: "consider whether it answers the user's question or whether it lacks sufficient detail"
Experimental Findings
Testing reveals that:
❌ Scratchpad alone (without metacognitive instruction): Still hallucinates
✅ Metacognitive instruction alone (without scratchpad): Prevents hallucination
✅ Both combined: Prevents hallucination
This suggests the metacognitive instruction - which essentially gives Claude permission to acknowledge insufficient information - is doing the heavy lifting, not the scratchpad mechanism, which is contrary to the the lesson/text.
Why This Matters
This is a confounding variable problem. The tutorial teaches that scratchpad/evidence-gathering prevents hallucinations, but doesn't isolate this variable. The actual effective technique appears to be explicitly prompting the model to evaluate information sufficiency, which is a well-known hallucination mitigation strategy distinct from chain-of-thought reasoning.
Suggested Fix
The lesson should either:
Test each technique independently to demonstrate their individual effects
Acknowledge both components and clarify that the metacognitive check (giving an "out") may be more critical than the evidence-gathering structure
Provide a cleaner example where the scratchpad technique can be isolated and its specific benefits demonstrated
- Dominant language
- Jupyter Notebook
- Stars
- 39.5k
- Forks
- 4.4k
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from anthropics/prompt-eng-interactive-tutorial
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
anthropics/prompt-eng-interactive-tutorial#101 · 2 comments · 3 reactions ·
-
Tutorial using Haiku 3- needs updateMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
anthropics/prompt-eng-interactive-tutorial#93 · 3 comments · 2 reactions ·
-
04_Separating_Data_and_Instructions.ipynb, panel [3]: incorrect description of response.May be free again A pull request for this issue was closed without being merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
handling spelled as handilgj in 10.2_Appendix_Tool UseMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
anthropics/prompt-eng-interactive-tutorial#6 · 3 comments · 2 reactions ·
All issues in anthropics/prompt-eng-interactive-tutorial
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Validate the Ollama AI provider with a real serverPossibly taken @visy-ani claimed this today. Opengood first issue help wanted
Difficulty 2/5 Half a day Newbie friendliness 70/100
Rullst/Rullst#812 · 1 comment ·
Maintainers usually reply within 1 day
-
Expose session state keys on the Go todo and agent-mode providersPossibly taken @gabisonia claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/agent-framework-go#1412 ·
Maintainers usually reply within 1 day
-
[Bug]: hf_model_if_cache retry decorator lacks reraise=True, surfaces RetryError instead of the real exceptionPossibly taken @Shizoqua claimed this today. Openbug triage
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
area: agents-mcp priority: normal
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day