Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Task] Add an illustrative worked example and unrun evaluation cases

Open
#35 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
github, python

Research direction

Start with the existing adopter templates and roles, evidence page, and navigation/conformance checks; the issue names nine owned paths and a specific base revision. Follow the acceptance checklist, keeping reference fixtures distinct from live evaluations and leaving all six cases NOT_RUN. Done means the documented local commands and required checks pass, the protected payload remains unchanged, and the candidate’s exact HEAD/tree and independent review are recorded; acceptance and merge remain pending.

Written by the indexing model from the issue text.

Description

Problem and expected behavior

After Task #33 / PR #34, the owner asked for self-checking and continued polish. The current PR remains technically green at its exact head, without unresolved review threads; final acceptance/merge is still a human decision. Continue the previously proposed public worked-example and evaluation-case documentation, without running a live agent experiment or changing shipped behavior.

Third-party readers need a small reproducible local example of the Issue -> plan -> bounded implementation -> verification -> human-decision workflow, and a concrete template for future evaluations. Illustrative records and locally tested reference code must not masquerade as live GitHub/Codex evidence or completed evaluations.

Plan, dependency and ownership

  • Exact base: bda21a5e2db733be5bb342f8b31e8d83456bb4d7, tree 8c33247049d546f50ac1e2685875d180e5c5869f, the unmerged PR #34 candidate (not accepted main).
  • Branch: codex/public-example-evaluation. Stack the new PR on codex/adoption-evidence-polish; disclose dependency on PR #34. Work may proceed before that merge, but integration cannot imply its approval. Do not merge, retarget or close either PR/Task without the applicable authority and exact-head recheck.
  • One active writer: current coordinating Codex session, one public-source checkout. Bounded documentation-task exemption: supervision and implementation are combined; no worker will be spawned for implementation. Read-only independent reviewers may inspect the completed candidate. Overlapping navigation/checker ownership is serialized after PR #34's clean committed checkpoint.
  • Record this plan before edits. Add a tiny existing Python report formatter with one supplied baseline test and an after-version containing four additional characterization tests; production source stays byte-identical. These are reference fixtures, not an application feature or proof of model performance.
  • Add a worked guide using shipped Epic/Task/PR headings, Task-local specification, real-identifier placeholders, desktop primary-folder setup, explicit instruction loading, claim/plan-before-commit and small-task exemption. No fabricated Issue, approval, CI, completed handoff or merge records. Preserve human decisions and read/write distinctions.
  • Add six unrun evaluation cases for scope, human clarification, normal-checkpoint handoff, code/test quality, infrastructure CI diagnosis and product-failure diagnosis with stale-head evidence. Include a result template and fair-comparison protocol covering exact revisions, requested/observed model, all attempts/non-success states, human interventions and unavailable cost. No runner, benchmark result, API call or automatic grading.
  • Link these from the existing evidence page; add the guides to public navigation validation and a focused test module to mandatory conformance discovery. Avoid README/provenance digest cascades and all installed-payload changes.

Owned paths (9):

  1. docs/evidence-status.md
  2. docs/worked-example.md
  3. docs/evaluation-cases.md
  4. docs/examples/report-summary/before/report.py
  5. docs/examples/report-summary/before/test_report.py
  6. docs/examples/report-summary/after/report.py
  7. docs/examples/report-summary/after/test_report.py
  8. .github/scripts/check-product.py (documentation/test inventories only)
  9. tests/conformance/test_public_examples.py

Acceptance and verification

  1. The worked example is self-contained, follows current adopter templates/roles and distinguishes supplied fixtures, illustrative records, future real execution and pending human acceptance. A baseline test already covers the small area; characterization adds no feature and does not invent a red/green production fix.
  2. Both documented isolated local commands work (one baseline test; five reference tests). Before/after report.py bytes match, and all four edge cases are meaningful. Neither command writes GitHub state or calls models.
  3. Six uniquely named evaluation cases remain NOT_RUN with no fabricated measurements. Missing evidence is non-success. The reusable result record distinguishes exact-head deterministic checks, advisory assessment and human-only decisions, requested/observed model, null/unmeasured cost, planned versus completed trials, all failures and exclusions. Synthetic reference tests do not fill live case results.
  4. Product/installer gates, guide navigation, focused positive/negative example tests, compilation and whitespace checks pass. All 47 payload bytes/paths/classes/modes, immutable fixtures, helpers, Skills, workflows and current integrity digests remain unchanged from the base. Checker logic changes are limited to inventory additions.
  5. Record exact candidate HEAD/tree, independent review and full PR quality/conformance results. No reuse of PR #34's success as this candidate's evidence. Publish a stacked unmerged PR; final acceptance and merge remain pending.

Planned checks: both production gates; both documented example commands; python3 -I -m unittest discover -s tests/conformance -p test_public_examples.py; existing adoption-evidence/navigation/receipt coverage where relevant; compileall; git diff --check; structural base comparison; mandatory PR CI including its real PowerShell-host requirement. No unchanged-head full-suite rerun solely for PR #34 self-check.

Risks and exclusions

Main risks are misleading examples, confusing fixture success with measured agent behavior, invented consent, unsupported client assumptions, accidental helper activation, and biased future comparisons. Mitigate with conspicuous evidence labels, unfilled real references, unrun case states, exact provenance, non-success categories, fixed comparison conditions and no network/API automation.

Excluded: payload/Skill changes, installer/helper/workflow behavior, new eval/runtime/benchmark execution framework, API spend, new chat or automation, live adopter/Project/Ruleset/CI configuration, support-program claims, private transcripts, E01 rerun, held Task 23 work, T12/runtime, release and merge/close. A future live evaluation requires its own target, budget, model/surface and operation authorization.

Dominant language
Python
Stars
5
Forks
0
Avg merge
3h 45m
Merged PRs (30d)
29

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mochan-tk/agentic-dev-kit-for-codex

All issues in mochan-tk/agentic-dev-kit-for-codex

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.