Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Run automated LLM playtests for new packs in CI

Open
  • Is something wrong?
#51 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
ci-cd, cli, testing

Research direction

Start with packages/engine/src/runtime.ts for the view and hint affordances, then read packages/cli/src/commands/play.ts and packages/cli/src/generate/llm.ts for existing command and LLM entry points. Run lexvm play on the demo pack to understand its walkthrough and canonical log. Done when the demo playtest wins within budget, a removed-clue test pack reports its stuck puzzle, the labeled workflow protects secrets, and docs/authoring/pipeline.md explains the command and report.

Written by the indexing model from the issue text.

Description

area: infra area: pipeline priority: medium type: feature

Context

Verification proves that a pack can be won; it says nothing about whether the puzzles are fairly clued. An agent that plays without the walkthrough shows where players are likely to get stuck, before a human reviewer spends an hour on it. The first playtest round of the noir pack, three scripted playthroughs with lexvm play, found a blocker that verification cannot see: combining notebook entries, the core deduction mechanic, was never taught (packs/letzte-nacht-der-reichsmark/design/CHANGES.md).

Scope

  • lexvm playtest <pack> (CLI, Anthropic API): the agent sees what a tap player sees (the rendered story text and the view affordances: objects with verbs, exits, dialogue options, inventory, notebook) and picks one offered command per turn; typed-only commands are not allowed. It may ask for hints, which are counted.
  • Budget: a turn limit (for example three times the walkthrough length) and a token limit; the run stops at a win.
  • Report as Markdown and JSON: won or not, turns compared with the walkthrough, hints used per hint id, stuck points (stretches without a new state or clue, rooms with the most turns), the share of rejected and no-op commands, the order in which puzzles were solved. The transcript and command log are uploaded as artifacts; the log replays with lexvm play --canonical.
  • Workflow: runs on pull requests that change packs/ when a maintainer adds a label (cost control, like the generation workflow), and posts the report as a comment. Only pack data from the pull request is used; no code from the pull request runs with access to secrets.

Acceptance criteria

  • A playtest of the demo pack wins within its budget and produces a report.
  • With a test pack whose key clue was removed, the report names the stuck puzzle.
  • The workflow runs only on a maintainer's label and never exposes secrets to code from forks.
  • docs/authoring/pipeline.md documents the command and how to read the report.

References

  • packages/engine/src/runtime.ts (view, hint)
  • packages/cli/src/generate/llm.ts
  • packages/cli/src/commands/play.ts
  • docs/authoring/guide.md (Fair clueing)
  • docs/adr/0005-forgiving-policy-verified-by-exploration.md
Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
28m
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nullthrone/lexvm

All issues in nullthrone/lexvm

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.