Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Benchmark pack generation across model tiers

Open
  • Is something wrong?
#58 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
node.js, typescript
Domain
testing, tooling

Research direction

Start with the generation code in packages/cli/src/generate/, then read docs/authoring/pipeline.md and the related issues #44, #43, and #51 to understand the model options and playtest inputs. Build a benchmark around fixed ideas, durations, model configurations, and seeds, with a fake LLM path for CI and a cost confirmation guard. Done means the CI harness runs without API calls and docs contain a real run’s dated results, cost, harness version, raw logs, and recommendations per pack length.

Written by the indexing model from the issue text.

Description

area: pipeline priority: medium type: research

Context

Which models can write a good content pack is currently a guess. Small model tiers have improved sharply and cost one to two orders of magnitude less than large ones; whether they reach publishable 30-45 minute packs, and where they fall short (schema, consistency, walkthrough, puzzle design), should be measured. The result becomes the documented recommendation per pack length and feeds the v0.3 issue-to-PR generation workflow.

Scope

  • A benchmark harness (script under packages/cli or a separate tools/ directory) that runs lexvm generate for a fixed set of ideas at 15, 30 and 45 minutes, per model configuration (small, mid and large tier, plus the hybrid from the per-stage option), with a fixed seed set of prompts.
  • Metrics per run: check pass (zero errors), repair rounds, wall time, input/output tokens and cost, play-time deviation from target, explored states, and an automated LLM playtest score (#51).
  • A results table committed under docs/ with date and harness version; raw run logs as artifacts, no API keys or secrets.
  • Budget guard: the harness prints the estimated cost and requires confirmation before spending.

Acceptance criteria

  • Harness runs end to end with a fake LLM in CI (no API calls).
  • One real benchmark run is documented with a results table and cost.
  • docs/authoring/pipeline.md recommends model configurations per pack length based on the results.

References

  • packages/cli/src/generate/
  • docs/authoring/pipeline.md
  • Related: per-stage model selection (#44), walkthrough derivation with lexvm solve (#43), automated LLM playtests (#51)
Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
28m
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nullthrone/lexvm

All issues in nullthrone/lexvm

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.