Using the test generator

Open Beginner friendly
#1,159 0 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
markdown, python
Domain
documentation

Research direction

Start by comparing the draft with the existing track documentation, especially .docs/instructions.append.md, and inspect bin/generate_tests.py for the generator details it describes. Check that the commands, paths, and track policies are consistent and readable. Done means the guidance is humanized, accurate, and consistent with the track’s existing documentation.

Written by the indexing model from the issue text.

Description

I was asked for advice on using the test generator. The version below is a Claude-generated draft. We might want to humanize it, and ensure track documentation is consistent.


Using the test generator

bin/generate_tests.py builds an exercise's runtests.jl from the problem-specifications canonical data, using a Jinja2 template at exercises/practice/<slug>/.meta/template.j2.

One-time setup

  • Python 3.10+ with the packages in requirements.txt:

    python3 -m venv .venv
    .venv/bin/pip install -r requirements.txt
    
  • JuliaFormatter, installed as a Pkg app. The generator runs jlfmt on every file it writes, so jlfmt must be on your PATH:

    julia -e 'using Pkg; Pkg.Apps.add("JuliaFormatter")'
    jlfmt --version
    

    If you upgrade or remove the Julia version the app was installed with, jlfmt stops working (exit code 127). Reinstall the app to fix it. Adding JuliaFormatter as an ordinary package doesn't update the jlfmt command.

  • A local clone of problem-specifications, e.g. next to this repo:

    git clone https://github.com/exercism/problem-specifications.git ../problem-specifications
    

    Pass its path with -p. Without -p, the generator clones it into .problem-specifications/ and deletes it again afterwards, on every run.

  • The track's Julia environment, used for running the tests:

    julia --project -e 'using Pkg; Pkg.instantiate()'
    

Everyday commands

# Generate one exercise (a glob such as 'r*' also works)
.venv/bin/python bin/generate_tests.py -p ../problem-specifications darts

# Check that committed tests match their templates, without writing anything
.venv/bin/python bin/generate_tests.py --check -p ../problem-specifications darts

# Add -v to see the template error, or the diff, when something fails
.venv/bin/python bin/generate_tests.py -v --check -p ../problem-specifications darts

# Run the generated tests against the example solution
julia --project runtests.jl darts

--check ignores the first three lines (the header with the "File last updated on" date), so a regenerated file only fails the check if the tests or their layout change. If you change a shared macro, run --check on every templated exercise, not just the one you're working on.

bin/missing-templates lists practice exercises that have canonical data but no template yet.

Writing a template

A minimal template looks like this:

{%- import "generator_macros.j2" as macros with context -%}
{{ macros.canonical_ref() }}

{{ macros.header() }}
{% for case in cases %}
    @testset "{{ case["description"] }}" begin
        @test score({{ case["input"]["x"] }}, {{ case["input"]["y"] }}) == {{ case["expected"] }}
    end
{% endfor %}
end
  • cases is the canonical data, already filtered by .meta/tests.toml: any case marked include = false is removed before the template sees it. Change which cases are included in tests.toml, never in the template.
  • Groups: a case may have a "cases" list of its own. Some exercises nest these groups more than one level deep, so use a recursive macro if you aren't sure.
  • Function names: the canonical property is usually camelCase (isPangram), but Julia functions are named differently (ispangram, is_leap_year). Write the Julia name into the template, and check the stub and example to find it.
  • Error cases: use {% if case is error_case %}. The canonical data gives only an error message, so choose a Julia exception type (DomainError, ArgumentError, ...) and check that it matches what the example throws. Use @test_throws Type f(...) and don't match on the message.
  • True/false results read better as @test f(x) / @test !f(x) than as == true.
  • Jinja2 has a limited set of built-in tests (for example, there is no regex search test). Use in, e.g. {% for case in cases if "grapheme" in case["description"] %}.
Shared macros (config/generator_macros.j2)
Macro Use
canonical_ref() Header comment pointing to the canonical data
header() using Test, include("<slug>.jl"), and the outer @testset verbose = true "tests" begin
julia_string(s) A quoted Julia string, with \, ", $, newlines and tabs escaped. Use it for any input that might contain them.
julia_vector(xs) A Julia vector literal. Strings are quoted, and nested lists become nested vectors.
julia_dict(d, char_keys=false) A Dict(...) literal. String keys are quoted, or written as Chars with char_keys=true.

The generator also provides filters such as to_snake, camel_case, regex_replace and wrap_overlong (see bin/generate_tests.py).

Julia pitfalls
  • Empty vectors as arguments. [] is a Vector{Any}. A student method typed as f(xs::Vector{Int}) won't accept it, so write a typed literal such as Int[] for empty inputs. Empty expected values can stay [], because == ignores element types.
  • Tuples vs vectors. (1, 2) == [1, 2] is false. If solutions may reasonably return either, compare collect(f()) == [...].
  • Chars vs strings. Some Julia APIs use Char ('A') where the canonical data has one-letter strings.
  • Large integers. Literals above typemax(Int64) are parsed as Int128, or as BigInt above that range. Whether to include BigInt cases is still undecided. For now, leave any include = false entries in tests.toml alone.
  • Opt-in bonus tests (e.g. reverse-string's TEST_GRAPHEMES) are described in .docs/instructions.append.md. Keep that mechanism in the template.

Track policy

  • Canonical cases only. When converting an exercise, drop the old hand-written tests that aren't in the canonical data, including input-validation tests. Mention in the PR what was removed.
  • Exceptions need maintainer approval. Track-specific cases then go in .meta/additional_tests.json as {"cases": [...]}, and the template loops over cases + additional_cases. At present the only one is pangram's Arabic case.

Before opening a PR

  1. Regenerate, then run julia --project runtests.jl <slug>. The example solution must pass every test.
  2. Run --check on the exercise. If you changed generator_macros.j2, run it on every templated exercise.
  3. Compare the new runtests.jl with the old one (git diff), and list in the PR description any tests that were removed or changed meaning.
  4. Commit the template together with the regenerated runtests.jl.
Dominant language
Julia
Stars
74
Forks
84
Avg merge
7h 51m
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from exercism/julia

All issues in exercism/julia

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.