Using the test generator
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- markdown, python
- Domain
- documentation
Research direction
Start by comparing the draft with the existing track documentation, especially .docs/instructions.append.md, and inspect bin/generate_tests.py for the generator details it describes. Check that the commands, paths, and track policies are consistent and readable. Done means the guidance is humanized, accurate, and consistent with the track’s existing documentation.
Written by the indexing model from the issue text.
Description
I was asked for advice on using the test generator. The version below is a Claude-generated draft. We might want to humanize it, and ensure track documentation is consistent.
Using the test generator
bin/generate_tests.py builds an exercise's runtests.jl from the problem-specifications canonical data, using a Jinja2 template at exercises/practice/<slug>/.meta/template.j2.
One-time setup
-
Python 3.10+ with the packages in
requirements.txt:python3 -m venv .venv .venv/bin/pip install -r requirements.txt -
JuliaFormatter, installed as a Pkg app. The generator runs
jlfmton every file it writes, sojlfmtmust be on yourPATH:julia -e 'using Pkg; Pkg.Apps.add("JuliaFormatter")' jlfmt --versionIf you upgrade or remove the Julia version the app was installed with,
jlfmtstops working (exit code 127). Reinstall the app to fix it. Adding JuliaFormatter as an ordinary package doesn't update thejlfmtcommand. -
A local clone of problem-specifications, e.g. next to this repo:
git clone https://github.com/exercism/problem-specifications.git ../problem-specificationsPass its path with
-p. Without-p, the generator clones it into.problem-specifications/and deletes it again afterwards, on every run. -
The track's Julia environment, used for running the tests:
julia --project -e 'using Pkg; Pkg.instantiate()'
Everyday commands
# Generate one exercise (a glob such as 'r*' also works)
.venv/bin/python bin/generate_tests.py -p ../problem-specifications darts
# Check that committed tests match their templates, without writing anything
.venv/bin/python bin/generate_tests.py --check -p ../problem-specifications darts
# Add -v to see the template error, or the diff, when something fails
.venv/bin/python bin/generate_tests.py -v --check -p ../problem-specifications darts
# Run the generated tests against the example solution
julia --project runtests.jl darts
--check ignores the first three lines (the header with the "File last updated on" date), so a regenerated file only fails the check if the tests or their layout change. If you change a shared macro, run --check on every templated exercise, not just the one you're working on.
bin/missing-templates lists practice exercises that have canonical data but no template yet.
Writing a template
A minimal template looks like this:
{%- import "generator_macros.j2" as macros with context -%}
{{ macros.canonical_ref() }}
{{ macros.header() }}
{% for case in cases %}
@testset "{{ case["description"] }}" begin
@test score({{ case["input"]["x"] }}, {{ case["input"]["y"] }}) == {{ case["expected"] }}
end
{% endfor %}
end
casesis the canonical data, already filtered by.meta/tests.toml: any case markedinclude = falseis removed before the template sees it. Change which cases are included intests.toml, never in the template.- Groups: a case may have a
"cases"list of its own. Some exercises nest these groups more than one level deep, so use a recursive macro if you aren't sure. - Function names: the canonical
propertyis usually camelCase (isPangram), but Julia functions are named differently (ispangram,is_leap_year). Write the Julia name into the template, and check the stub and example to find it. - Error cases: use
{% if case is error_case %}. The canonical data gives only an error message, so choose a Julia exception type (DomainError,ArgumentError, ...) and check that it matches what the example throws. Use@test_throws Type f(...)and don't match on the message. - True/false results read better as
@test f(x)/@test !f(x)than as== true. - Jinja2 has a limited set of built-in tests (for example, there is no regex
searchtest). Usein, e.g.{% for case in cases if "grapheme" in case["description"] %}.
Shared macros (config/generator_macros.j2)
| Macro | Use |
|---|---|
canonical_ref() |
Header comment pointing to the canonical data |
header() |
using Test, include("<slug>.jl"), and the outer @testset verbose = true "tests" begin |
julia_string(s) |
A quoted Julia string, with \, ", $, newlines and tabs escaped. Use it for any input that might contain them. |
julia_vector(xs) |
A Julia vector literal. Strings are quoted, and nested lists become nested vectors. |
julia_dict(d, char_keys=false) |
A Dict(...) literal. String keys are quoted, or written as Chars with char_keys=true. |
The generator also provides filters such as to_snake, camel_case, regex_replace and wrap_overlong (see bin/generate_tests.py).
Julia pitfalls
- Empty vectors as arguments.
[]is aVector{Any}. A student method typed asf(xs::Vector{Int})won't accept it, so write a typed literal such asInt[]for empty inputs. Empty expected values can stay[], because==ignores element types. - Tuples vs vectors.
(1, 2) == [1, 2]isfalse. If solutions may reasonably return either, comparecollect(f()) == [...]. - Chars vs strings. Some Julia APIs use
Char('A') where the canonical data has one-letter strings. - Large integers. Literals above
typemax(Int64)are parsed asInt128, or asBigIntabove that range. Whether to include BigInt cases is still undecided. For now, leave anyinclude = falseentries intests.tomlalone. - Opt-in bonus tests (e.g. reverse-string's
TEST_GRAPHEMES) are described in.docs/instructions.append.md. Keep that mechanism in the template.
Track policy
- Canonical cases only. When converting an exercise, drop the old hand-written tests that aren't in the canonical data, including input-validation tests. Mention in the PR what was removed.
- Exceptions need maintainer approval. Track-specific cases then go in
.meta/additional_tests.jsonas{"cases": [...]}, and the template loops overcases + additional_cases. At present the only one is pangram's Arabic case.
Before opening a PR
- Regenerate, then run
julia --project runtests.jl <slug>. The example solution must pass every test. - Run
--checkon the exercise. If you changedgenerator_macros.j2, run it on every templated exercise. - Compare the new
runtests.jlwith the old one (git diff), and list in the PR description any tests that were removed or changed meaning. - Commit the template together with the regenerated
runtests.jl.
- Dominant language
- Julia
- Stars
- 74
- Forks
- 84
- Avg merge
- 7h 51m
- Merged PRs (30d)
- 36
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from exercism/julia
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
exercism/julia#1190 · 1 reaction ·
Maintainers usually reply within 1 day
-
Syllabus errorsOpen
Difficulty 4/5 3-5 days Newbie friendliness 35/100
exercism/julia#1174 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
Any more syllabus?Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
exercism/julia#1121 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
exercism/julia#1098 · 3 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
exercism/julia#1049 · 10 comments · 1 reaction ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Sienna-Platform/PowerSimulations.jl#1737 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
FluxML/Optimisers.jl#254 ·
-
Minor: floodplain warning percentage, scalar output error message and a never-true identity checkOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
JuliaAI/MLJTransforms.jl#101 ·
Maintainers usually reply within 1 day
-
@contents blocks fail to render on Julia 1.14Possibly taken @sbfnk claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LuxDL/DocumenterVitepress.jl#410 ·