Author notebooks as jupytext-paired `py:percent` sources instead of committing raw `.ipynb`
#708 opened on Jul 16, 2026
Repository metrics
- Stars
- (282 stars)
- PR merge metrics
- (PR metrics pending)
Description
docs/notebooks/create_reduced_meps_dataset.ipynb (and the DANRA tutorial coming in #577) are committed as raw .ipynb. That gives us:
- unreviewable diffs: a one-line code change shows up as a JSON blob, so notebook changes effectively go unreviewed;
- executed outputs committed into git, including absolute local paths from whoever ran it last;
- notebook code that no linter ever sees, and nothing that detects when it drifts from the library.
There is no jupytext/myst tooling in the repo today (nothing in pyproject.toml or .pre-commit-config.yaml).
I suggest to make a py:percent file the source of truth and keep it paired with the .ipynb via a pre-commit hook, i.e. the same self-consistency contract as the graph-storage-spec hook from #323 @leifdenby @observingClouds:
- id: jupytext-sync
name: Keep paired notebooks in sync with their py:percent source
entry: jupytext --sync
language: python
additional_dependencies: [jupytext]
files: ^docs/notebooks/.*\.(ipynb|py)$
The .py is diffable, reviewable and carries no outputs. Two practical notes from trying it. But if you can fix them instead of ignoring them, even better.
- the generated
.pyis prose-heavy (markdown cells become long comments), so it needs excluding from black/isort/flake8/mypy; - it also needs excluding from
trailing-whitespace/end-of-file-fixer, which otherwise strip whitespace jupytext regenerates and the two hooks fight forever.
This pairs naturally with giving notebooks a Python API to call (see https://github.com/mllam/neural-lam/issues/707): the source then reads as plain, reviewable Python instead of subprocess calls plus path globbing.