mllam/neural-lam

Author notebooks as jupytext-paired `py:percent` sources instead of committing raw `.ipynb`

Open

#708 opened on Jul 16, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (276 forks)auto 404
cicddocumentationgood first issuehelp wanted

Repository metrics

Stars
 (282 stars)
PR merge metrics
 (PR metrics pending)

Description

docs/notebooks/create_reduced_meps_dataset.ipynb (and the DANRA tutorial coming in #577) are committed as raw .ipynb. That gives us:

  • unreviewable diffs: a one-line code change shows up as a JSON blob, so notebook changes effectively go unreviewed;
  • executed outputs committed into git, including absolute local paths from whoever ran it last;
  • notebook code that no linter ever sees, and nothing that detects when it drifts from the library.

There is no jupytext/myst tooling in the repo today (nothing in pyproject.toml or .pre-commit-config.yaml).

I suggest to make a py:percent file the source of truth and keep it paired with the .ipynb via a pre-commit hook, i.e. the same self-consistency contract as the graph-storage-spec hook from #323 @leifdenby @observingClouds:

- id: jupytext-sync
  name: Keep paired notebooks in sync with their py:percent source
  entry: jupytext --sync
  language: python
  additional_dependencies: [jupytext]
  files: ^docs/notebooks/.*\.(ipynb|py)$

The .py is diffable, reviewable and carries no outputs. Two practical notes from trying it. But if you can fix them instead of ignoring them, even better.

  • the generated .py is prose-heavy (markdown cells become long comments), so it needs excluding from black/isort/flake8/mypy;
  • it also needs excluding from trailing-whitespace/end-of-file-fixer, which otherwise strip whitespace jupytext regenerates and the two hooks fight forever.

This pairs naturally with giving notebooks a Python API to call (see https://github.com/mllam/neural-lam/issues/707): the source then reads as plain, reviewable Python instead of subprocess calls plus path globbing.

Contributor guide