Opt-in allocator arena release after Model.solve() to bound RSS across sequential solves
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- backend, performance
Research direction
Start with linopy/_memory.py and the existing Model.solve path. Review the platform-mocked dispatch, backward-compatibility, repeated-call, and allocator-lookup failure tests described in the issue. Done means an opt-in release path is exposed as agreed, defaults remain unchanged, supported platforms are handled safely, and the proposed tests pass.
Written by the indexing model from the issue text.
Description
Summary
Sequential PyPSA-Eur-style chunked solves (weekly LPs, identical structure, N iterations in one process) exhibit monotonic RSS growth until OOM, even after del model + gc.collect() between iterations. Root cause is platform allocator arena retention (glibc on Linux, libSystem on macOS), not Python-level leaks. Proposing a small opt-in helper that releases arena pages back to the kernel after the existing solve path completes.
This is orthogonal to:
- #630 (CSR
freeze=True) — reduces peak within a single solve. - #699 (persistent solver /
ModelDiff) — avoids rebuilds between solves. - #219 — NetCDF round-trip to lower peak during a single solve (has an unresolved SOS-undo round-trip gap, see discussion there).
Neither addresses the allocator-arena pattern: glibc / libSystem keep freed pages cached, so RSS grows monotonically across the chunk loop even when the Python-side working set is bounded.
Proposal
linopy/_memory.py (~30 LOC):
def release_allocator_pages() -> None:
"""gc.collect() then ask platform allocator to return arenas to kernel.
Linux: libc.malloc_trim(0)
macOS: libSystem.malloc_zone_pressure_relief(NULL, 0)
Other: gc only, silent no-op.
"""
Wired into Model.solve(release_memory: bool = False). Default unchanged. Also exposed at top level (linopy.release_allocator_pages()) so callers in chunked-solve loops can trim mid-pipeline without going through Model.solve.
Benchmark (macOS arm64, linopy master 37af4ba)
5 sequential solves of an identical 5k var × 2k con dense-block LP via HiGHS, single Python process:
| metric | baseline | release_memory=True |
Δ |
|---|---|---|---|
| RSS post-cleanup, iter 4 (GB) | 4.41 | 2.02 | −54 % |
ru_maxrss (GB) |
5.30 | 4.32 | −19 % |
| wall-clock total (s) | 30.97 | 32.09 | +3.6 % (noise) |
Baseline RSS climbs 3.71 → 4.41 GB monotonically across the 5 iterations. With trim, post-cleanup RSS drops back to 1.5–2.0 GB after each solve. Linux numbers via malloc_trim typically more dramatic — happy to add a Linux benchmark before opening the PR.
Tests
- Platform-mocked dispatch (Linux / Darwin / Windows / FreeBSD).
- Kwarg default backward-compat smoke test.
- Repeated-call stability (no crashes if
libc/libSystemlookup misses). - Swallows
OSError/AttributeError, debug-logs on miss.
Open questions
- Naming:
release_memory=TrueonModel.solvevs. top-levellinopy.release_allocator_pages()vs. both. Leaning toward both — kwarg for the common single-solve case, public helper for chunked loops where users want to trim between solves without re-enteringModel.solve. - Interaction with #699: once persistent solver lands, native solver model + factor stay resident. Python-side arena cache still grows; trim hook stays useful. Agree?
- Should the helper also clear
m._xCounter/m._cCounterstyle state onModel.solve(release_memory=True)? Leaning no — out of scope for an allocator helper, separate concern.
Will open PR once direction is confirmed.
Refs: #219, #630, #699
- Dominant language
- Python
- Stars
- 257
- Forks
- 87
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 32
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PyPSA/linopy
-
bug solver interface
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
documentation sparse
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
enhancement sparse
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) Openenhancement sparse
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
enhancement performance sparse
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·