Tracking: sparse/CSR path follow-ups after #961
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- backend, data-engineering, tooling
Research direction
This is a tracking issue for a series of complex refactors to the sparse/CSR data path in linopy. Start by reading the linked pull request #961 and the umbrella issue #756 to understand the context. The work is broken into three PRs, each with multiple parts (A-H) that touch core files like linopy/expressions.py, constraints.py, and test/test_csr.py. The first step is implementing observability (part A) to add shape/sizes/coords access and a warn_on_densify helper. 'Done' means all child issues are addressed and the sparse path preserves backing through operations without silent densification.
Written by the indexing model from the issue text.
Description
[!NOTE]
The following content was generated by AI.
Describe the feature you'd like to see
Tracking issue for the CSR follow-ups after #961 (umbrella #756 is closed). Each child names a place where a CSR-backed expression silently densifies, or a gap in the frozen-constraint API. This issue fixes the order in which to address them.
Key observations
LinearExpression.data(linopy/expressions.py:2358) is where a CSR-backed expression densifies on read. Other paths drop the backing without passing through it:@returns dense when the input was dense andsparse_groupbyis off (expressions.py:2609),CSRConstraint.to_dense()/mutable(), the rebuild atmodel.py:1345, and thereturn Nonefallbacks in_sparse_matmul,_try_csr_merge,_alignedandcsr_rhs, which lose the reason.Gridholds only per-dim indexes. Aux coords live next to it inCSRLinearExpression.coordsand are dropped byCSRConstraint.from_csr(constraints.py:1401), which is #941.- Zero policy on the sparse path: every operation keeps explicit zeros (merge, scaling,
sum,groupby, selection), so sparse and dense give the same terms up to order; only@/dotprunes them. Cell activeness is carried byconstalone. Export culls zeros regardless; #925 is an export/persistent-diff question and does not block this series.
Plan
Three pull requests, grouped by shared design rather than one per issue:
- Expression kernel (parts A, C, B, D): observability, aux coords in
Grid, and all operations that keep the backing. One commit per issue, so it can be reviewed commit by commit. - Boundaries (parts E, F): constraint side and export, after #806 has merged.
- Control + docs (parts G, H): public-API decisions and documentation, once the preserving set is final.
| Part | Issues | Content | Depends on |
|---|---|---|---|
| A: observability | #962, #969 (parts 1+2) | Serve shape/sizes/coords/dims/isnull from the CSR store. .is_sparse, repr marker. Opt-in warn_on_densify through one _densify_notice(reason) helper, called at data, at each return None fallback and in CSRConstraint.to_dense. data stays a one-way conversion (no cached dense copy next to _csr, the setters would make it stale). Move aux coords into Grid. |
– |
| C | #965 | scaled_by / shifted for non-scalar constants, aligned through _matmul_operand_to_matrix |
A |
| B | #964 | sum(dim) by merging rows directly (CSRLinearExpression.aggregated), keeping explicit zeros. Chained groupby from a CSR source; fix the gate at expressions.py:613 to use coord_dims instead of self.data.dims. |
A |
| D | #966, mask= part of #970 |
Row masking, sel through reindexed, isel as row gather. Keep _try_csr_merge sparse when aux coords differ across grids. |
A |
| E: constraint side | #941, #963, rest of #970 | Aux coords on CSRConstraint (trivial once in Grid), incl. netcdf round trip (io.py:1260). Explanatory errors for loc/update/from_rule naming .mutable(). Expression rhs moved to lhs. |
D, #806 |
| F: export | #968, then #967 | flat/to_polars from the CSR store, modelled on CSRConstraint.to_polars. Sparse objective across objective.py, matrices.py, io.py, persistent/diff.py; decide where the objective name lives (CSRLinearExpression has no attrs). |
B |
| G: control | #969 (part 3) | Per-call sparse= on @/dot/merge; remove the coupling of @ to options["sparse_groupby"]. Collapsing the three opt-ins (semantics="v1", sparse=/sparse_groupby, freeze_constraints) goes to a separate discussion issue. |
B–F |
| H: docs | #971 | API entries, user-guide section, list of operations that keep the backing | G |
Ordering notes:
- Within PR 1, commit in the order A, C, B, D. PR 1 stays out of
constraints.pyso #806 can land before PR 2. - Tests extend the existing helpers in
test/test_csr.py(assert_frozen_equal,assert_terms_equal,assert_cells_equal,assert_contracted_equal) with a "backing kept" assertion instead of adding a new fixture. - Out of scope for now: DataArray
signinto_constraintand non-scalarfillnastill densify. #745 is likely covered by sparsegroupby; re-check and close or narrow.
Children
- #962
- #969
- #965
- #964
- #966
- #970
- #941
- #963
- #968
- #967
- #971
- Dominant language
- Python
- Stars
- 257
- Forks
- 87
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 32
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PyPSA/linopy
-
bug solver interface
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
documentation sparse
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
enhancement sparse
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) Openenhancement sparse
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
enhancement performance sparse
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
area: harness bug status: needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Human-Agent-Society/reef#625 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
learningequality/kolibri#15351 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Name consistency Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
eellak/triplestore#65 · 1 comment ·