Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Tracking: sparse/CSR path follow-ups after #961

Open
#972 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

This is a tracking issue for a series of complex refactors to the sparse/CSR data path in linopy. Start by reading the linked pull request #961 and the umbrella issue #756 to understand the context. The work is broken into three PRs, each with multiple parts (A-H) that touch core files like linopy/expressions.py, constraints.py, and test/test_csr.py. The first step is implementing observability (part A) to add shape/sizes/coords access and a warn_on_densify helper. 'Done' means all child issues are addressed and the sparse path preserves backing through operations without silent densification.

Written by the indexing model from the issue text.

Description

performance sparse

[!NOTE]
The following content was generated by AI.

Describe the feature you'd like to see

Tracking issue for the CSR follow-ups after #961 (umbrella #756 is closed). Each child names a place where a CSR-backed expression silently densifies, or a gap in the frozen-constraint API. This issue fixes the order in which to address them.

Key observations
  • LinearExpression.data (linopy/expressions.py:2358) is where a CSR-backed expression densifies on read. Other paths drop the backing without passing through it: @ returns dense when the input was dense and sparse_groupby is off (expressions.py:2609), CSRConstraint.to_dense()/mutable(), the rebuild at model.py:1345, and the return None fallbacks in _sparse_matmul, _try_csr_merge, _aligned and csr_rhs, which lose the reason.
  • Grid holds only per-dim indexes. Aux coords live next to it in CSRLinearExpression.coords and are dropped by CSRConstraint.from_csr (constraints.py:1401), which is #941.
  • Zero policy on the sparse path: every operation keeps explicit zeros (merge, scaling, sum, groupby, selection), so sparse and dense give the same terms up to order; only @/dot prunes them. Cell activeness is carried by const alone. Export culls zeros regardless; #925 is an export/persistent-diff question and does not block this series.
Plan

Three pull requests, grouped by shared design rather than one per issue:

  1. Expression kernel (parts A, C, B, D): observability, aux coords in Grid, and all operations that keep the backing. One commit per issue, so it can be reviewed commit by commit.
  2. Boundaries (parts E, F): constraint side and export, after #806 has merged.
  3. Control + docs (parts G, H): public-API decisions and documentation, once the preserving set is final.
Part Issues Content Depends on
A: observability #962, #969 (parts 1+2) Serve shape/sizes/coords/dims/isnull from the CSR store. .is_sparse, repr marker. Opt-in warn_on_densify through one _densify_notice(reason) helper, called at data, at each return None fallback and in CSRConstraint.to_dense. data stays a one-way conversion (no cached dense copy next to _csr, the setters would make it stale). Move aux coords into Grid.
C #965 scaled_by / shifted for non-scalar constants, aligned through _matmul_operand_to_matrix A
B #964 sum(dim) by merging rows directly (CSRLinearExpression.aggregated), keeping explicit zeros. Chained groupby from a CSR source; fix the gate at expressions.py:613 to use coord_dims instead of self.data.dims. A
D #966, mask= part of #970 Row masking, sel through reindexed, isel as row gather. Keep _try_csr_merge sparse when aux coords differ across grids. A
E: constraint side #941, #963, rest of #970 Aux coords on CSRConstraint (trivial once in Grid), incl. netcdf round trip (io.py:1260). Explanatory errors for loc/update/from_rule naming .mutable(). Expression rhs moved to lhs. D, #806
F: export #968, then #967 flat/to_polars from the CSR store, modelled on CSRConstraint.to_polars. Sparse objective across objective.py, matrices.py, io.py, persistent/diff.py; decide where the objective name lives (CSRLinearExpression has no attrs). B
G: control #969 (part 3) Per-call sparse= on @/dot/merge; remove the coupling of @ to options["sparse_groupby"]. Collapsing the three opt-ins (semantics="v1", sparse=/sparse_groupby, freeze_constraints) goes to a separate discussion issue. B–F
H: docs #971 API entries, user-guide section, list of operations that keep the backing G

Ordering notes:

  • Within PR 1, commit in the order A, C, B, D. PR 1 stays out of constraints.py so #806 can land before PR 2.
  • Tests extend the existing helpers in test/test_csr.py (assert_frozen_equal, assert_terms_equal, assert_cells_equal, assert_contracted_equal) with a "backing kept" assertion instead of adding a new fixture.
  • Out of scope for now: DataArray sign in to_constraint and non-scalar fillna still densify. #745 is likely covered by sparse groupby; re-check and close or narrow.
Children
  • #962
  • #969
  • #965
  • #964
  • #966
  • #970
  • #941
  • #963
  • #968
  • #967
  • #971
Dominant language
Python
Stars
257
Forks
87
Avg merge
1d 8h
Merged PRs (30d)
32

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PyPSA/linopy

All issues in PyPSA/linopy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.