Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Meta: separate dense and sparse stores from the expression and constraint frontend

Aperta
  • Qualcosa non va?
#1,019 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Rishabhkanhaiya ci sta già lavorando.

Dal 7/10/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Refactoring
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
numpy, pandas, python
Ambito
backend

Direzione di ricerca

This is a meta issue; start with child #1020 for the first step, and read the referenced sections in linopy/expressions.py and linopy/csr.py to understand the current dispatch and sparse operations. The plan also identifies test/test_csr.py, constraints.py, matrices.py, and model.py for later steps. Done means completing the staged store refactor and conformance tests; several steps depend on other work, including legacy-semantics removal.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

discussion refactor sparse

[!NOTE]
The following content was generated by AI.

Describe the feature you'd like to see

Meta issue. With #972 the sparse (CSR) backing is complete and observable, and Model(sparse=True) makes dense and sparse two equivalent ways to hold the same model. The code does not reflect that yet: sparse support is layered onto the dense classes by overrides and isinstance switches. This issue fixes the target architecture and the order of the refactoring. Child issues are spawned per step. Code links point to master at 69dd5c97.

Target

One LinearExpression class and one Constraint class, each holding either a dense or a sparse store behind the same small interface. Every conversion between stores passes through one chokepoint that names the operation. Model, export, persistent diff and solvers never branch on the store.

Where the coupling is today
Layer State Reference
LinearExpression one class, 34 explicit branch points on _csr / is_sparse / _try_csr_* / _sparse_*; every non-overridden method falls back implicitly by reading .data, which converts the receiver in place and for good with a generic reason linopy/expressions.py:2443-3100, groupby :485-669, _try_csr_merge :3740, _densify_all :3698
CSRLinearExpression already has the shape of a store interface, mirroring the frontend one to one (summed, scaled, added, contracted, taken, reindexed, filled, renamed, with_const); the dense equivalent is inline xarray code in BaseExpression, mixed with alignment and semantics checks linopy/csr.py:262-694
Constraints Constraint and CSRConstraint are siblings under ConstraintBase(ABC) with about twenty abstract methods each implements; CSRConstraint stores a raw csr_array plus seven side arrays, not a CSRLinearExpression, so lhs and data rebuild a Dataset on every access and from_csr / from_dense are separate conversions; _csr means different things on the two classes linopy/constraints.py:145, :762, :797-812, :1229, :1595, :1664, :1739
Model the read-only sparse key is clean; three reads remain (groupby sum, @, add_constraints fast path) linopy/model.py:572, :1375; expressions.py:615, :2888
Export and solvers mostly polymorphic through ConstraintBase; MatrixAccessor and the dual write-back still switch on type; Model.copy densifies model-level expressions linopy/matrices.py:48-55, :289; model.py:2533; io.py:1424
Tests equivalence by twin builds (twin_models, assert_sparse_matches, assert_frozen_equal) and ten hand-picked sparse parametrizations; no backend fixture, no conformance suite over the store operations test/test_csr.py

Reproduction of the implicit fallback:

import warnings
import numpy as np, pandas as pd, linopy

linopy.options["semantics"] = "v1"
linopy.options["warn_on_densify"] = True
i, t = pd.RangeIndex(20, name="i"), pd.RangeIndex(5, name="t")
m = linopy.Model(sparse=True)
x = m.add_variables(0, 1, coords=[i, t], name="x")
g = pd.Series(np.arange(20) % 3, index=i, name="g")

for op in ["shift(t=1)", "diff('t')", "coeffs", "to_quadexpr()", "isnull()"]:
    s = x.groupby(g).sum()
    assert s.is_sparse
    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        eval(f"s.{op}")
    msgs = [str(x.message) for x in w if "dens" in str(x.message).lower()]
    print(f"{op:16} sparse after: {s.is_sparse!s:5}  warning: {msgs}")

Output on master (69dd5c97):

shift(t=1)       sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
diff('t')        sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
coeffs           sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
to_quadexpr()    sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
isnull()         sparse after: True   warning: []
Plan
Step Content Depends on
0: legacy removal Make v1 the default in the next release, then delete legacy semantics (linopy/semantics.py and the is_v1() gates). Not part of this issue, but the prerequisite for steps 1, 3 and 4: those steps rewrite exactly the methods that carry the gates, so doing them first means doing them twice. #845, #846
1: chokepoint One _delegate(name, ...) on LinearExpression: an operation either has a sparse implementation or converts with reason=name. The 34 branches become a dispatch table mapping frontend methods to CSRLinearExpression counterparts. .data becomes a plain read through the same chokepoint. The receiver keeps its backing; the converted copy is the operand of the fallback. Partial fallbacks (_selected, _sparse_matmul, _try_csr_merge, csr_rhs) return a reason instead of calling _densify_all. The table is the authoritative list of backing-preserving operations: #971 documents it, #1011 adds rows. 0
2: conformance tests A backend fixture (dense, sparse) and one parametrized suite that runs every table entry on both stores and compares cells, terms and isnull, plus a test that every fallback warns with its own operation name. Replace twin builds where the fixture covers them. Only needs require_v1 and a list of operations, so it can land before step 0 and then guards the legacy removal as well. –
3: constraint store CSRConstraint holds a CSRLinearExpression plus sign, rhs, scaling and labels. lhs returns a sparse LinearExpression without a rebuild, from_csr becomes a constructor, from_dense goes through CSRLinearExpression.from_dense. Then Constraint and CSRConstraint collapse into one class with a frozen flag whose storage is its lhs; ConstraintBase goes away. MatrixAccessor and the dual write-back lose their isinstance branches. 0, 1, 2
4: dense store Move the xarray Dataset manipulation out of BaseExpression into a DenseLinearExpression implementing the same protocol as CSRLinearExpression. LinearExpression keeps only the user API, alignment and semantics checks, and delegates to its store. Align the store method names with the frontend names in this step. 0, 3

Ordering notes:

  • Step 2 first, at any time. Step 0 after the release. Steps 1, 3 and 4 back to back after step 0.
  • Step 1 before #1011 would absorb its new branches, but #1011 is user-facing and may land first; the four extra hand-written branches are then absorbed by step 1.
  • #1010 is internal to csr.py and independent of this plan.
  • Step 4 depends on step 0 because the sparse form is valid only under v1 (linopy/csr.py:1-27): a store protocol that one backend implements only under one semantics is worse than the current branches.

Rejected:

  • A lazy expression graph (already rejected in #972).
  • A sparse store for variables and one-term expressions (no gain in the #972 benchmark).
  • Two public classes (DenseLinearExpression / SparseLinearExpression) at the frontend. Users should not pick a class; the model key picks the store.
Children
  • #845, #846 (step 0: v1 default, then legacy semantics removal)
  • #1020 (step 1: densification chokepoint and dispatch table)
  • step 2: backend-parametrized conformance suite
  • step 3: CSRConstraint on a CSRLinearExpression, single Constraint class
  • step 4: DenseLinearExpression store
Lingua principale
Python
Stelle
259
Fork
89
Merge medio
1g 18h
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PyPSA/linopy

Tutte le issue di PyPSA/linopy

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.