Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Meta: separate dense and sparse stores from the expression and constraint frontend

Đang mở
  • Có gì không ổn?
#1,019 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@Rishabhkanhaiya đang làm issue này rồi.

Từ ngày 7/10/2026.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Tái cấu trúc
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
numpy, pandas, python
Lĩnh vực
backend

Hướng nghiên cứu

This is a meta issue; start with child #1020 for the first step, and read the referenced sections in linopy/expressions.py and linopy/csr.py to understand the current dispatch and sparse operations. The plan also identifies test/test_csr.py, constraints.py, matrices.py, and model.py for later steps. Done means completing the staged store refactor and conformance tests; several steps depend on other work, including legacy-semantics removal.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

discussion refactor sparse

[!NOTE]
The following content was generated by AI.

Describe the feature you'd like to see

Meta issue. With #972 the sparse (CSR) backing is complete and observable, and Model(sparse=True) makes dense and sparse two equivalent ways to hold the same model. The code does not reflect that yet: sparse support is layered onto the dense classes by overrides and isinstance switches. This issue fixes the target architecture and the order of the refactoring. Child issues are spawned per step. Code links point to master at 69dd5c97.

Target

One LinearExpression class and one Constraint class, each holding either a dense or a sparse store behind the same small interface. Every conversion between stores passes through one chokepoint that names the operation. Model, export, persistent diff and solvers never branch on the store.

Where the coupling is today
Layer State Reference
LinearExpression one class, 34 explicit branch points on _csr / is_sparse / _try_csr_* / _sparse_*; every non-overridden method falls back implicitly by reading .data, which converts the receiver in place and for good with a generic reason linopy/expressions.py:2443-3100, groupby :485-669, _try_csr_merge :3740, _densify_all :3698
CSRLinearExpression already has the shape of a store interface, mirroring the frontend one to one (summed, scaled, added, contracted, taken, reindexed, filled, renamed, with_const); the dense equivalent is inline xarray code in BaseExpression, mixed with alignment and semantics checks linopy/csr.py:262-694
Constraints Constraint and CSRConstraint are siblings under ConstraintBase(ABC) with about twenty abstract methods each implements; CSRConstraint stores a raw csr_array plus seven side arrays, not a CSRLinearExpression, so lhs and data rebuild a Dataset on every access and from_csr / from_dense are separate conversions; _csr means different things on the two classes linopy/constraints.py:145, :762, :797-812, :1229, :1595, :1664, :1739
Model the read-only sparse key is clean; three reads remain (groupby sum, @, add_constraints fast path) linopy/model.py:572, :1375; expressions.py:615, :2888
Export and solvers mostly polymorphic through ConstraintBase; MatrixAccessor and the dual write-back still switch on type; Model.copy densifies model-level expressions linopy/matrices.py:48-55, :289; model.py:2533; io.py:1424
Tests equivalence by twin builds (twin_models, assert_sparse_matches, assert_frozen_equal) and ten hand-picked sparse parametrizations; no backend fixture, no conformance suite over the store operations test/test_csr.py

Reproduction of the implicit fallback:

import warnings
import numpy as np, pandas as pd, linopy

linopy.options["semantics"] = "v1"
linopy.options["warn_on_densify"] = True
i, t = pd.RangeIndex(20, name="i"), pd.RangeIndex(5, name="t")
m = linopy.Model(sparse=True)
x = m.add_variables(0, 1, coords=[i, t], name="x")
g = pd.Series(np.arange(20) % 3, index=i, name="g")

for op in ["shift(t=1)", "diff('t')", "coeffs", "to_quadexpr()", "isnull()"]:
    s = x.groupby(g).sum()
    assert s.is_sparse
    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        eval(f"s.{op}")
    msgs = [str(x.message) for x in w if "dens" in str(x.message).lower()]
    print(f"{op:16} sparse after: {s.is_sparse!s:5}  warning: {msgs}")

Output on master (69dd5c97):

shift(t=1)       sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
diff('t')        sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
coeffs           sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
to_quadexpr()    sparse after: False  warning: ['Sparse (CSR) backing densified: `.data` read, e.g. by an operation without a sparse path.']
isnull()         sparse after: True   warning: []
Plan
Step Content Depends on
0: legacy removal Make v1 the default in the next release, then delete legacy semantics (linopy/semantics.py and the is_v1() gates). Not part of this issue, but the prerequisite for steps 1, 3 and 4: those steps rewrite exactly the methods that carry the gates, so doing them first means doing them twice. #845, #846
1: chokepoint One _delegate(name, ...) on LinearExpression: an operation either has a sparse implementation or converts with reason=name. The 34 branches become a dispatch table mapping frontend methods to CSRLinearExpression counterparts. .data becomes a plain read through the same chokepoint. The receiver keeps its backing; the converted copy is the operand of the fallback. Partial fallbacks (_selected, _sparse_matmul, _try_csr_merge, csr_rhs) return a reason instead of calling _densify_all. The table is the authoritative list of backing-preserving operations: #971 documents it, #1011 adds rows. 0
2: conformance tests A backend fixture (dense, sparse) and one parametrized suite that runs every table entry on both stores and compares cells, terms and isnull, plus a test that every fallback warns with its own operation name. Replace twin builds where the fixture covers them. Only needs require_v1 and a list of operations, so it can land before step 0 and then guards the legacy removal as well. –
3: constraint store CSRConstraint holds a CSRLinearExpression plus sign, rhs, scaling and labels. lhs returns a sparse LinearExpression without a rebuild, from_csr becomes a constructor, from_dense goes through CSRLinearExpression.from_dense. Then Constraint and CSRConstraint collapse into one class with a frozen flag whose storage is its lhs; ConstraintBase goes away. MatrixAccessor and the dual write-back lose their isinstance branches. 0, 1, 2
4: dense store Move the xarray Dataset manipulation out of BaseExpression into a DenseLinearExpression implementing the same protocol as CSRLinearExpression. LinearExpression keeps only the user API, alignment and semantics checks, and delegates to its store. Align the store method names with the frontend names in this step. 0, 3

Ordering notes:

  • Step 2 first, at any time. Step 0 after the release. Steps 1, 3 and 4 back to back after step 0.
  • Step 1 before #1011 would absorb its new branches, but #1011 is user-facing and may land first; the four extra hand-written branches are then absorbed by step 1.
  • #1010 is internal to csr.py and independent of this plan.
  • Step 4 depends on step 0 because the sparse form is valid only under v1 (linopy/csr.py:1-27): a store protocol that one backend implements only under one semantics is worse than the current branches.

Rejected:

  • A lazy expression graph (already rejected in #972).
  • A sparse store for variables and one-term expressions (no gain in the #972 benchmark).
  • Two public classes (DenseLinearExpression / SparseLinearExpression) at the frontend. Users should not pick a class; the model key picks the store.
Children
  • #845, #846 (step 0: v1 default, then legacy semantics removal)
  • #1020 (step 1: densification chokepoint and dispatch table)
  • step 2: backend-parametrized conformance suite
  • step 3: CSRConstraint on a CSRLinearExpression, single Constraint class
  • step 4: DenseLinearExpression store
Ngôn ngữ chính
Python
Star
260
Fork
89
Merge trung bình
1 ngày 7 giờ
Pull request đã merge (30 ngày)
25

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của PyPSA/linopy

Tất cả issue của PyPSA/linopy

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.