Meta: separate dense and sparse stores from the expression and constraint frontend

This is a meta issue; start with child #1020 for the first step, and read the referenced sections in linopy/expressions.py and linopy/csr.py to understand the current dispatch and sparse operations. The plan also identifies test/test_csr.py, constraints.py, matrices.py, and model.py for later steps. Done means completing the staged store refactor and conformance tests; several steps depend on other work, including legacy-semantics removal.

[!NOTE]
The following content was generated by AI.

Describe the feature you'd like to see

Meta issue. With #972 the sparse (CSR) backing is complete and observable, and Model(sparse=True) makes dense and sparse two equivalent ways to hold the same model. The code does not reflect that yet: sparse support is layered onto the dense classes by overrides and isinstance switches. This issue fixes the target architecture and the order of the refactoring. Child issues are spawned per step. Code links point to master at f6da2877.

Revision 2026-10-22: legacy semantics are gone (#1041, #1042), so the plan targets v1-only code and the 1.0 release. 1.0 is a breaking release; the plan removes compatibility layers instead of deprecating them. The densification chokepoint and the dense store (formerly steps 1 and 4) are one step, the dispatch table is dropped, the KeySet step is dropped. A second revision the same day folds in a review of the plan against the code: fallbacks are per argument case, not per method; a sparse operand pulls a merge sparse; Grid is the declared key set of the store protocol; step 4 is split in three.

Target

One LinearExpression class and one Constraint class. The expression always holds one store object, dense or sparse, behind one Store protocol. The constraint holds its lhs expression plus sign, right-hand side, labels, scaling and dual, and never learns about stores. Model.store_cls picks the store at the construction seam; nothing downstream branches on it. Every conversion between stores passes through Store.to_dense(reason) / Store.to_sparse(reason), which name the operation. Both stores expose .grid: the store abstracts term storage on a fixed Grid, not the Dataset. LinearExpression is mutable by store swap; Constraint.lhs returns a new wrapper around the current store on each access. Model, export, persistent diff and solvers never branch on the store.

Where the coupling is today

Reproduction of the implicit fallback:

import warnings
import numpy as np, pandas as pd, linopy

linopy.options["warn_on_densify"] = True
i, t = pd.RangeIndex(20, name="i"), pd.RangeIndex(5, name="t")
m = linopy.Model(sparse=True)
x = m.add_variables(0, 1, coords=[i, t], name="x")
g = pd.Series(np.arange(20) % 3, index=i, name="g")

for op in ["coeffs", "to_quadexpr()", "isnull()"]:
    s = x.groupby(g).sum()
    assert s.is_sparse
    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        eval(f"s.{op}")
    msgs = [str(x.message) for x in w if "dens" in str(x.message).lower()]
    print(f"{op:16} sparse after: {s.is_sparse!s:5}  warning: {msgs}")

The receiver is converted in place and the warning does not name the operation.

Plan

Ordering notes:

• Step 1 first, then 2 and 3 back to back (3 is small once 2 exists), then 4a, 4b, 4c. Step 5 is independent of 2 to 4 and can start once step 1 exists.
• A KeySet interface (former step 5) is dropped: an interface with one implementation and a deferred second one. Instead Grid is the declared key set of the store protocol from step 2 on, so a table-based key set later touches construction only. Grid already carries the netcdf, alignment and concatenation paths.
• Sparse and dense results agree under assert_equivalent, not on term order, padding or explicit zeros. The docs state this once; the code does not pay for layout identity.
• .data on a sparse expression keeps converting with a notice rather than raising. After step 2 only user code reads .data.

Long-form data

PyPSA plans a long-form data model built on datarecord (beta). Measurements on a synthetic PyPSA-like model (10k generators, 1k buses, 2920 snapshots):

• Converting long form to a stacked 1-D dim (one entry per existing generator) and building the current stores is feasible: 9.5 s / 3.8 GB peak for the sparse store including the pivot. A naive build straight from the long table took 11.9 s. Only a full generator × bus cube is prohibitive (fill about 0.1 %). A cube of a long-form parameter costs less memory than the long table above about 40–45 % fill.
• A hand-rolled numpy build with the same layout (static relations as tables, time axis by stride) assembles the same matrix in 2.4 s / 3.2 GB peak. It is a lower bound for a table-based key set, not a linopy prototype.

So long form is served by step 5 now. A table-based key set (coords= a data frame, alignment as an equi-join) is deferred until datarecord is stable and a PyPSA benchmark shows the gap matters. Only join="override" and operations that depend on axis order have no meaning on a table key set.

Rejected:

• A lazy expression graph (already rejected in #972).
• Two public classes (DenseLinearExpression / SparseLinearExpression) at the frontend. Users should not pick a class; the model key picks the store.
• A dispatch table mapping frontend methods to sparse counterparts (former step 1), and its successor, a per-method fallback as base-class default. Fallbacks are per argument case (reindex on method=, where on drop=, sum on non-grid dims, ...), so the authoritative list is the conformance suite, not the class.
• A cached dense twin on the sparse store. It doubles memory exactly where .data is read today (netcdf, Model.copy); those paths go through the store instead.
• "The model's store wins" for mixed merges. Fallback results, constants and from_rule produce dense expressions in a sparse model; converting them up silently at every + and down at every unsupported operation hides a ping-pong. A sparse operand pulls sparse, as today, and the conversion up is named.
• A sparse store for variables. Variables stay label arrays; the store choice starts at the variable → expression step (one-term expressions are sparse since #1034).
• A long table as a third term store. Long form changes which cells exist, not how terms are stored.
• Deprecating the freeze API. 1.0 is the breaking release; aliases with a FutureWarning through 1.x would outlive the concept they alias.

Children

• #846 (legacy semantics removal, #1041); #845 is superseded
• #1034 (PR: one-term sparse expressions and sparse broadcast in merges)
• #1044 (step 1: backend-parametrized conformance suite with assert_equivalent)
• #1020 (step 2: store protocol with per-case fallback reasons)
• #1045 (step 3: Model.store_cls and the store constructors as the construction seam)
• #1046 (step 4a: dense Constraint over lhs with derived data)
• #1047 (step 4b: one Constraint class, CSRConstraint / ConstraintBase / freeze API deleted; closes #1033)
• #1048 (step 4c: one netcdf layout written from the store)
• #1049 (step 5: long-form inputs and relation group-by)