spec: sparse parameter data is densified at build and in model.parameters
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 领域
- backend, performance
调研方向
Start with the sparse parameter attach and storage path used by Model.from_spec, then inspect how model.parameters represents dimensioned data and where constraint rows consume parameters. The issue provides a reproducible example and memory measurements; compare stored and peak memory with the expected input-sized behavior while checking that constraints still receive the needed values.
由索引模型根据 Issue 内容生成。
描述
[!NOTE]
The following content was generated by AI.
Found in the API review for #1068 (stacked on #958); linopy.spec is unreleased.
Describe the feature you'd like to see
Sparse parameter data is densified onto the full grid of its dimensions at attach time, and the dense array is kept in model.parameters. 50k rows on an 8760 x 2000 grid occupy about 0.8 MB as input but 140 MB as stored parameter, and the build peaks at about 500 MB traced memory.
Minimal reproducible example
import tracemalloc
import warnings
import numpy as np
import pandas as pd
import linopy
warnings.simplefilter("ignore", linopy.EvolvingAPIWarning)
SPEC = """
dimensions: {t: {dtype: int}, g: {}}
parameters: {c: {dims: [t, g], missing: absent}}
variables: {x: {dims: [g], bounds: {lower: 0, upper: 1}}}
objective: {sense: minimize, expression: "sum(x)"}
"""
T = pd.Index(range(8760), name="t")
G = pd.Index([f"g{i}" for i in range(2000)], name="g")
rng = np.random.default_rng(0)
idx = pd.MultiIndex.from_arrays([rng.integers(0, len(T), 50_000), G[rng.integers(0, len(G), 50_000)]], names=["t", "g"])
c = pd.Series(1.0, index=idx[~idx.duplicated()])
print(f"sparse input: {len(c)} rows, {c.memory_usage(deep=True) / 1e6:.1f} MB")
tracemalloc.start()
m = linopy.Model.from_spec(SPEC, {"t": T, "g": G, "c": c})
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
stored = m.parameters["c"]
print(f"m.parameters['c']: shape {stored.shape}, {stored.nbytes / 1e6:.1f} MB, {int(stored.notnull().sum())} non-NaN")
print(f"peak traced during from_spec: {peak / 1e6:.0f} MB")
Output on spec-simplification:
sparse input: 49938 rows, 0.8 MB
m.parameters['c']: shape (8760, 2000), 140.2 MB, 49938 non-NaN
peak traced during from_spec: 500 MB
Expected
Memory proportional to the supplied rows, not to the product of the dimension sizes, both while building and in model.parameters.
Proposal
Keep sparse parameters sparse through attach and storage (e.g. a long-format or sparse-backed array), densifying only where a constraint row needs it.
- 主要语言
- Python
- 星标
- 263
- 派生
- 92
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 32
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
PyPSA/linopy 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
Constant LinearExpression * multi-term LinearExpression raises TypeError (mirrored product works)可能已有人在做 @anveshachuri 于 2 天前认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
Dense LinearExpression.to_constraint('==') keeps sign '==': repr raises KeyError, differs from sparse '='可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭bug
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
bug solver interface
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
-
Validate finite output clamping limits in StepPredictor.prepare_clamping_params to prevent silent NaN/Inf propagation可能已有人在做 @muskan-creates352 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
mllam/neural-lam#848 ·
维护者通常 2 天内回复
-
infra
难度 1/5 1 小时以内 新手友好度 88/100
fab-geocommuns/RNB-coeur#1166 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
deepspeedai/DeepSpeed#8951 ·
维护者通常 2 天内回复