log_prob leaves a temporary output directory per call until interpreter exit
评估
调研方向
The leak is in CmdStanModel.log_prob in cmdstanpy/model.py, where tempfile.mkdtemp creates an output directory under _TMPDIR that is never removed. Read how temp_single_json cleans up the neighbouring input files, then check the linked pull request #869 before starting, since it already targets this fix. Done means the output directory is removed on both the success and RuntimeError paths, and a test shows the _TMPDIR entry count is unchanged after repeated calls.
由索引模型根据 Issue 内容生成。
描述
Summary
Each call to CmdStanModel.log_prob creates a new directory under cmdstanpy._TMPDIR and never removes it. _TMPDIR is cleaned only at interpreter exit (atexit), so a long-running process accumulates one directory, holding an output.csv, per call.
Where
cmdstanpy/model.py, CmdStanModel.log_prob (1.3.0):
output_dir = tempfile.mkdtemp(prefix=self.name, dir=_TMPDIR)
output = os.path.join(output_dir, "output.csv")
...
result = pd.read_csv(output, comment="#")
return result
Nothing removes output_dir, on either the success path or the RuntimeError path. The data and params files right beside it use temp_single_json(...) context managers and are cleaned up.
Why it matters
We run log_prob once per optimizer start, as a gradient diagnostic, inside a long-lived worker process. A single fit runs hundreds of starts, and the worker runs indefinitely. The directories grow without bound until the process exits.
Today we work around it by calling the compiled program's log_prob method directly through subprocess, with an output directory we own. That means duplicating cmdstanpy's argument handling, which we would rather not do.
Suggested fix
Let log_prob own its output directory for the duration of the call, as the JSON inputs already do:
with tempfile.TemporaryDirectory(prefix=self.name, dir=_TMPDIR) as output_dir:
output = os.path.join(output_dir, "output.csv")
...
result = pd.read_csv(output, comment="#")
return result
read_csv loads everything before the directory is removed, so the returned DataFrame is unaffected. An optional output_dir= argument, as optimize/sample have, would also let callers keep the CSV when they want it.
We are happy to open a PR with a test, for example asserting that _TMPDIR's entry count is unchanged after N calls.
Versions
cmdstanpy 1.3.0, CmdStan 2.40.0, Python 3.12, Linux x86_64.
- 主要语言
- Python
- 星标
- 198
- 派生
- 82
- 平均合并
- 4 天 11 小时
- 30 天内合并 PR
- 3
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
stan-dev/cmdstanpy 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 62/100
-
documentation
难度 2/5 1-3 小时 新手友好度 70/100
-
laplace_sample: expose CmdStan's diagnostic_file (the Hessian at the mode)可能已有人在做 @atarutin 于 8 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 25/100
-
Release 2.0未关闭
难度 5/5 一周以上 新手友好度 30/100
-
feature
难度 5/5 一周以上 新手友好度 35/100
查看 stan-dev/cmdstanpy 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 64/100
aws-samples/sample-per-model-bedrock#295 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 76/100
pyrevitlabs/pyRevit#3792 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 69/100
zephyrproject-rtos/reqmgmt#177 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
polarsource/polar#15580 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
521xueweihan/HelloGitHub#4011 ·