Understanding `CmdStanModel` Behavior with `os.fork`
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- operating-systems, performance
Research direction
Start with the interaction between CmdStanModel.sample, cpp_options={"STAN_THREADS": True}, and the os.fork path shown in the report. Reproduce the two cases with a minimal working example, then compare chain scheduling after fork; done means the discrepancy is explained and any necessary behavior or documentation change is covered by a regression test.
Written by the indexing model from the issue text.
Description
I've run into a somewhat odd (and likely edge-case) situation where CmdStanModel.sample behaves unexpectedly after a call to os.fork. I'm providing pseudocode now in the hopes that there is a fairly simple explanation, but happy to make a minimum working example if need be.
Say I run something structured like the following:
import os
from cmdstanpy import CmdStanModel
def run_job():
# Compile the model
mod = CmdStanModel(stanfile, cpp_options={"STAN_THREADS": True})
# Run sampling on a detached child process
pid = os.fork()
if pid == 0:
os.setsid()
mod.sample(data = data, chains=4)
os._exit()
else:
return pid
Even though my CPU has 32 cores, this will only run two chains at once--specifically, Chains 1 and 3 run, then Chains 2 and 4 run. If, however, I remove the cpp_options={"STAN_THREADS": True} kwarg, then everything behaves as expected: all chains sample in parallel.
Is there any immediate explanation for this discrepancy in behavior?
- Dominant language
- Python
- Stars
- 198
- Forks
- 81
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stan-dev/cmdstanpy
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
feature
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
-
feature method outputs
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
All issues in stan-dev/cmdstanpy
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·