Pick better default for mp_context
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start in src/spikeinterface/core/job_tools.py around lines 418-440 and compare the earlier selection at lines 419-427. Review Python's multiprocessing context defaults and the reported Kilosort/scikit-learn hang, especially the referenced spikedetect.py line 72. Done means agreeing on and implementing a safe default or preserving None with clear documentation, then validating the affected workload.
Written by the indexing model from the issue text.
Description
#4333 changes how mp_context is chosen from:
to
Before the PR; job_kwargs.mp_context=None and recording.get_preferred_mp_context()=None (the default and most common setup) will call mp_context = multiprocessing.get_context(None) that returned the default context for that specific machine (spawn for windows & Mac and fork/forkserver for Linux). Now, the behavior has changed for Linux: it always picks fork. Default mp_context for Linux from python 3.14 onwards is forkserver (a thread-safe, less problematic alternative to fork) docs.
When running kilosort4 in python 3.14. fork did not play nice with some multi-threaded calls inside scikit learn, particularly the process hangs forever in this line:
https://github.com/MouseLand/Kilosort/blob/7a19a57bef39f3e07ab7fa1edf5aa8635c69a850/kilosort/spikedetect.py#L72
This was run inside a kubernetes pod with GPUs but I couldn't replicate it locally: 3.13 and 3.14 both worked with fork or forkserver :/ so not sure if it will be a problem for other people.
In any case, it might be worth considering making forkserver (rather than fork) the Linux default (it's been around since python 3.4) or allow multiprocessing to pick the best default based on system and python version as it did before this PR, i.e., let mp_context=None with a comment saying what that means).
- Dominant language
- Python
- Stars
- 858
- Forks
- 281
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 38
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from SpikeInterface/spikeinterface
-
testing
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
SpikeInterface/spikeinterface#4756 ·
Maintainers usually reply within 2 days
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
SpikeInterface/spikeinterface#4510 · 2 comments ·
Maintainers usually reply within 2 days
-
Difficulty 3/5 1-2 days Newbie friendliness 57/100
SpikeInterface/spikeinterface#4835 ·
Maintainers usually reply within 2 days
-
Extend `TimeSeriesExecutor` to `num_chunks_per_job`Possibly taken @samuelgarcia claimed this 1 day ago. Openconcurrency
SpikeInterface/spikeinterface#4831 · 1 comment · 1 assignee ·
Maintainers usually reply within 2 days
-
enhancement motion correction
Difficulty 3/5 Half a day Newbie friendliness 65/100
SpikeInterface/spikeinterface#4829 · 2 comments · 1 reaction ·
Maintainers usually reply within 2 days
All issues in SpikeInterface/spikeinterface
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Task
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
war-and-code/dircue#200 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day