[Documentation] Worker for executing multiple Python functions
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- distributed-systems, hpc
Research direction
The issue names no file or test. Start by comparing SlurmClusterExecutor and FluxClusterExecutor with SlurmJobExecutor and FluxJobExecutor, including the existing block_allocation behavior described in the issue. Clarify whether the goal is documentation or a new worker capability, then define completion around executing multiple Python functions on shared cluster resources.
Written by the indexing model from the issue text.
Description
There are basically three ways to use a Python Task Managers:
- Submit each Python Function as an individual job to the HPC cluster - supported in executorlib by the
SlurmClusterExecutorand theFluxClusterExecutor. - Submit multiple Python Functions within one HPC job allocation - supported in executorlib by the
SlurmJobExecutorand theFluxJobExecutor. - Submit multiple Python Functions to the same resources on a given HPC cluster - similar to the
block_allocation=Truefor theSingleNodeExecutor, theSlurmJobExecutorand theFluxJobExecutorbut for theSlurmClusterExecutorand theFluxClusterExecutor. This is the standard use case for solutions like https://www.dask.org and https://parsl.readthedocs.io/ so it might be an interesting addition to executorlib.
- Dominant language
- Python
- Stars
- 77
- Forks
- 7
- Avg merge
- 10h 32m
- Merged PRs (30d)
- 12
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pyiron/executorlib
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pyiron/executorlib#1054 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pyiron/executorlib#1032 ·
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pyiron/executorlib#1005 ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
pyiron/executorlib#1049 · 1 comment ·
-
documentation
Difficulty 3/5 1-2 days Newbie friendliness 35/100
pyiron/executorlib#999 · 1 comment ·
All issues in pyiron/executorlib
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
learningequality/ricecooker#747 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
run-llama/llama_index#23199 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/glTF-Blender-IO#2769 ·