[Documentation] Call shell scripts with extra resources inside the submitted Python function

Open

#567 opened on Feb 10, 2025 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 3/5
Estimated time: 1-2 days
Newbie friendliness: 42/100

Issue type: Documentation
Clarity: Mostly clear
Activity status: Stale
Tech stack: python, shell
Domain: documentation, hpc

Research direction

Start with the linked nested-executors documentation and the repository’s existing HPC job documentation; no specific file is named in the issue. Document how SLURM shell scripts and Flux nested executors receive extra resources at submission time, using the shown Python example as the reference for what done looks like.

Written by the indexing model from the issue text.

Description

documentation

For SLURM each call of srun creates a new job step, so there is no need to assign any resources to the specific Python function, just assign the resources to the external shell script directly.

For flux there is the option to use nested executors:
https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#nested-executors

Still this option can also be assigned during the submission - just like all the other options:

from executorlib import FluxJobExecutor

def get_available_gpus(lst):
    import socket
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [
        (x.name, x.physical_device_desc, socket.gethostname())
        for x in local_device_protos if x.device_type == "GPU"
    ] + lst

with FluxJobExecutor() as exe:
    fs = []
    for i in range(1, 4):
        fs = exe.submit(
            get_available_gpus,
            lst=fs,
            resource_dict={"cores": 1, "gpus_per_core": 1, "flux_executor_nesting": True},
        )
    print(fs.result())

Dominant language: Python
Stars: 77
Forks: 7
Avg merge: 10h 32m
Merged PRs (30d): 12

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

slack-gif-creator: validate_gif() multiplies the last frame's duration by the frame count, so reported duration/fps are wrong (1.0s GIF reported as 1.8s) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

anthropics/skills#1811 · 1 comment · Opened Sep 23, 2026
App fails to start with gradio 6 (allowed by `gradio>=5.49.1`) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

speaches-ai/speaches#678 · Opened Sep 23, 2026
ModuleNotFoundError: No module named 'mcp.server.fastmcp'. Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

datalayer/mcp-compose#42 · Opened Sep 23, 2026
import spacy fails with ModuleNotFoundError: No module named 'click' Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

conda-forge/spacy-feedstock#177 · Opened Sep 23, 2026
humaneval: extract_function_body corrupts a fenced body-only reply, so the same correct code scores INCORRECT fenced and CORRECT unfenced Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

UKGovernmentBEIS/inspect_evals#2523 · Opened Sep 23, 2026

More Python issues