[Feature]: Scheduler protocol + Slurm scheduler + generic BatchBackend

Open
#20 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python

Research direction

Start with IC-ADR-001 §3 and the Scheduler protocol requirements, then trace the planned _slurm module, Slurm registration via intercede.schedulers, and the generic BatchBackend interfaces. Run the integration contract suite against the ssh-slurm and local-slurm stacks first. Done means both stacks pass submit, status, fetch, kill, fetch-twice, and partial-failure checks, with array submission and the listed output and load-reporting behavior verified.

Written by the indexing model from the issue text.

Description

User Story

As the WMS consumer,
I want BatchBackend(SSHTransport, Slurm) and BatchBackend(LocalTransport, Slurm) working
end-to-end against the containerised Slurm stacks,
So that the N+M composition promise of IC-ADR-001 §3 is proven on the first real scheduler.

Feature Description

  • Scheduler protocol finalised: submit_cmd(spec), parse_status(raw), kill_cmd(ids),
    stages_own_files flag (+ stage_inputs/collect_outputs hooks where needed). Evaluate the
    ADR's open question — promote staging to a Stager collaborator — and record the outcome.
  • _slurm internal module (Tier C): sbatch script/option generation from SubmissionSpec
    (incl. count--array for identical fan-out), squeue/sacct parsing, native-state →
    JobStatus map (port DIRAC's mapping: PENDING/SUSPENDED/CONFIGURING→waiting, COMPLETED→done,
    CANCELLED/PREEMPTED→aborted…).
  • Slurm scheduler consuming _slurm; registered via intercede.schedulers.
  • Generic BatchBackend (Tier B): satisfies JobBackend + OutputRetriever
    (destructive=False) + Cancellable + Purgeable + LoadReporter; sandbox staging via
    Transport.put/get when stages_own_files is false; workdir layout per job; JobHandle
    carries host/workdir routing (replaces DIRAC's ssh<batch>:// ref encoding).
  • Phase 2 wiring: the integration contract suite (submit → status → fetch → kill, fetch-twice,
    partial-failure maps) runs against ssh-slurm and local-slurm stacks via markers.

Definition of Done

  • Contract suite green on ssh-slurm and local-slurm stacks in CI
  • count > 1 submits via a single --array (asserted, not N submits)
  • fetch_output streams an arbitrary output sandbox (not just stdout/stderr) with bounded
    materialisation (size/count/timeout/path-containment)
  • LoadReporter.counts() from squeue; unknown ids → JobStatus.UNKNOWN

Alternatives Considered

  • Per-combination classes (SSHSlurmBackend, LocalSlurmBackend…) — N×M explosion; rejected.
  • Porting DIRAC's remote-executed SLURM.py driver — requires remote python; dropped (see
    issue-10).

Additional Context

This issue is the template for every further scheduler.

Dominant language
Python
Stars
3
Forks
3
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DIRACGrid/intercede

All issues in DIRACGrid/intercede

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.