Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Slurm] Run collection natively on CPU nodes and support portable publication

Cerrado
  • ¿Algo no está bien?
#964 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
infrastructure

Línea de trabajo

Begin by tracing CollectionPlan, the generated collection scripts, the CPU Slurm job entry point, and the host-destination publication path described in the issue. Verify the existing unit, contract, security, formatting, and lint checks, then confirm that collection uses the captured interpreter and runtime bundle, avoids Enroot and client-SQSH, and publishes without overwriting existing destinations on unsupported filesystems.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

task
Task Summary

Make the zero-GPU collection/merge job run through the submitting Data Designer shared Python environment instead of starting the pinned client SQSH with Enroot.

This removes an unnecessary dependency on Enroot, FUSE, and squashfuse on CPU-only nodes. Collection should have one native execution path, without a mode selector or fallback configuration.

The collection must also complete on filesystems that reject renameat2(RENAME_NOREPLACE) while preserving collision-safe, no-overwrite publication.

Technical Details & Implementation Plan
  • Capture the submitting process's absolute Python executable in the immutable CollectionPlan.
  • Verify inside the CPU allocation that:
    • the interpreter exists and is executable;
    • its installed data-designer-slurm version matches the resolved run plan;
    • the collection plan digest matches its persisted bytes;
    • the content-addressed runtime bundle matches its recorded digest.
  • Extract the pinned runtime bundle into allocation-local scratch and run the collection worker with the captured interpreter.
  • Remove collection's client-SQSH, Enroot, container-mount, and container-destination dependencies.
  • Operate directly on the profile-authorized host destination.
  • When atomic no-replace rename is unsupported, reserve the absent destination with an exclusive private directory and atomically replace that reservation with the completed staging directory.
  • Preserve existing collision detection and never use an exists() followed by an unprotected rename.
  • Migration compatibility is out of scope because data-designer-slurm is new and has no users.
Acceptance Criteria
  • Collection remains a one-node, zero-GPU CPU Slurm job.
  • Generated collection scripts contain no Enroot or client-SQSH invocation.
  • There is no execution-mode option or fallback selector.
  • The exact Python executable and pinned runtime code are immutable inputs.
  • Collection reads and publishes through host paths only.
  • Existing destinations are never overwritten.
  • Publication succeeds when RENAME_NOREPLACE returns an unsupported-filesystem error.
  • Unit, contract, security, formatting, and lint checks pass.
Investigation / Context

Related work:

  • #850 — Slurm epic
  • #869 — deterministic CPU collection
  • #870 — real-cluster acceptance
  • #874 — public Slurm service/profile integration
  • #938 — allocation-local Enroot scratch

No existing issue, pull request, branch, or discussion implements native collection or a safe unsupported-filesystem publication fallback.

Dependencies

None.

Lenguaje dominante
Python
Estrellas
2.4k
Forks
234
Merge medio
20 h 48 min
PR fusionados (30 d)
78

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA-NeMo/DataDesigner

Todos los issues de NVIDIA-NeMo/DataDesigner

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.