Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Slurm] Run collection natively on CPU nodes and support portable publication

クローズ
  • 何か問題がありますか？
#964 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
42/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
infrastructure

調査の方向性

Begin by tracing CollectionPlan, the generated collection scripts, the CPU Slurm job entry point, and the host-destination publication path described in the issue. Verify the existing unit, contract, security, formatting, and lint checks, then confirm that collection uses the captured interpreter and runtime bundle, avoids Enroot and client-SQSH, and publishes without overwriting existing destinations on unsupported filesystems.

索引モデルが issue の本文から書いたものです。

説明

task
Task Summary

Make the zero-GPU collection/merge job run through the submitting Data Designer shared Python environment instead of starting the pinned client SQSH with Enroot.

This removes an unnecessary dependency on Enroot, FUSE, and squashfuse on CPU-only nodes. Collection should have one native execution path, without a mode selector or fallback configuration.

The collection must also complete on filesystems that reject renameat2(RENAME_NOREPLACE) while preserving collision-safe, no-overwrite publication.

Technical Details & Implementation Plan
  • Capture the submitting process's absolute Python executable in the immutable CollectionPlan.
  • Verify inside the CPU allocation that:
    • the interpreter exists and is executable;
    • its installed data-designer-slurm version matches the resolved run plan;
    • the collection plan digest matches its persisted bytes;
    • the content-addressed runtime bundle matches its recorded digest.
  • Extract the pinned runtime bundle into allocation-local scratch and run the collection worker with the captured interpreter.
  • Remove collection's client-SQSH, Enroot, container-mount, and container-destination dependencies.
  • Operate directly on the profile-authorized host destination.
  • When atomic no-replace rename is unsupported, reserve the absent destination with an exclusive private directory and atomically replace that reservation with the completed staging directory.
  • Preserve existing collision detection and never use an exists() followed by an unprotected rename.
  • Migration compatibility is out of scope because data-designer-slurm is new and has no users.
Acceptance Criteria
  • Collection remains a one-node, zero-GPU CPU Slurm job.
  • Generated collection scripts contain no Enroot or client-SQSH invocation.
  • There is no execution-mode option or fallback selector.
  • The exact Python executable and pinned runtime code are immutable inputs.
  • Collection reads and publishes through host paths only.
  • Existing destinations are never overwritten.
  • Publication succeeds when RENAME_NOREPLACE returns an unsupported-filesystem error.
  • Unit, contract, security, formatting, and lint checks pass.
Investigation / Context

Related work:

  • #850 — Slurm epic
  • #869 — deterministic CPU collection
  • #870 — real-cluster acceptance
  • #874 — public Slurm service/profile integration
  • #938 — allocation-local Enroot scratch

No existing issue, pull request, branch, or discussion implements native collection or a safe unsupported-filesystem publication fallback.

Dependencies

None.

主要言語
Python
スター
2.3k
フォーク
211
平均マージ
3日 12時間
マージ済み PR（30日）
45

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA-NeMo/DataDesigner のほかの issue

NVIDIA-NeMo/DataDesigner の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。