Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Ray backend creates per-worker venvs that fill disk and crash with concurrent experiments

オープン
  • 何か問題がありますか？
#331 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
22/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
python, pytorch
領域
ai-infra-agents, distributed-systems

調査の方向性

Start in agentlab/experiments/launch_exp.py around line 85, where ray.init(num_cpus=n_jobs) runs, and check how the working directory and pyproject.toml get packaged into worker runtime_env. Then look at the cleanup around ray.shutdown() near line 89. Done means the Ray temp directory is removed after each run and concurrent launches no longer each build a full venv, which needs a test or a reproduction with multiple Study.run() calls. The issue lists several competing fixes, so a maintainer decision on the approach is needed first.

索引モデルが issue の本文から書いたものです。

説明

Summary

When using parallel_backend="ray" (the default), Ray auto-packages the working directory and creates a fresh virtual environment per worker in a temporary directory. For projects with heavy dependencies (e.g., PyTorch ~12GB), this causes:

  1. Disk exhaustion: Each Ray cluster creates a full venv copy (~12GB+). With 4-10 concurrent experiments, this can consume 50-120GB+ in /tmp, filling the root partition.
  2. Worker startup hangs: Workers hang during uv sync / pip install in the temp venv, producing repeated worker_pool.cc: Some workers of the worker process have not registered within the timeout errors.
  3. GCS crashes: When too many clusters compete for resources, Ray's GCS (Global Control Store) becomes unresponsive, causing Failed to connect to GCS within 60 seconds and terminating experiments.
  4. AF_UNIX socket path limit: If the temp directory path is long (e.g., a scratch filesystem), the Unix socket path exceeds the 107-byte limit, causing OSError: AF_UNIX path length cannot exceed 107 bytes.

Reproduction

  1. Have a project with pyproject.toml that depends on PyTorch (or similar large packages)
  2. Launch 3+ concurrent Study.run() calls with parallel_backend="ray" (default)
  3. Observe /tmp filling up with ray_* directories, each containing a full venv

Root Cause

In agentlab/experiments/launch_exp.py:85:

ray.init(num_cpus=n_jobs)

This bare ray.init() causes Ray to auto-detect the working directory (which contains pyproject.toml) and package it for workers. Each worker then runs uv sync to create a fresh venv in the Ray temp directory, re-installing all dependencies from scratch.

Key behaviors:

  • Ray creates a new temp directory per ray.init() call (each experiment gets its own cluster)
  • Each cluster's workers build an independent venv copy
  • Failed/completed experiments leave their temp directories behind (no cleanup)
  • ray.shutdown() in launch_exp.py:89 does not clean up the temp directory

Impact

  • Experiments silently fail with ENOSPC errors (Playwright can't create browser profiles when disk is full)
  • Hundreds of tasks get recorded as errors that are actually disk-full failures, requiring full reruns
  • The problem compounds: each relaunch creates additional temp directories

Workaround

Using parallel_backend="joblib" avoids Ray entirely and doesn't have this issue. However, joblib doesn't support Ray's task graph execution (dependency tracking between tasks).

Another workaround is to set RAY_TMPDIR to a large filesystem and create isolated Ray temp dirs there, but this hits the AF_UNIX 107-byte socket path limit if the path is too long.

Suggested Fix

  1. Disable Ray's auto-packaging by setting runtime_env={"worker_process_setup_hook": ...} or RAY_RUNTIME_ENV_HOOK to prevent venv creation in workers
  2. Or pass runtime_env={"py_modules": [...]} with only the necessary module instead of the full working directory
  3. Or set ray.init(runtime_env={"working_dir": None}) to prevent auto-packaging
  4. Add temp directory cleanup in the finally block after ray.shutdown() -- clean up the Ray temp directory

Environment

  • AgentLab v0.4.0
  • Ray 2.51.1
  • Python 3.12
  • Ubuntu 22.04
  • Dependencies include PyTorch 2.8.0 (~12GB installed)
主要言語
Python
スター
644
フォーク
134
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ServiceNow/AgentLab のほかの issue

ServiceNow/AgentLab の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。