Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Ray backend creates per-worker venvs that fill disk and crash with concurrent experiments

未关闭
  • 哪里不对？
#331 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
22/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
停滞
技术栈
python, pytorch
领域
ai-infra-agents, distributed-systems

调研方向

Start in agentlab/experiments/launch_exp.py around line 85, where ray.init(num_cpus=n_jobs) runs, and check how the working directory and pyproject.toml get packaged into worker runtime_env. Then look at the cleanup around ray.shutdown() near line 89. Done means the Ray temp directory is removed after each run and concurrent launches no longer each build a full venv, which needs a test or a reproduction with multiple Study.run() calls. The issue lists several competing fixes, so a maintainer decision on the approach is needed first.

由索引模型根据 Issue 内容生成。

描述

Summary

When using parallel_backend="ray" (the default), Ray auto-packages the working directory and creates a fresh virtual environment per worker in a temporary directory. For projects with heavy dependencies (e.g., PyTorch ~12GB), this causes:

  1. Disk exhaustion: Each Ray cluster creates a full venv copy (~12GB+). With 4-10 concurrent experiments, this can consume 50-120GB+ in /tmp, filling the root partition.
  2. Worker startup hangs: Workers hang during uv sync / pip install in the temp venv, producing repeated worker_pool.cc: Some workers of the worker process have not registered within the timeout errors.
  3. GCS crashes: When too many clusters compete for resources, Ray's GCS (Global Control Store) becomes unresponsive, causing Failed to connect to GCS within 60 seconds and terminating experiments.
  4. AF_UNIX socket path limit: If the temp directory path is long (e.g., a scratch filesystem), the Unix socket path exceeds the 107-byte limit, causing OSError: AF_UNIX path length cannot exceed 107 bytes.

Reproduction

  1. Have a project with pyproject.toml that depends on PyTorch (or similar large packages)
  2. Launch 3+ concurrent Study.run() calls with parallel_backend="ray" (default)
  3. Observe /tmp filling up with ray_* directories, each containing a full venv

Root Cause

In agentlab/experiments/launch_exp.py:85:

ray.init(num_cpus=n_jobs)

This bare ray.init() causes Ray to auto-detect the working directory (which contains pyproject.toml) and package it for workers. Each worker then runs uv sync to create a fresh venv in the Ray temp directory, re-installing all dependencies from scratch.

Key behaviors:

  • Ray creates a new temp directory per ray.init() call (each experiment gets its own cluster)
  • Each cluster's workers build an independent venv copy
  • Failed/completed experiments leave their temp directories behind (no cleanup)
  • ray.shutdown() in launch_exp.py:89 does not clean up the temp directory

Impact

  • Experiments silently fail with ENOSPC errors (Playwright can't create browser profiles when disk is full)
  • Hundreds of tasks get recorded as errors that are actually disk-full failures, requiring full reruns
  • The problem compounds: each relaunch creates additional temp directories

Workaround

Using parallel_backend="joblib" avoids Ray entirely and doesn't have this issue. However, joblib doesn't support Ray's task graph execution (dependency tracking between tasks).

Another workaround is to set RAY_TMPDIR to a large filesystem and create isolated Ray temp dirs there, but this hits the AF_UNIX 107-byte socket path limit if the path is too long.

Suggested Fix

  1. Disable Ray's auto-packaging by setting runtime_env={"worker_process_setup_hook": ...} or RAY_RUNTIME_ENV_HOOK to prevent venv creation in workers
  2. Or pass runtime_env={"py_modules": [...]} with only the necessary module instead of the full working directory
  3. Or set ray.init(runtime_env={"working_dir": None}) to prevent auto-packaging
  4. Add temp directory cleanup in the finally block after ray.shutdown() -- clean up the Ray temp directory

Environment

  • AgentLab v0.4.0
  • Ray 2.51.1
  • Python 3.12
  • Ubuntu 22.04
  • Dependencies include PyTorch 2.8.0 (~12GB installed)
主要语言
Python
星标
644
派生
134
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ServiceNow/AgentLab 的其他 Issue

查看 ServiceNow/AgentLab 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。