Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Ray backend creates per-worker venvs that fill disk and crash with concurrent experiments

Ouverte
  • Quelque chose ne va pas ?
#331 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
22/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
À l'abandon
Stack technique
python, pytorch
Domaine
ai-infra-agents, distributed-systems

Piste de recherche

Start in agentlab/experiments/launch_exp.py around line 85, where ray.init(num_cpus=n_jobs) runs, and check how the working directory and pyproject.toml get packaged into worker runtime_env. Then look at the cleanup around ray.shutdown() near line 89. Done means the Ray temp directory is removed after each run and concurrent launches no longer each build a full venv, which needs a test or a reproduction with multiple Study.run() calls. The issue lists several competing fixes, so a maintainer decision on the approach is needed first.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

When using parallel_backend="ray" (the default), Ray auto-packages the working directory and creates a fresh virtual environment per worker in a temporary directory. For projects with heavy dependencies (e.g., PyTorch ~12GB), this causes:

  1. Disk exhaustion: Each Ray cluster creates a full venv copy (~12GB+). With 4-10 concurrent experiments, this can consume 50-120GB+ in /tmp, filling the root partition.
  2. Worker startup hangs: Workers hang during uv sync / pip install in the temp venv, producing repeated worker_pool.cc: Some workers of the worker process have not registered within the timeout errors.
  3. GCS crashes: When too many clusters compete for resources, Ray's GCS (Global Control Store) becomes unresponsive, causing Failed to connect to GCS within 60 seconds and terminating experiments.
  4. AF_UNIX socket path limit: If the temp directory path is long (e.g., a scratch filesystem), the Unix socket path exceeds the 107-byte limit, causing OSError: AF_UNIX path length cannot exceed 107 bytes.

Reproduction

  1. Have a project with pyproject.toml that depends on PyTorch (or similar large packages)
  2. Launch 3+ concurrent Study.run() calls with parallel_backend="ray" (default)
  3. Observe /tmp filling up with ray_* directories, each containing a full venv

Root Cause

In agentlab/experiments/launch_exp.py:85:

ray.init(num_cpus=n_jobs)

This bare ray.init() causes Ray to auto-detect the working directory (which contains pyproject.toml) and package it for workers. Each worker then runs uv sync to create a fresh venv in the Ray temp directory, re-installing all dependencies from scratch.

Key behaviors:

  • Ray creates a new temp directory per ray.init() call (each experiment gets its own cluster)
  • Each cluster's workers build an independent venv copy
  • Failed/completed experiments leave their temp directories behind (no cleanup)
  • ray.shutdown() in launch_exp.py:89 does not clean up the temp directory

Impact

  • Experiments silently fail with ENOSPC errors (Playwright can't create browser profiles when disk is full)
  • Hundreds of tasks get recorded as errors that are actually disk-full failures, requiring full reruns
  • The problem compounds: each relaunch creates additional temp directories

Workaround

Using parallel_backend="joblib" avoids Ray entirely and doesn't have this issue. However, joblib doesn't support Ray's task graph execution (dependency tracking between tasks).

Another workaround is to set RAY_TMPDIR to a large filesystem and create isolated Ray temp dirs there, but this hits the AF_UNIX 107-byte socket path limit if the path is too long.

Suggested Fix

  1. Disable Ray's auto-packaging by setting runtime_env={"worker_process_setup_hook": ...} or RAY_RUNTIME_ENV_HOOK to prevent venv creation in workers
  2. Or pass runtime_env={"py_modules": [...]} with only the necessary module instead of the full working directory
  3. Or set ray.init(runtime_env={"working_dir": None}) to prevent auto-packaging
  4. Add temp directory cleanup in the finally block after ray.shutdown() -- clean up the Ray temp directory

Environment

  • AgentLab v0.4.0
  • Ray 2.51.1
  • Python 3.12
  • Ubuntu 22.04
  • Dependencies include PyTorch 2.8.0 (~12GB installed)
Langage dominant
Python
Étoiles
644
Forks
134
Aucune PR mergée en 30 j

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ServiceNow/AgentLab

Toutes les issues de ServiceNow/AgentLab

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.