[Core]Add env var to disable log_monitor stdout/stderr file redirection

Open Beginner friendly
#66,054 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
88/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
kubernetes, python

Research direction

Start in ray/_private/logging_utils.py at the log_monitor StreamRedirector calls and trace the existing tee_to_stdout and tee_to_stderr parameters. Verify that RAY_LOG_MONITOR_TEE_STDOUT=1 reaches both redirect calls while unset or non-1 preserves current behavior; done means output is duplicated to terminal and log files without changes to Cython or C++.

Written by the indexing model from the issue text.

Description

community-backlog core enhancement observability triage usability
Description

Environment

  • Ray version: 2.58.0
  • Deployment: KubeRay on Kubernetes
  • Related code path: Python (logging_utils.py) → Cython (_raylet.pyx StreamRedirector) → C++ (stream_redirector.cc RedirectStdoutOncePerProcess)

Problem

log_monitor.py redirects stdout/stderr to log files on startup via StreamRedirector. The underlying Cython binding already supports tee_to_stdout and tee_to_stderr parameters, but they are hardcoded to False in ray/_private/logging_utils.py, making it impossible for users to enable tee mode (output to both file and terminal) without patching source code.

Proposed Solution

Add environment variable RAY_LOG_MONITOR_TEE_STDOUT=1. When set, log_monitor's output is written to both log files and stdout/stderr simultaneously.

Proposed Change in ray/_private/logging_utils.py

tee_enabled = os.environ.get("RAY_LOG_MONITOR_TEE_STDOUT", "0") == "1"

if stdout_filepath:
StreamRedirector.redirect_stdout(
stdout_filepath, rotation_bytes, rotation_backup_count,
tee_enabled, # tee_to_stdout (currently hardcoded False)
tee_enabled, # tee_to_stderr (currently hardcoded False)
)
if stderr_filepath:
StreamRedirector.redirect_stderr(
stderr_filepath, rotation_bytes, rotation_backup_count,
tee_enabled,
tee_enabled,
)

The change is minimal — one env var read, two False replacements. No C++ or Cython changes needed.

Use case

We run Ray on Kubernetes via KubeRay, and use a sidecar-based log collector (Fluentd/Loki) to aggregate all container stdout/stderr into our centralized logging platform.

When RAY_LOG_MONITOR_TEE_STDOUT=1:

log_monitor's output is written to both log files on disk and stdout/stderr. Log files remain available for debugging via container filesystem, and the sidecar log collector can simultaneously capture all output through the standard container log pipeline.

When RAY_LOG_MONITOR_TEE_STDOUT is unset or not equal to 1 (default, unchanged):

log_monitor redirects its stdout/stderr to log files only. No output reaches stdout/stderr.

Dominant language
Python
Stars
43.9k
Forks
8.1k
Avg merge
4d 15h
Merged PRs (30d)
245

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ray-project/ray

All issues in ray-project/ray

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.