apache/airflow

Display entire KubernetesExecutor worker pod logs when task is running instead of only the last 100 lines

Open

#48,710 opened on Apr 2, 2025

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
area:logginggood first issuekind:featureprovider:cncf-kubernetes

Description

Description

When a task is in the running state, only the last 100 lines are streamed from the worker pod (with KubernetesExecutor). The entire logs appear once the task is completed. The 100 line limit is hardcoded here.

We should allow this limit to be configurable or display all logs that have been generated so far instead of only the last 100 lines.

Use case/motivation

Users might need the entire log history or more context to ensure that long running tasks are executing as expected.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
pythonkubernetes
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Familiarity with Airflow KubernetesExecutorBasic Python
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
The issue identifies a hardcoded limit of 100 lines in `kubernetes executor.py`. The fix involves making `tail lines` configurable via an Airflow configuration option or environment variable. Review the existing configuration pattern in Airflow (e.g., in `airflow.cfg` or environment variables) and modify the KubernetesExecutor to read the limit from config. Consider backward compatibility by defaulting to 100. Files to modify: `airflow/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes executor.py` and possibly configuration files. No open PRs or blockers currently.