apache/dolphinscheduler

[Improvement][K8s] too old resource version

Open

#17330 opened on Jul 9, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (11,659 stars) (4,324 forks)batch import
help wantedimprovement

Description

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

I use the K8s plugin to complete job,but if the job takes a long time ,it will cause the problem. Here is the log： [LOG-PATH]: /data/tsf/dolphinscheduler/worker-server/logs/20250709/18098005626112_15-3763-56288.log, [HOST]: Host{address='11.157.0.108:1234', ip='11.157.0.108', port=1234}

[INFO] 2025-07-09 09:46:46.094 +0800 - Begin to pulling task

[INFO] 2025-07-09 09:46:46.094 +0800 - Begin to initialize task

[INFO] 2025-07-09 09:46:46.094 +0800 - Set task startTime: Wed Jul 09 09:46:46 GMT+08:00 2025

[INFO] 2025-07-09 09:46:46.094 +0800 - Set task envFile: /data/tsf/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh

[INFO] 2025-07-09 09:46:46.095 +0800 - Set task appId: 3763_56288

[INFO] 2025-07-09 09:46:46.095 +0800 - End initialize task

[INFO] 2025-07-09 09:46:46.095 +0800 - Set task status to TaskExecutionStatus{code=1, desc='running'}

[INFO] 2025-07-09 09:46:46.095 +0800 - TenantCode:tencent check success

[INFO] 2025-07-09 09:46:46.095 +0800 - ProcessExecDir:/tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288 check success

[INFO] 2025-07-09 09:46:46.095 +0800 - Resources:{} check success

[INFO] 2025-07-09 09:46:46.095 +0800 - Task plugin: K8S create success

[INFO] 2025-07-09 09:46:46.095 +0800 - Success initialized task plugin instance success

[INFO] 2025-07-09 09:46:46.095 +0800 - Success set taskVarPool: null

[INFO] 2025-07-09 09:46:46.098 +0800 - [K8sJobExecutor-sr-sql-1-56288] start to submit job

[INFO] 2025-07-09 09:46:52.031 +0800 - [K8sJobExecutor-sr-sql-1-56288] submitted job successfully

[ERROR] 2025-07-09 10:19:01.458 +0800 - [K8sJobExecutor-sr-sql-1-56288] fail in k8s: too old resource version: 44517703052 (44517715422)

[INFO] 2025-07-09 10:19:01.460 +0800 - Send task execute result to master, the current task status: TaskExecutionStatus{code=6, desc='failure'}

[INFO] 2025-07-09 10:19:01.460 +0800 - Remove the current task execute context from worker cache

[INFO] 2025-07-09 10:19:01.460 +0800 - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288

[INFO] 2025-07-09 10:19:01.460 +0800 - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
javakubernetes
Domain
backenddevops
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaKubernetesApache DolphinScheduler architecture
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Examine the K8s plugin source code in the dolphinscheduler repository, specifically the classes related to Kubernetes job execution and resource watching. The error 'too old resource version' indicates that the watch connection has become stale and the client's resource version is outdated. Look for how the K8s client is initialized and how it handles watch timeouts or reconnections. A potential fix could involve restarting the watch with the latest resource version or configuring a longer timeout. Also check if there are any existing configurations for K8s client properties in the DolphinScheduler configuration files. Refer to the Kubernetes client library documentation for handling 'too old resource version' errors.