apache/dolphinscheduler

[Improvement][K8s] too old resource version

Open

#17,330 建立於 2025年7月9日

在 GitHub 查看
 (0 留言) (0 反應) (0 負責人)Java (11,659 star) (4,324 fork)batch import
help wantedimprovement

描述

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

I use the K8s plugin to complete job,but if the job takes a long time ,it will cause the problem. Here is the log： [LOG-PATH]: /data/tsf/dolphinscheduler/worker-server/logs/20250709/18098005626112_15-3763-56288.log, [HOST]: Host{address='11.157.0.108:1234', ip='11.157.0.108', port=1234}

[INFO] 2025-07-09 09:46:46.094 +0800 - Begin to pulling task

[INFO] 2025-07-09 09:46:46.094 +0800 - Begin to initialize task

[INFO] 2025-07-09 09:46:46.094 +0800 - Set task startTime: Wed Jul 09 09:46:46 GMT+08:00 2025

[INFO] 2025-07-09 09:46:46.094 +0800 - Set task envFile: /data/tsf/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh

[INFO] 2025-07-09 09:46:46.095 +0800 - Set task appId: 3763_56288

[INFO] 2025-07-09 09:46:46.095 +0800 - End initialize task

[INFO] 2025-07-09 09:46:46.095 +0800 - Set task status to TaskExecutionStatus{code=1, desc='running'}

[INFO] 2025-07-09 09:46:46.095 +0800 - TenantCode:tencent check success

[INFO] 2025-07-09 09:46:46.095 +0800 - ProcessExecDir:/tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288 check success

[INFO] 2025-07-09 09:46:46.095 +0800 - Resources:{} check success

[INFO] 2025-07-09 09:46:46.095 +0800 - Task plugin: K8S create success

[INFO] 2025-07-09 09:46:46.095 +0800 - Success initialized task plugin instance success

[INFO] 2025-07-09 09:46:46.095 +0800 - Success set taskVarPool: null

[INFO] 2025-07-09 09:46:46.098 +0800 - [K8sJobExecutor-sr-sql-1-56288] start to submit job

[INFO] 2025-07-09 09:46:52.031 +0800 - [K8sJobExecutor-sr-sql-1-56288] submitted job successfully

[ERROR] 2025-07-09 10:19:01.458 +0800 - [K8sJobExecutor-sr-sql-1-56288] fail in k8s: too old resource version: 44517703052 (44517715422)

[INFO] 2025-07-09 10:19:01.460 +0800 - Send task execute result to master, the current task status: TaskExecutionStatus{code=6, desc='failure'}

[INFO] 2025-07-09 10:19:01.460 +0800 - Remove the current task execute context from worker cache

[INFO] 2025-07-09 10:19:01.460 +0800 - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288

[INFO] 2025-07-09 10:19:01.460 +0800 - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/tencent/17858597217280/18098005626112_15/3763/56288

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

貢獻者指南

技術棧
javakubernetes
領域
backenddevops
議題類型
bug
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-2 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
fresh
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
JavaKubernetesApache DolphinScheduler architecture
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
40
研究方向
Examine the K8s plugin source code in the dolphinscheduler repository, specifically the classes related to Kubernetes job execution and resource watching. The error 'too old resource version' indicates that the watch connection has become stale and the client's resource version is outdated. Look for how the K8s client is initialized and how it handles watch timeouts or reconnections. A potential fix could involve restarting the watch with the latest resource version or configuring a longer timeout. Also check if there are any existing configurations for K8s client properties in the DolphinScheduler configuration files. Refer to the Kubernetes client library documentation for handling 'too old resource version' errors.