[Bug]: await_lock masks exceptions as FileLockError; minion job_queue requeues forever ("Failed to acquire job_queue lock" log flood)

未关闭
  • 哪里不对？
#70,229 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
45/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
devops, distributed-systems

调研方向

从 salt.utils.files.await_lock 开始，跟踪其围绕 with 代码体的异常处理，然后检查 _handle_decoded_payload 和 _process_process_queue_async_impl。确认代码体中的异常仍能与获取锁失败区分开来，确认 FileLockError 日志包含该异常，并确认反复失败的队列任务最终会停止重新入队，并向 master 返回错误。

由索引模型根据 Issue 内容生成。

描述

What happened?

On 3007.13 two minions got stuck in an infinite requeue loop, logging ~180 lines/s of

[salt.minion :1960][WARNING ] Failed to acquire job_queue lock for jid 2026..., queuing anyway.

for a week, until /var/log/salt/minion reached 9 GB and filled /var. 59 stale jids cycled every 0.3 s. No job_queue.lock file existed, fds were 45/8192, plenty of RAM. The loop started 9 s after a Timeout encountered while sending ... _return request.

Root cause, two defects that combine:

  1. salt.utils.files.await_lock hides the real exception. The yield is inside the outer try, whose except Exception re-raises everything as FileLockError("Error encountered obtaining file lock ..."). Any exception raised by the async with await_lock(...) body is reported as a lock failure and its traceback is lost. In 3007.13 the body included _invoke_execution() and the proc-file write, so a job-start failure became "Failed to acquire job_queue lock". 3007.14 moved _invoke_execution out of the lock, but await_lock itself is unchanged in 3008.x, and the minion's except FileLockError handler still logs without the exception.

  2. The disk job queue has no exit. _process_process_queue_async_impl resubmits every queued_*.p file every 0.3 s (0.2 s in 3008.x) with no retry counter, max age or backoff. A job that fails deterministically at start is requeued forever.

Workaround used: stop minion, move cachedir/job_queue aside, truncate log, start.

Expected
  • await_lock should not convert exceptions from the with body into FileLockError (re-raise them unchanged, keep the wrapping for the lock-acquire path only).
  • _handle_decoded_payload's except FileLockError should log the exception (exc_info=True).
  • Queued jobs need a retry limit and/or max age; after that the job is dropped with an error and a return to the master.
Type of salt install

onedir

Major version

3007 (3007.13); defect 1 and 2 still present in 3008.2 / master

What supported OS are you seeing the problem on?

Ubuntu

salt --versions-report output
Salt Version:
          Salt: 3007.13
Python: /opt/saltstack/salt/bin/python3.10 (onedir)
OS: Ubuntu, cPanel hosts
主要语言
Python
星标
15.8k
派生
5.6k
平均合并
5 天 9 小时
30 天内合并 PR
99

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

saltstack/salt 的其他 Issue

查看 saltstack/salt 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。