[Bug]: salt-call causes ~60s hangs on minions due to ZMQ identity collision with running salt-minion daemon (3006.27 regression)

オープン
  • 何か問題がありますか？
#70,127 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

  • #70140 @waterWang による — マージされずにクローズ
  • #70196 @jzandbergen による — マージされずにクローズ

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
58/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
distributed-systems, networking

調査の方向性

salt/transport/zeromq.py から始め、次に AsyncReqMessageClient と、コミット 36e914a72af330e8e0c09bfea5f43893f5e972b1 で導入された minion/syndic の identity branch を追跡します。salt-minion daemon の実行中に salt-call を再現し、その後、それらの ZMQ identities が衝突しなくなり、pillar requests が完全な REQUEST_TIMEOUT を待たないことを確認します。

索引モデルが issue の本文から書いたものです。

説明

bug needs-triage
What happened?
What happened?

Pillar (and any other) requests made via salt-call on a minion take ~60 seconds
(the full REQUEST_TIMEOUT) to complete on 3006.27, when the same command
completes in a few seconds on 3006.26. The affected code path is unchanged
in 3006.26, so this is a regression introduced in 3006.27.

[06:31:40] alma98-p3g ~ # time salt-call pillar.data dontcarejustatest
local:
    ----------
    dontcarejustatest:

real    1m3.404s
user    0m1.045s
sys    0m0.137s

Trace logging on the minion shows the request hangs for ~60s waiting on a
reply from the master before finally timing out and retrying successfully:

[DEBUG   ] Initializing new AsyncAuth for (...)
[TRACE   ] ext_pillar_extra_data = {}
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key

<-- ~60s stall -->

[TRACE   ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE   ] Request timed out while waiting for a response. reconnecting.
Root cause

3006.27 includes commit 36e914a72af330e8e0c09bfea5f43893f5e972b1
("Use stable ZMQ identity for minion + syndic daemon ret-port REQ
sockets"), which sets a stable ZMQ IDENTITY on the AsyncReqMessageClient
REQ socket for long-lived minion/syndic daemons:

elif _role in ("minion", "syndic") and _minion_id:
    identity = "salt-req/{role}/{minion_id}/{slot}".format(
        role=_role,
        minion_id=_minion_id,
        slot=next(_REQ_IDENTITY_SLOT),
    )
    self.socket.setsockopt(zmq.IDENTITY, identity.encode("utf-8"))

_REQ_IDENTITY_SLOT is a module-level itertools.count() that starts at 0
per process. A running salt-minion daemon and a one-off salt-call
invocation both set opts["__role"] = "minion" and share the same
minion_id, so both processes independently compute the identity
salt-req/minion/<minion_id>/0 for their first REQ socket.

Because the master's ROUTER socket has ROUTER_HANDOVER=1 set (see
salt/transport/zeromq.py, MWorkerQueue), when the second process
(salt-call) connects with the same identity as the first (salt-minion),
libzmq hands the peer slot over to the newcomer and silently drops any
reply still in flight to the original owner. This causes whichever process
was waiting on that reply to stall for the full REQUEST_TIMEOUT (60s)
before retrying.

This directly contradicts the stated goal of the commit itself, which
explicitly calls out that giving two sockets the same identity causes
exactly this failure mode ("Giving them all the same stable identity would
cause ROUTER_HANDOVER on the upstream ROUTER to silently drop any reply
still in flight..."), but the fix only accounted for multiple sockets
within one process, not multiple processes (salt-minion and
salt-call) sharing the same minion_id and hence the same identity
namespace.

Suggested fix

Include something process-unique (e.g. os.getpid()) in the identity
string for the minion/syndic branch, so that salt-call,
salt-minion, and any other short-lived process using the minion config
never collide with the long-running daemon's identity slots.

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

almalinux-9

salt --versions-report output
salt-minion 3006.27 (Sulfur)
主要言語
Python
スター
15.8k
フォーク
5.6k
平均マージ
5日 9時間
マージ済み PR（30日）
99

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

saltstack/salt のほかの issue

saltstack/salt の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。