[Bug]: salt-call causes ~60s hangs on minions due to ZMQ identity collision with running salt-minion daemon (3006.27 regression)

Aperta
  • Qualcosa non va?
#70,127 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

  • #70140 di @waterWang — chiusa senza merge
  • #70196 di @jzandbergen — chiusa senza merge

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
58/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
distributed-systems, networking

Direzione di ricerca

Inizia da salt/transport/zeromq.py, quindi segui AsyncReqMessageClient e il ramo di identità minion/syndic introdotto dal commit 36e914a72af330e8e0c09bfea5f43893f5e972b1. Riproduci salt-call mentre è in esecuzione un daemon salt-minion, quindi verifica che le loro identità ZMQ non collidano più e che le richieste pillar non attendano l’intero REQUEST_TIMEOUT.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug needs-triage
What happened?
What happened?

Pillar (and any other) requests made via salt-call on a minion take ~60 seconds
(the full REQUEST_TIMEOUT) to complete on 3006.27, when the same command
completes in a few seconds on 3006.26. The affected code path is unchanged
in 3006.26, so this is a regression introduced in 3006.27.

[06:31:40] alma98-p3g ~ # time salt-call pillar.data dontcarejustatest
local:
    ----------
    dontcarejustatest:

real    1m3.404s
user    0m1.045s
sys    0m0.137s

Trace logging on the minion shows the request hangs for ~60s waiting on a
reply from the master before finally timing out and retrying successfully:

[DEBUG   ] Initializing new AsyncAuth for (...)
[TRACE   ] ext_pillar_extra_data = {}
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key

<-- ~60s stall -->

[TRACE   ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE   ] Request timed out while waiting for a response. reconnecting.
Root cause

3006.27 includes commit 36e914a72af330e8e0c09bfea5f43893f5e972b1
("Use stable ZMQ identity for minion + syndic daemon ret-port REQ
sockets"), which sets a stable ZMQ IDENTITY on the AsyncReqMessageClient
REQ socket for long-lived minion/syndic daemons:

elif _role in ("minion", "syndic") and _minion_id:
    identity = "salt-req/{role}/{minion_id}/{slot}".format(
        role=_role,
        minion_id=_minion_id,
        slot=next(_REQ_IDENTITY_SLOT),
    )
    self.socket.setsockopt(zmq.IDENTITY, identity.encode("utf-8"))

_REQ_IDENTITY_SLOT is a module-level itertools.count() that starts at 0
per process. A running salt-minion daemon and a one-off salt-call
invocation both set opts["__role"] = "minion" and share the same
minion_id, so both processes independently compute the identity
salt-req/minion/<minion_id>/0 for their first REQ socket.

Because the master's ROUTER socket has ROUTER_HANDOVER=1 set (see
salt/transport/zeromq.py, MWorkerQueue), when the second process
(salt-call) connects with the same identity as the first (salt-minion),
libzmq hands the peer slot over to the newcomer and silently drops any
reply still in flight to the original owner. This causes whichever process
was waiting on that reply to stall for the full REQUEST_TIMEOUT (60s)
before retrying.

This directly contradicts the stated goal of the commit itself, which
explicitly calls out that giving two sockets the same identity causes
exactly this failure mode ("Giving them all the same stable identity would
cause ROUTER_HANDOVER on the upstream ROUTER to silently drop any reply
still in flight..."), but the fix only accounted for multiple sockets
within one process, not multiple processes (salt-minion and
salt-call) sharing the same minion_id and hence the same identity
namespace.

Suggested fix

Include something process-unique (e.g. os.getpid()) in the identity
string for the minion/syndic branch, so that salt-call,
salt-minion, and any other short-lived process using the minion config
never collide with the long-running daemon's identity slots.

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

almalinux-9

salt --versions-report output
salt-minion 3006.27 (Sulfur)
Lingua principale
Python
Stelle
15.8k
Fork
5.6k
Merge medio
5g 9h
PR unite (30g)
99

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di saltstack/salt

Tutte le issue di saltstack/salt

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.