[Bug]: salt-call causes ~60s hangs on minions due to ZMQ identity collision with running salt-minion daemon (3006.27 regression)

Abierto
  • ¿Algo no está bien?
#70,127 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

  • #70140 de @waterWang — cerrado sin fusionar
  • #70196 de @jzandbergen — cerrado sin fusionar

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
58/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
distributed-systems, networking

Línea de trabajo

Empieza en salt/transport/zeromq.py y sigue después AsyncReqMessageClient y la rama de identidad de minion/syndic introducida por el commit 36e914a72af330e8e0c09bfea5f43893f5e972b1. Reproduce salt-call mientras se está ejecutando un daemon de salt-minion y verifica después que sus identidades ZMQ ya no colisionan y que las solicitudes de pillar no esperan durante todo REQUEST_TIMEOUT.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug needs-triage
What happened?
What happened?

Pillar (and any other) requests made via salt-call on a minion take ~60 seconds
(the full REQUEST_TIMEOUT) to complete on 3006.27, when the same command
completes in a few seconds on 3006.26. The affected code path is unchanged
in 3006.26, so this is a regression introduced in 3006.27.

[06:31:40] alma98-p3g ~ # time salt-call pillar.data dontcarejustatest
local:
    ----------
    dontcarejustatest:

real    1m3.404s
user    0m1.045s
sys    0m0.137s

Trace logging on the minion shows the request hangs for ~60s waiting on a
reply from the master before finally timing out and retrying successfully:

[DEBUG   ] Initializing new AsyncAuth for (...)
[TRACE   ] ext_pillar_extra_data = {}
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key

<-- ~60s stall -->

[TRACE   ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE   ] Request timed out while waiting for a response. reconnecting.
Root cause

3006.27 includes commit 36e914a72af330e8e0c09bfea5f43893f5e972b1
("Use stable ZMQ identity for minion + syndic daemon ret-port REQ
sockets"), which sets a stable ZMQ IDENTITY on the AsyncReqMessageClient
REQ socket for long-lived minion/syndic daemons:

elif _role in ("minion", "syndic") and _minion_id:
    identity = "salt-req/{role}/{minion_id}/{slot}".format(
        role=_role,
        minion_id=_minion_id,
        slot=next(_REQ_IDENTITY_SLOT),
    )
    self.socket.setsockopt(zmq.IDENTITY, identity.encode("utf-8"))

_REQ_IDENTITY_SLOT is a module-level itertools.count() that starts at 0
per process. A running salt-minion daemon and a one-off salt-call
invocation both set opts["__role"] = "minion" and share the same
minion_id, so both processes independently compute the identity
salt-req/minion/<minion_id>/0 for their first REQ socket.

Because the master's ROUTER socket has ROUTER_HANDOVER=1 set (see
salt/transport/zeromq.py, MWorkerQueue), when the second process
(salt-call) connects with the same identity as the first (salt-minion),
libzmq hands the peer slot over to the newcomer and silently drops any
reply still in flight to the original owner. This causes whichever process
was waiting on that reply to stall for the full REQUEST_TIMEOUT (60s)
before retrying.

This directly contradicts the stated goal of the commit itself, which
explicitly calls out that giving two sockets the same identity causes
exactly this failure mode ("Giving them all the same stable identity would
cause ROUTER_HANDOVER on the upstream ROUTER to silently drop any reply
still in flight..."), but the fix only accounted for multiple sockets
within one process, not multiple processes (salt-minion and
salt-call) sharing the same minion_id and hence the same identity
namespace.

Suggested fix

Include something process-unique (e.g. os.getpid()) in the identity
string for the minion/syndic branch, so that salt-call,
salt-minion, and any other short-lived process using the minion config
never collide with the long-running daemon's identity slots.

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

almalinux-9

salt --versions-report output
salt-minion 3006.27 (Sulfur)
Lenguaje dominante
Python
Estrellas
15.8k
Forks
5.6k
Merge medio
5 d 9 h
PR fusionados (30 d)
99

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de saltstack/salt

Todos los issues de saltstack/salt

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.