[Bug]: Master and minion communication (continue of #69914)

Aperta
  • Qualcosa non va?
#70,392 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
distributed-systems

Direzione di ricerca

Start by comparing Salt 3006.13 with 3006.27 and 3006.28, then trace the reported _return path through salt/master.py, salt/utils/job.py, salt/utils/event.py, and salt/transport/ipc.py. Reproduce the intermittent failure with the supplied Ubuntu 22.04 and Python 3.11 setup. Done means identifying and fixing the cause of the missing minion response and confirming master and minion communication remains reliable.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug needs-triage severity-medium
What happened?

After updating salt-masters from 3006.13 to 3006.28 some of the deployments fail like the master didn't get the response from the minion. The previous and the next deployments may go without any issues at all. And with version 3006.13 there were no issues in communication between the master and that minion.
This can happen even on the test salt master with just a few minions.

Errors in the logs:

2026-10-06 10:57:35,350 [salt.master      :1956][ERROR   ][3159148] Error in function _return:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1950, in run_func
    ret = getattr(self, func)(load)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1743, in _return
    salt.utils.job.store_job(
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 31, in store_job
    return _store_job(opts, load, event, mminion, endtime=endtime)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 91, in _store_job
    event.fire_event(
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/event.py", line 881, in fire_event
    self.pusher.send(msg)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 154, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 170, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
              ^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/ipc.py", line 445, in send
    yield self.stream.write(pack)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 409, in write
    self._add_io_state(self.io_loop.WRITE)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 975, in _add_io_state
    self.io_loop.update_handler(self.fileno(), self._state)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 734, in update_handler
    self._impl.modify(fd, events | self.ERROR)
FileNotFoundError: [Errno 2] No such file or directory

The same errors were in the logs after updating to 3006.27. https://github.com/saltstack/salt/issues/69914

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-22.04

salt --versions-report output
Salt Version:
          Salt: 3006.28
 
Python Version:
        Python: 3.11.16 (main, Sep 30 2026, 03:33:49) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 50.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.62
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.2.1
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.27
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5
 
System Versions:
          dist: ubuntu 22.04.5 jammy
        locale: utf-8
       machine: x86_64
       release: 6.8.0-124-generic
        system: Linux
       version: Ubuntu 22.04.5 jammy
Lingua principale
Python
Stelle
15.8k
Fork
5.6k
Merge medio
5g 9h
PR unite (30g)
99

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di saltstack/salt

Tutte le issue di saltstack/salt

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.