[Bug]: DeltaProxy startup waits on an unaccepted child, blocking controller and preaccepted sibling

Aperta
  • Qualcosa non va?
#70,709 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, distributed-systems

Direzione di ricerca

Start with salt/metaproxy/deltaproxy.py, especially post_master_init and the sequential and parallel child initialization paths described in the report. Use the linked gist's reproduce.py and result.json to confirm the startup behavior with the built-in dummy proxy. Done when the controller and preaccepted sibling respond while another child awaits acceptance, and the pending child can join after acceptance; test both child orders and parallel startup modes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happened?

A DeltaProxy controller with an accepted key does not become responsive while its first child is waiting for Salt key acceptance. A second child's key is already accepted before startup, but that child also cannot finish initialization or answer jobs until the first child's key is accepted.

This reproduces with Salt's built-in dummy proxy, without appliance connections or Cisco extension APIs.

Expected: the controller and independently authenticated children should become responsive while another child awaits acceptance. The unaccepted child must remain unauthenticated; no bypass of key acceptance is requested. Its initialization should be bounded/retried independently.

Actual verified result:

Check First child pending After accepting first child
Controller ready in log No Yes
Preaccepted sibling finished tune-in No Yes
Controller test.ping No return true
Preaccepted sibling test.ping No return No successful return

The child job-return failure after tune-in is an additional observation, not assumed to share the same cause. This report focuses on startup/readiness. It does not establish that a device becoming unreachable after successful startup stops the controller.

Reproduction script and recorded results

https://gist.github.com/ITJamie/f48af048f8b84d38ba80508f2c277fea

The gist includes reproduce.py, instructions and the verified result.json. Run using a complete Salt 3008.4 Python environment with salt-master, salt-proxy, salt and salt-key alongside its Python executable:

/path/to/salt-environment/bin/python reproduce.py --artifacts ./results

The script starts real master/proxy daemons on loopback with isolated temporary configuration, keypairs and caches; it does not use an existing master. It stops its daemon process groups and discards temporary keypairs afterward.

  1. Preaccept the controller and healthy sibling keys. Leave the first child unaccepted.
  2. Start the controller and wait for the first child's key to appear in minions_pre.
  3. Check controller and healthy sibling readiness and bounded test.ping results.
  4. Accept only the first child's key and record readiness/response changes.

The master uses ordinary file pillar with these mappings:

# top.sls
base:
  repro-controller:
    - controller
  repro-unaccepted:
    - dummy
  repro-healthy:
    - dummy

# controller.sls
proxy:
  proxytype: deltaproxy
  parallel_startup: false
  ids:
    - repro-unaccepted
    - repro-healthy

# dummy.sls
proxy:
  proxytype: dummy

The proxy configuration uses metaproxy: deltaproxy, transport: tcp, and multiprocessing: false. The script generates the complete configurations and chooses free loopback ports. It also supports --healthy-first and --parallel-startup as comparison cases; those modes have not yet been verified and are not claimed as fixes.

During the wait the proxy repeatedly logs:

The Salt Master has cached the public key for this node, this salt minion will wait for 1 seconds before attempting to re-authenticate

After acceptance it progresses through both child tune-in steps to:

Tune in for sub proxy 'repro-unaccepted' finished
Tune in for sub proxy 'repro-healthy' finished
Minion is ready to receive requests!
Suspected mechanism

Inference from salt/metaproxy/deltaproxy.py: sequential post_master_init awaits each child's subproxy_post_master_init before setting the controller ready. A child's pillar/module synchronization can wait for key acceptance. Catching initialization exceptions does not bound an operation that keeps retrying. The parallel branch also awaits a collective asyncio.gather, so it should be tested rather than assumed to provide isolation.

Related history: #61805 concerns startup performance and is closed; I am not asserting that it covers this authentication/readiness failure.

Suggested regression: an accepted controller and preaccepted sibling must answer jobs while another child's key stays pending; repeat with the pending child first/last and parallel startup on/off, then verify the child can join after acceptance.

The Linux reproduction used an existing disposable lab container. The ASA extension was installed in that image (as shown below), but the reproducer invokes only built-in dummy proxies and Salt CLI commands. No ASA hardware or extension API is involved.

Type of salt install

pip (pypi)

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output
Salt Version:
                  Salt: 3008.4
 
Python Version:
                Python: 3.11.18 (main, Nov  5 2026, 08:12:05) [GCC 12.2.0]
 
Dependency Versions:
                  cffi: 2.1.1
              cherrypy: 18.10.0
          cryptography: 50.0.2
              dateutil: 4.0.0
             docker-py: Not Installed
                 gitdb: 4.0.12
             gitpython: Not Installed
                Jinja2: 3.1.6
               libgit2: Not Installed
          looseversion: 1.3.0
              M2Crypto: Not Installed
                  Mako: Not Installed
               msgpack: 1.2.4
          msgpack-pure: Not Installed
          mysql-python: Not Installed
             packaging: 26.3
             pycparser: 3.12
              pycrypto: Not Installed
          pycryptodome: 3.24.2
                pygit2: Not Installed
          python-gnupg: 0.5.7
                PyYAML: 6.0.3
                 PyZMQ: 27.3.0
                relenv: Not Installed
                 smmap: 5.0.3
               timelib: 0.3.0
               Tornado: 6.6
                   ZMQ: 4.3.5
 
Salt Extensions:
 pytest-salt-factories: 1.0.5
      saltext.ciscoasa: 0.0.0
 
Salt Package Information:
          Package Type: Not Installed
 
System Versions:
                  dist: debian 12.15 bookworm
                locale: utf-8
               machine: aarch64
               release: 7.0.12-linuxkit
                system: Linux
               version: Debian GNU/Linux 12.15 bookworm
Lingua principale
Python
Stelle
15.8k
Fork
5.6k
Merge medio
5g 9h
PR unite (30g)
99

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di saltstack/salt

Tutte le issue di saltstack/salt

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.