[Bug]: await_lock masks exceptions as FileLockError; minion job_queue requeues forever ("Failed to acquire job_queue lock" log flood)

Abierto
  • ¿Algo no está bien?
#70,229 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
devops, distributed-systems

Línea de trabajo

Comienza con salt.utils.files.await_lock y sigue su manejo de excepciones alrededor del cuerpo de with; después, inspecciona _handle_decoded_payload y _process_process_queue_async_impl. Confirma que las excepciones del cuerpo sigan siendo distinguibles de los fallos al adquirir el lock, que el registro de FileLockError incluya la excepción y que los jobs encolados que fallan repetidamente dejen finalmente de volver a encolarse y devuelvan un error al master.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happened?

On 3007.13 two minions got stuck in an infinite requeue loop, logging ~180 lines/s of

[salt.minion :1960][WARNING ] Failed to acquire job_queue lock for jid 2026..., queuing anyway.

for a week, until /var/log/salt/minion reached 9 GB and filled /var. 59 stale jids cycled every 0.3 s. No job_queue.lock file existed, fds were 45/8192, plenty of RAM. The loop started 9 s after a Timeout encountered while sending ... _return request.

Root cause, two defects that combine:

  1. salt.utils.files.await_lock hides the real exception. The yield is inside the outer try, whose except Exception re-raises everything as FileLockError("Error encountered obtaining file lock ..."). Any exception raised by the async with await_lock(...) body is reported as a lock failure and its traceback is lost. In 3007.13 the body included _invoke_execution() and the proc-file write, so a job-start failure became "Failed to acquire job_queue lock". 3007.14 moved _invoke_execution out of the lock, but await_lock itself is unchanged in 3008.x, and the minion's except FileLockError handler still logs without the exception.

  2. The disk job queue has no exit. _process_process_queue_async_impl resubmits every queued_*.p file every 0.3 s (0.2 s in 3008.x) with no retry counter, max age or backoff. A job that fails deterministically at start is requeued forever.

Workaround used: stop minion, move cachedir/job_queue aside, truncate log, start.

Expected
  • await_lock should not convert exceptions from the with body into FileLockError (re-raise them unchanged, keep the wrapping for the lock-acquire path only).
  • _handle_decoded_payload's except FileLockError should log the exception (exc_info=True).
  • Queued jobs need a retry limit and/or max age; after that the job is dropped with an error and a return to the master.
Type of salt install

onedir

Major version

3007 (3007.13); defect 1 and 2 still present in 3008.2 / master

What supported OS are you seeing the problem on?

Ubuntu

salt --versions-report output
Salt Version:
          Salt: 3007.13
Python: /opt/saltstack/salt/bin/python3.10 (onedir)
OS: Ubuntu, cPanel hosts
Lenguaje dominante
Python
Estrellas
15.8k
Forks
5.6k
Merge medio
5 d 9 h
PR fusionados (30 d)
99

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de saltstack/salt

Todos los issues de saltstack/salt

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.