Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Bug: `Reaper.delete_instance()` raises 409 and keeps the dead reaper when ryuk was killed from outside

Abierto
  • ¿Algo no está bien?
#1,125 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
76/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
docker, python
Área
devops, testing-qa

Línea de trabajo

Start in testcontainers.core.container at Reaper.delete_instance() and Reaper.get_instance(), then run the provided reproduction after killing the ryuk container externally. Verify that removal-in-progress is handled like an already removed container, class state is reset after cleanup, and a subsequent get_instance() can create a replacement without stale state or an atexit exception.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

When ryuk has been killed from outside (docker kill, an OOM kill, a Docker Desktop restart), Reaper.delete_instance() raises APIError 409 and leaves Reaper holding the dead reaper. After that, Reaper.get_instance() returns the dead instance instead of creating a new one.

The reaper container is created with auto_remove=True, so Docker starts removing it as soon as it dies. delete_instance() calls Reaper._container.stop(), whose remove(force=True) arrives while that removal is still running, and Docker answers:

409 Client Error: Conflict ... removal of container <id> is already in progress

delete_instance() suppresses only docker.errors.NotFound, so the 409 propagates. The socket has already been closed and Reaper._socket set to None, but the exception skips the two resets after it: Reaper._container and Reaper._instance still point at the dead reaper. get_instance() only creates a reaper when _instance is None, so it keeps returning the dead one, and the session has no reaper and no way back to one.

How we got there: a pytest session whose reaper was removed mid-run, with the session's containers still running. After ryuk's ACK the reaper link carries no data, so a lost reaper shows up on Reaper._socket as end-of-file within milliseconds. We check for that before each test and replace the reaper with delete_instance() + get_instance(). The replacement is where this surfaced: 30 of 40 SIGKILLs in one harness, and 10 of 10 in the script below. delete_instance() also runs from atexit, and it hits the same 409 there when the process exits during the removal. The script below does, and ends with Exception ignored in atexit callback.

To Reproduce

"""Reaper.delete_instance() after ryuk was killed from outside: 409, and stale state."""

import selectors
import uuid

import docker
import docker.errors
import testcontainers.core.container as tcc
from testcontainers.core.config import testcontainers_config as c
from testcontainers.core.container import DockerContainer, Reaper

# Publish ryuk's 8080 on 127.0.0.1 only (host port still kernel-assigned), just to keep the
# experiment's Docker-socket-holding ryuk off the LAN. Not part of the issue.
_orig_with_exposed_ports = DockerContainer.with_exposed_ports


def _loopback(self, *ports):
    out = _orig_with_exposed_ports(self, *ports)
    if self.image == c.ryuk_image:
        for p in ports:
            self.ports[str(p)] = ("127.0.0.1", None)
    return out


DockerContainer.with_exposed_ports = _loopback

tcc.SESSION_ID = f"repro-{uuid.uuid4()}"
reaper = Reaper.get_instance()
ryuk_id = Reaper._container.get_wrapped_container().id

client = docker.from_env()
client.api.kill(ryuk_id)  # what `docker kill` or an OOM kill does from outside
with selectors.DefaultSelector() as sel:  # the kill shows on the reaper link as end-of-file
    sel.register(Reaper._socket, selectors.EVENT_READ)
    sel.select(5)

try:
    Reaper.delete_instance()
except docker.errors.APIError as exc:
    print("delete_instance() raised", exc.status_code, exc.explanation)
print("_socket:", Reaper._socket)
print("_instance is still the dead reaper:", Reaper._instance is reaper)
print("get_instance() returns the dead reaper:", Reaper.get_instance() is reaper)

Output (the first four lines were the same in 10 of 10 runs of a loop version):

delete_instance() raised 409 removal of container 4dfcbdd146b5512c26004bbf887100f12d8cb0e3e67dad883e902485659fa24a is already in progress
_socket: None
_instance is still the dead reaper: True
get_instance() returns the dead reaper: True
Exception ignored in atexit callback <bound method Reaper.delete_instance of <class 'testcontainers.core.container.Reaper'>>:
Traceback (most recent call last):
  ...
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.55/containers/<id>?v=True&link=False&force=True: Conflict ("removal of container <id> is already in progress")

Suggested fix

Treat "removal already in progress" like "already gone", and reset the class state whatever stop() does. For example:

    @classmethod
    def delete_instance(cls) -> None:
        container = Reaper._container
        try:
            if Reaper._socket is not None:
                Reaper._socket.close()
            if container is not None and container._container is not None:
                try:
                    container.stop()
                except docker.errors.NotFound:
                    pass
                except docker.errors.APIError as exc:
                    if exc.status_code != 409:  # 409: auto_remove is already removing it
                        raise
        finally:
            Reaper._socket = None
            Reaper._container = None
            Reaper._instance = None

One more thing for anyone recreating a reaper right after this: the new container reuses the name testcontainers-ryuk-{SESSION_ID}, which is only free once the old container is fully removed. We measured the old container still listed up to 0.29 s after the kill, and still listed in 4 of 20 checks made with no wait at all. So we wait for its id to disappear before calling get_instance(). We saw no name conflict with that wait in place (80+ replacements), and did not test without it. #1093 covers a related name collision from another path.

Runtime environment

  • testcontainers-python 4.15.0 (Reaper is unchanged on main), ryuk 0.8.1 (the default), docker-py 7.2.0, Python 3.13.13
  • Docker Desktop 4.88.1, Engine 29.7.2 (API 1.55), linuxkit 7.0.12, linux/arm64, 12 CPUs
  • macOS 27.0 (26A428), Darwin 27.0.0 arm64
Lenguaje dominante
Python
Estrellas
2.3k
Forks
388
Merge medio
4 h 40 min
PR fusionados (30 d)
1

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de testcontainers/testcontainers-python

Todos los issues de testcontainers/testcontainers-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.