Use /livez/ for the API liveness probe and a cheaper readiness probe
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- Medio día
- Aptitud para principiantes
- 62/100
Línea de trabajo
The default probes are defined in controllers/deployment.go: the liveness httpGet and the readiness exec of /usr/bin/readyz.py, both pointing at api/v3/status/. Start there and change the liveness path to api/v3/livez/ with the existing timings, then decide the readiness timings or endpoint with a maintainer. Done means the defaults match the issue and the Pulp CR overrides still apply.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Is your feature request related to a problem? Please describe.
The default API probes call /status/. This is the same in 2.0.0 and on main, in controllers/deployment.go:
- Liveness:
httpGetto<API_ROOT>api/v3/status/, every 20s, timeout 10s,failureThreshold: 10. - Readiness:
exec/usr/bin/readyz.py <API_ROOT>api/v3/status/, every 10s, timeout 10s,failureThreshold: 1.
/status/ is not a light check. It runs some database queries, and with object storage it runs SUM(size) over all artifacts (pulp/pulpcore#8196). Because the probes call it on every API pod all the time, this causes two problems:
- Database load. The load grows with the number of API replicas and the number of artifacts.
- Restarts that do not help. When the database is slow,
/status/takes more than 10s, and the liveness probe restarts API pods that are healthy. A restart does not make the database faster, and requests in progress fail.
Describe the solution you'd like
- Liveness: use
<API_ROOT>api/v3/livez/. Since pulp/pulpcore#7721,/livez/does not use the database, so it checks only that the API process answers. This is the correct check for a liveness probe. - Readiness: keep the database and Redis check, but make it cheaper. For example, call
/status/less often, with a longer timeout andfailureThresholdabove 1. Or add a light endpoint in pulpcore that checks only the database and Redis connections.
Describe alternatives you've considered
We override the probes in the Pulp CR (spec.api.livenessProbe, spec.api.readinessProbe):
- Liveness:
httpGet<API_ROOT>api/v3/livez/, with the default timings. - Readiness: the default
readyz.pycommand, every 60s, timeout 30s,failureThreshold: 3.
This works for us, but every user of the defaults has the same problem.
Additional context
/status/cost in pulpcore: pulp/pulpcore#8196- Similar aggregate in the telemetry task: pulp/pulpcore#7475
/livez/without the database: pulp/pulpcore#7721
- Lenguaje dominante
- Go
- Estrellas
- 88
- Forks
- 69
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 1
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de pulp/pulp-operator
-
settings.py flaps between reconciles: needsMigrationSetting depends on map iteration orderPosiblemente ocupada @fontaineajulien la tomó hace 25 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
pulp/pulp-operator#1691 ·
-
Documentation for metadata signing (APT/RPM) removed but still relevantQuizá libre de nuevo Un pull request para esta issue se cerró sin fusionarse. AbiertoIssue Triage-Needed
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
pulp/pulp-operator#1644 ·
-
Feature Triage-Needed
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
pulp/pulp-operator#1697 · 2 reacciones ·
-
Issue Triage-Needed
Dificultad 3/5 1-2 días Aptitud para principiantes 74/100
pulp/pulp-operator#1685 ·
-
Gateway API compatibilityAbiertoFeature Triage-Needed
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
pulp/pulp-operator#1682 ·
Todos los issues de pulp/pulp-operator
Issues similares
-
area:recipes event:contribfest format:human-first source:agent status:needs-triage type:recipe
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
cadence-workflow/cadence-ai-samples#140 ·
Los mantenedores suelen responder en 1 día
-
ux
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
evcc-io/evcc#35605 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
project-setup
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Los mantenedores suelen responder en 1 día
-
bug report
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Los mantenedores suelen responder en 1 día
-
type:bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
nightgauge/nightgauge#2732 ·
Los mantenedores suelen responder en 1 día