Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Tree builds endpoint exhausts memory and took production down

Abierto
  • ¿Algo no está bien?
#2,202 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@felipebergamin ya está trabajando en esto.

Desde el 28/10/2026.

  • #2206 de @felipebergamin — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
azure, django, nginx, postgresql, python
Área
api, backend, databases, infrastructure, performance

Línea de trabajo

Start with backend/kernelCI_app/queries/tree.py and treeDetailsBuildsView._sanitize_rows; trace how the builds endpoint fetches and filters rows, then compare that with the response fields it uses. The issue identifies filtering in SQL, selecting fewer columns, and bounding or paginating results as the proper fix; done means large tree requests no longer exhaust backend memory and the endpoint can be re-enabled.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug URGENT

Summary

GET /api/tree/<tree>/<branch>/<commit_hash>/builds exhausts backend memory on large trees and took production down on 2026-10-27 (dashboard.kernelci.org and staging, ~21:50–22:17 UTC). The VM had to be hard-restarted.

Why this is an emergency

  • Production and staging share one 8 GB VM. One heavy request on this endpoint can freeze the whole host: web, API and SSH all become unreachable.
  • It is not load-related. Normal browsing of a linux-next commit page is enough, so it will happen again.
  • It has been happening for days: serial console shows repeated OOM kills of gunicorn workers at 1–2 GB RSS each, the latest ~5 days before this outage.

Evidence

  • Azure metrics: available memory 4.6 GB → 1.3 GB at 18:26 UTC (never freed). At 21:48, ~360 MB in from DB and CPU 90%. By 21:51, 70 MB free, disk reads pinned at 132 MB/s (thrashing). CPU credits were full, so this is not throttling.
  • The last API request completed at 21:48:21, after which the host was unresponsive.
  • nginx, 2026-10-27: 49 client aborts (499) on this endpoint: next/master ×28, android14-6.1-lts ×12, stable-rc/linux-5.10.y ×8. The 18:26 memory drop coincides with a 499 on /api/tree/next/master/<sha>/builds.

Root cause

get_tree_data(data_type="builds") (backend/kernelCI_app/queries/tree.py):

  • joins builds × incidents × issues for the checkout;
  • selects large text columns (misc, comment, command, log_url);
  • fetchall()s everything, then applies filter_* params in Python (treeDetailsBuildsView._sanitize_rows).

On top of that, gunicorn has 5 workers per backend × 2 backends, no --max-requests, no container memory limit and a 250 s timeout. Workers keep working after the client disconnects and never release memory.

Fix

  1. Immediate workaround: disable the endpoint (503), recycle workers, cap backend memory. Must be merged and deployed within 24 h.
  2. Proper fix: push filters into SQL, select only the columns the response needs, bound or paginate the result, then re-enable the endpoint.
  3. Follow-up: move staging off the production host (as was discussed on meetings), or we might make it on-demand
Lenguaje dominante
Python
Estrellas
10
Forks
32
Merge medio
4 d 7 min
PR fusionados (30 d)
31

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de kernelci/dashboard

Todos los issues de kernelci/dashboard

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.