[BUG] - Cardano-node 11.1.3 crashes due to increasing memory consumption
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 44/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- docker, docker-compose, grafana, haskell
- Área
- backend, performance
Línea de trabajo
Start by comparing cardano-node 11.1.3 with 11.1.2 under the reported Docker setup, using docker stats to track memory and the supplied RTS options. The issue does not identify a source file or failing test, so tracing the memory growth will require familiarity with the node. Done means reproducing and locating the regression and confirming memory stays within the reported 16–24 GB range.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
External
Area
Other cardano-node 11.1.3
Summary
cardano-node 11.1.3 consumes over 28 Gb memory within 6 hours , exhausts resources and crashes (restarts my container). This is on a bare metal machine with 64 Gb memory running two nodes. On the same machine running one node this problem doesn't occur. But it should never consume more than 28 Gb memory in the first place and the memory consumption crawls up over time.
Running this in my own docker environment on basis of Debian-Slim. Switching back to cardano-node 11.1.2 shows that the node stays within the 24Gb memory with the same docker environment, just change of the binary.
So there is definitely a problem with Cardano-node 11.1.3 binary.
Steps to reproduce
I think it should be easy to reproduce on your end by monitoring the memory or by installing two nodes on a machine with 64 Gb and restricting the container with the following settings in .env NODE_MEM_LIMIT=32g
NODE_MEM_RESERVATION=28g and under services relay in docker-compose.yaml
services:
relay:
mem_limit: ${NODE_MEM_LIMIT:-0}
mem_reservation: ${NODE_MEM_RESERVATION:-0}
I increased the values of NODE-MEM-LIMIT and NODE_MEM_RESERVATION because of this bug, but this didn't help still crashing the node after exhausting the memory.
cardano-node started with the following rts_opts="${RTS_OPTS:--N4 -xn -T -A128m -n1m -I0.3 -Iw3600 -F1.5 -H8000m}"
I switched back to cardano-node 11.1.2 with the exact same setup and settings and it stays within the boundaries of 24 Gb memory consumption, the memory consumption doesn't crawl up like with cardano-node 11.1.3
Checked memory consumption with: docker stats --no-stream
Monitoring the memory consumption of my nodes with Grafana and get an alert when the node exceeds 24 Gb or crashes.
Expected behavior
The cardano-node stays within the range of 16 - 24 Gb memory consumption, one shouldn't be forced to run with the disk option when they have memory enough to run it comfortably in memory.
Screenshots and attachments
Can provide if required, but I think on the above you should have enough to reproduce the behavior.
Additional context
It seems the node was not tested well and rushed out to fix an issue, or assuming everyone runs on local storage instead of memory.
- Lenguaje dominante
- Haskell
- Estrellas
- 3.2k
- Forks
- 762
- Merge medio
- 2 d 9 h
- PR fusionados (30 d)
- 19
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de IntersectMBO/cardano-node
-
Stale
Dificultad 1/5 1-3 horas Aptitud para principiantes 75/100
IntersectMBO/cardano-node#6597 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Stale
Dificultad 1/5 1-3 horas Aptitud para principiantes 72/100
IntersectMBO/cardano-node#6578 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
IntersectMBO/cardano-node#6522 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
-
needs triage type: enhancement
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
IntersectMBO/cardano-node#6737 ·
Los mantenedores suelen responder en 1 día
-
increase in memory usageAbierto
Dificultad 4/5 3-5 días Aptitud para principiantes 28/100
IntersectMBO/cardano-node#6718 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
Todos los issues de IntersectMBO/cardano-node
Issues similares
-
documentation
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
alunduil/collection-json.hs#421 ·
Los mantenedores suelen responder en 1 día
-
A `where` step with a misspelled key, such as `arg:` for `args:`, is accepted, so `join` silently answers no bindings (src/Yaml.hs:190)Posiblemente ocupada @sherenkov la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
objectionary/phino#1994 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
alunduil/siren-json.hs#259 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
-
enhancement triage
Dificultad 1/5 Menos de una hora Aptitud para principiantes 77/100
simplex-chat/simplex-chat#7781 ·
Los mantenedores suelen responder en 1 día