netdata state container: runaway FD use
Offen
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 30/100
- Issue-Typ
- Bug
- Klarheit
- Muss geklärt werden
- Aktivitätsstatus
- Veraltet
- Tech-Stack
- helm, kubernetes
- Bereich
- infrastructure
Rechercherichtung
Beginne damit, das Netdata-state-container-Deployment zu reproduzieren, und untersuche, wie das Helm-Chart seine WAL- und Datenbankpfade konfiguriert. Verwende die lsof-Ausgabe im Issue, um zu überprüfen, ob wiederholt Handles für netdata-meta.db und context-meta.db geöffnet werden; abgeschlossen ist die Aufgabe, wenn der State-Container nicht mehr alle verfügbaren Dateideskriptoren aufbraucht.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Netdata needed to be removed due to consuming ALL available file descriptors (mildly entertaining that this isn't a monitored metric in netdata, that I could find).
From lsof it appears that it is just opening the WAL/db in a loop:
netdata 1781992 1782297 RRDCONTEX 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 25u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 26r FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 27w FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 28u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 29u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 30r FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 31w FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 32u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 33r FIFO 0,13 0t0 62176463 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 34u sock 0,8 0t0 62161843 protocol: UDPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 35w FIFO 0,13 0t0 62166944 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 36u sock 0,8 0t0 62161844 protocol: UDP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 37u sock 0,8 0t0 62161847 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 38u sock 0,8 0t0 62161848 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 39u sock 0,8 0t0 62178506 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 40u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 41u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 42r FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 43w FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 44r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 45r FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 46w FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 47r REG 0,273 0 62169795 /proc/1/task/209/stat
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 48u sock 0,8 0t0 62194554 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 cwd DIR 0,121 4096 13901928 /etc/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 rtd DIR 0,121 4096 14034715 /
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 txt REG 0,121 6986240 13902420 /usr/sbin/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044315 /var/cache/netdata/context-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044312 /var/cache/netdata/netdata-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896738 /usr/lib/libzstd.so.1.5.5 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896436 /usr/lib/libgcc_s.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896627 /usr/lib/libstdc++.so.6.0.30 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896405 /usr/lib/libbson-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896522 /usr/lib/libmongoc-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896615 /usr/lib/libsnappy.so.1.1.10 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896595 /usr/lib/libprotobuf.so.32.0.12 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896491 /usr/lib/libjson-c.so.5.2.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896512 /usr/lib/liblz4.so.1.9.4 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896695 /usr/lib/libuv.so.1.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13895904 /lib/libuuid.so.1.3.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672957 /lib/libssl.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672956 /lib/libcrypto.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672959 /lib/libz.so.1.2.13 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672951 /lib/ld-musl-x86_64.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 0w CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 1w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 2w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 3u sock 0,8 0t0 62173481 protocol: UNIX
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 4w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 5w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 6w REG 0,121 0 14044307 /var/log/netdata/health.log
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 7u sock 0,8 0t0 62173487 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 8u sock 0,8 0t0 62173488 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 9u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 10r FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 11w FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 12r FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 13w FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 14u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
approximately ^ 100's of thousands of times until the node eventually becomes unavailable due to an inability to open any more file descriptors.
- Vorherrschende Sprache
- Go Template
- Sterne
- 177
- Forks
- 82
- Ø Merge
- 1 T. 55 Min.
- Gemergte PRs (30 T.)
- 4
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Keine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus netdata/helmchart
-
Schwierigkeit 5/5 Über eine Woche Anfängerfreundlichkeit 25/100
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 35/100
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 35/100
-
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 42/100
Alle Issues in netdata/helmchart
Ähnliche Issues
-
automated issue report
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
camunda/camunda-deployment-references#3903 ·
Maintainer antworten meist innerhalb von 1 Tag
-
VLAgent `spec.k8sCollector.includePodLabels: false` has no effectEvtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
VictoriaMetrics/operator#2772 ·
Maintainer antworten meist innerhalb von 1 Tag
-
admission-deferred priority: medium status: ready work-class: scoped
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
melodic-software/claude-code-plugins#10887 · 4 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
personalrobotics/geodude#243 ·
Maintainer antworten meist innerhalb von 1 Tag
-
needs: investigation
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
Maintainer antworten meist innerhalb von 1 Tag