netdata state container: runaway FD use
オープン
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 30/100
- issue の種類
- バグ
- 明瞭さ
- 説明が足りない
- 活発さ
- 停滞
- 技術スタック
- helm, kubernetes
調査の方向性
まず Netdata の state-container のデプロイを再現し、Helm chart が WAL とデータベースのパスをどのように設定しているかを調べます。issue の lsof 出力を使って、netdata-meta.db と context-meta.db のハンドルが繰り返し開かれているかを確認します。state-container が利用可能なファイルディスクリプタを使い果たさなくなれば完了です。
索引モデルが issue の本文から書いたものです。
説明
Netdata needed to be removed due to consuming ALL available file descriptors (mildly entertaining that this isn't a monitored metric in netdata, that I could find).
From lsof it appears that it is just opening the WAL/db in a loop:
netdata 1781992 1782297 RRDCONTEX 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 25u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 26r FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 27w FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 28u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 29u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 30r FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 31w FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 32u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 33r FIFO 0,13 0t0 62176463 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 34u sock 0,8 0t0 62161843 protocol: UDPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 35w FIFO 0,13 0t0 62166944 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 36u sock 0,8 0t0 62161844 protocol: UDP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 37u sock 0,8 0t0 62161847 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 38u sock 0,8 0t0 62161848 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 39u sock 0,8 0t0 62178506 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 40u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 41u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 42r FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 43w FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 44r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 45r FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 46w FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 47r REG 0,273 0 62169795 /proc/1/task/209/stat
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 48u sock 0,8 0t0 62194554 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 cwd DIR 0,121 4096 13901928 /etc/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 rtd DIR 0,121 4096 14034715 /
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 txt REG 0,121 6986240 13902420 /usr/sbin/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044315 /var/cache/netdata/context-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044312 /var/cache/netdata/netdata-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896738 /usr/lib/libzstd.so.1.5.5 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896436 /usr/lib/libgcc_s.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896627 /usr/lib/libstdc++.so.6.0.30 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896405 /usr/lib/libbson-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896522 /usr/lib/libmongoc-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896615 /usr/lib/libsnappy.so.1.1.10 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896595 /usr/lib/libprotobuf.so.32.0.12 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896491 /usr/lib/libjson-c.so.5.2.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896512 /usr/lib/liblz4.so.1.9.4 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896695 /usr/lib/libuv.so.1.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13895904 /lib/libuuid.so.1.3.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672957 /lib/libssl.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672956 /lib/libcrypto.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672959 /lib/libz.so.1.2.13 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672951 /lib/ld-musl-x86_64.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 0w CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 1w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 2w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 3u sock 0,8 0t0 62173481 protocol: UNIX
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 4w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 5w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 6w REG 0,121 0 14044307 /var/log/netdata/health.log
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 7u sock 0,8 0t0 62173487 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 8u sock 0,8 0t0 62173488 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 9u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 10r FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 11w FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 12r FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 13w FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 14u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
approximately ^ 100's of thousands of times until the node eventually becomes unavailable due to an inability to open any more file descriptors.
- 主要言語
- Go Template
- スター
- 177
- フォーク
- 81
- 平均マージ
- 1時間 50分
- マージ済み PR（30日）
- 2
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
netdata/helmchart のほかの issue
-
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 35/100
-
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
-
難易度 3/5 1〜2日 初心者へのやさしさ 42/100
netdata/helmchart の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
メンテナーはふだん 1 日以内に返信
-
area:provisioning effort:S priority:P2 provider:local-libvirt risk:daytime-only status:in-progress type:bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
randomparity/kdive#3203 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
MetricsHub/metricshub-community#1396 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
alphagov/notifications-api#5004 ·
メンテナーはふだん 2 日以内に返信