netdata state container: runaway FD use
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức phù hợp với người mới
- 30/100
- Loại issue
- Lỗi
- Độ rõ ràng
- Cần làm rõ
- Mức độ hoạt động
- Đình trệ
- Công nghệ
- helm, kubernetes
- Lĩnh vực
- infrastructure
Hướng nghiên cứu
Bắt đầu bằng cách tái hiện việc triển khai state-container của Netdata và kiểm tra cách Helm chart cấu hình các đường dẫn WAL và cơ sở dữ liệu của nó. Sử dụng đầu ra của lsof trong issue để xác minh liệu các handle của netdata-meta.db và context-meta.db có được mở lặp đi lặp lại hay không; hoàn tất có nghĩa là state-container không còn làm cạn kiệt các bộ mô tả tệp khả dụng.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Netdata needed to be removed due to consuming ALL available file descriptors (mildly entertaining that this isn't a monitored metric in netdata, that I could find).
From lsof it appears that it is just opening the WAL/db in a loop:
netdata 1781992 1782297 RRDCONTEX 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 25u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 26r FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 27w FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 28u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 29u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 30r FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 31w FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 32u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 33r FIFO 0,13 0t0 62176463 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 34u sock 0,8 0t0 62161843 protocol: UDPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 35w FIFO 0,13 0t0 62166944 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 36u sock 0,8 0t0 62161844 protocol: UDP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 37u sock 0,8 0t0 62161847 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 38u sock 0,8 0t0 62161848 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 39u sock 0,8 0t0 62178506 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 40u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 41u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 42r FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 43w FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 44r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 45r FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 46w FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 47r REG 0,273 0 62169795 /proc/1/task/209/stat
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 48u sock 0,8 0t0 62194554 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 cwd DIR 0,121 4096 13901928 /etc/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 rtd DIR 0,121 4096 14034715 /
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 txt REG 0,121 6986240 13902420 /usr/sbin/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044315 /var/cache/netdata/context-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044312 /var/cache/netdata/netdata-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896738 /usr/lib/libzstd.so.1.5.5 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896436 /usr/lib/libgcc_s.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896627 /usr/lib/libstdc++.so.6.0.30 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896405 /usr/lib/libbson-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896522 /usr/lib/libmongoc-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896615 /usr/lib/libsnappy.so.1.1.10 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896595 /usr/lib/libprotobuf.so.32.0.12 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896491 /usr/lib/libjson-c.so.5.2.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896512 /usr/lib/liblz4.so.1.9.4 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896695 /usr/lib/libuv.so.1.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13895904 /lib/libuuid.so.1.3.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672957 /lib/libssl.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672956 /lib/libcrypto.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672959 /lib/libz.so.1.2.13 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672951 /lib/ld-musl-x86_64.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 0w CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 1w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 2w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 3u sock 0,8 0t0 62173481 protocol: UNIX
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 4w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 5w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 6w REG 0,121 0 14044307 /var/log/netdata/health.log
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 7u sock 0,8 0t0 62173487 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 8u sock 0,8 0t0 62173488 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 9u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 10r FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 11w FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 12r FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 13w FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 14u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
approximately ^ 100's of thousands of times until the node eventually becomes unavailable due to an inability to open any more file descriptors.
- Ngôn ngữ chính
- Go Template
- Star
- 177
- Fork
- 81
- Merge trung bình
- 25 phút
- Pull request đã merge (30 ngày)
- 2
Chuẩn bị môi trường
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của netdata/helmchart
-
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 25/100
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 35/100
-
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 35/100
-
Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 42/100
Tất cả issue của netdata/helmchart
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 76/100
rossoctl/context-guru#366 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 88/100
aws-samples/sample-pacer#76 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
UX
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 65/100
ProfessionalWiki/NeoWiki#1570 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100