NetworkTracker leaks a connectTrackerTimes node per outbound TCP connection, and PurgeStale never runs on busy hosts

オープン
  • 何か問題がありますか？
#247 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
c
領域
networking

調査の方向性

Start by reading networkTracker.cpp, especially NetworkTracker::PurgeStale and NetworkTracker::SeenConnect; the issue identifies both faulty paths and includes a proposed fix. Build the project and run a reproduction with a NetworkConnect rule and repeated loopback TCP connections. Done when the tracker removes established connections using the stored timestamp and stale entries are swept on busy hosts; the issue says the proposed patch has not been built or tested.

索引モデルが issue の本文から書いたものです。

説明

Version: sysmonforlinux 1.5.3 (networkTracker.cpp is unchanged on main at b2f11cc07a). Seen on Ubuntu 22.04 (kernel 5.15) and Ubuntu with kernel 7.0.

Symptom

The sysmon heap grows linearly with the host's rate of outbound TCP connections and is only freed by a restart. On a 4 GB nginx reverse proxy making about 11 outbound connections per second, the sysmon heap (VmData) reached 2.6 GB after 36 days (about 70 MB/day), leaving the host with 60 MB available. Hosts with few outbound connections grow only a few MB per day, which hides the bug.

Any configuration with a NetworkConnect rule triggers it, because processNetworkEvent feeds the tracker before rule matching. An include rule that matches only wget/curl, plus a 127. exclude, still leaks on every connection.

Reproduction

With any NetworkConnect include rule loaded, open 20,000 loopback TCP connections (connect, then close). The sysmon RSS grows by about 2 MB (roughly 100 bytes per connection) and never shrinks.

Evidence

A sample of the long-running process's heap was almost entirely 64-byte malloc chunks, each a red-black tree node whose value is (LONGLONG time in 100 ns ticks, kernel socket pointer). That is the node layout of std::map<LONGLONG, const void *>, i.e. connectTrackerTimes. The heap held about 42 million of them, with timestamps three weeks old, although staleDuration is 15 minutes.

Cause

Two bugs in networkTracker.cpp:

  1. Wrong erase key on TCP_ESTABLISHED. In SeenConnect, the branch for an already-tracked socket reaching TCP_ESTABLISHED calls connectTrackerTimes.erase(event->m_EventTime.QuadPart). That is the time of the ESTABLISHED event, but the entry was inserted under the time of the earlier SYN_SENT event, which is stored in ret.first->second.second. The erase misses, so every successful outbound connection orphans one connectTrackerTimes node. The TCP_CLOSE branch already erases by the stored time correctly.

  2. PurgeStale never runs on a busy host. PurgeStale refreshes timeLastChecked at the end of every call, not only after a sweep. It is called on every connect and accept, so curTime - timeLastChecked > checkDuration (60 s) holds only after a full minute with no TCP connect or accept events at all. On a server that never happens, so the 15-minute stale sweep that would collect the orphans never runs. This also stops acceptTrackerTimes from ever being swept.

Proposed fix

--- a/networkTracker.cpp
+++ b/networkTracker.cpp
@@ void NetworkTracker::PurgeStale()
                 break; // as this is an ordered map, we have exhausted all stale items
             }
         }
-    }
-
-    GetSystemTimeAsLargeInteger(&timeLastChecked);
+        timeLastChecked = curTime;
+    }
 }
@@ pid_t NetworkTracker::SeenConnect(PSYSMON_EVENT_HEADER eventHdr)
             if (event->m_NewState == TCP_ESTABLISHED) {
                 // connection is established, so remove it
-                connectTracker.erase(event->m_SockId);
-                connectTrackerTimes.erase(event->m_EventTime.QuadPart);
+                connectTrackerTimes.erase(ret.first->second.second);
+                connectTracker.erase(ret.first);
             }

The first hunk lets the periodic sweep run as intended, which also acts as a backstop for any other path that leaves entries behind. The second removes the orphan at its source. The patch has not been built or tested yet.

Workaround

Cap the service with a systemd drop-in, for example MemoryMax=512M with the unit's existing Restart=on-failure, plus StartLimitIntervalSec=0. The restart limit matters because a restarted sysmon can fail with libbpf: failed to mmap perf buffer: -ENOMEM on a fragmented host and would otherwise give up after five attempts.

Possibly related: #211 (memory growth, though its Valgrind output points at ProcessCache).

主要言語
C
スター
2.2k
フォーク
222
平均マージ
4時間 50分
マージ済み PR（30日）
3

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/Sysmon-Linux のほかの issue

microsoft/Sysmon-Linux の issue をすべて見る

似ている issue

C の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。