NetworkTracker leaks a connectTrackerTimes node per outbound TCP connection, and PurgeStale never runs on busy hosts

Offen
  • Stimmt etwas nicht?
#247 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
55/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
c
Bereich
networking

Rechercherichtung

Start by reading networkTracker.cpp, especially NetworkTracker::PurgeStale and NetworkTracker::SeenConnect; the issue identifies both faulty paths and includes a proposed fix. Build the project and run a reproduction with a NetworkConnect rule and repeated loopback TCP connections. Done when the tracker removes established connections using the stored timestamp and stale entries are swept on busy hosts; the issue says the proposed patch has not been built or tested.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Version: sysmonforlinux 1.5.3 (networkTracker.cpp is unchanged on main at b2f11cc07a). Seen on Ubuntu 22.04 (kernel 5.15) and Ubuntu with kernel 7.0.

Symptom

The sysmon heap grows linearly with the host's rate of outbound TCP connections and is only freed by a restart. On a 4 GB nginx reverse proxy making about 11 outbound connections per second, the sysmon heap (VmData) reached 2.6 GB after 36 days (about 70 MB/day), leaving the host with 60 MB available. Hosts with few outbound connections grow only a few MB per day, which hides the bug.

Any configuration with a NetworkConnect rule triggers it, because processNetworkEvent feeds the tracker before rule matching. An include rule that matches only wget/curl, plus a 127. exclude, still leaks on every connection.

Reproduction

With any NetworkConnect include rule loaded, open 20,000 loopback TCP connections (connect, then close). The sysmon RSS grows by about 2 MB (roughly 100 bytes per connection) and never shrinks.

Evidence

A sample of the long-running process's heap was almost entirely 64-byte malloc chunks, each a red-black tree node whose value is (LONGLONG time in 100 ns ticks, kernel socket pointer). That is the node layout of std::map<LONGLONG, const void *>, i.e. connectTrackerTimes. The heap held about 42 million of them, with timestamps three weeks old, although staleDuration is 15 minutes.

Cause

Two bugs in networkTracker.cpp:

  1. Wrong erase key on TCP_ESTABLISHED. In SeenConnect, the branch for an already-tracked socket reaching TCP_ESTABLISHED calls connectTrackerTimes.erase(event->m_EventTime.QuadPart). That is the time of the ESTABLISHED event, but the entry was inserted under the time of the earlier SYN_SENT event, which is stored in ret.first->second.second. The erase misses, so every successful outbound connection orphans one connectTrackerTimes node. The TCP_CLOSE branch already erases by the stored time correctly.

  2. PurgeStale never runs on a busy host. PurgeStale refreshes timeLastChecked at the end of every call, not only after a sweep. It is called on every connect and accept, so curTime - timeLastChecked > checkDuration (60 s) holds only after a full minute with no TCP connect or accept events at all. On a server that never happens, so the 15-minute stale sweep that would collect the orphans never runs. This also stops acceptTrackerTimes from ever being swept.

Proposed fix

--- a/networkTracker.cpp
+++ b/networkTracker.cpp
@@ void NetworkTracker::PurgeStale()
                 break; // as this is an ordered map, we have exhausted all stale items
             }
         }
-    }
-
-    GetSystemTimeAsLargeInteger(&timeLastChecked);
+        timeLastChecked = curTime;
+    }
 }
@@ pid_t NetworkTracker::SeenConnect(PSYSMON_EVENT_HEADER eventHdr)
             if (event->m_NewState == TCP_ESTABLISHED) {
                 // connection is established, so remove it
-                connectTracker.erase(event->m_SockId);
-                connectTrackerTimes.erase(event->m_EventTime.QuadPart);
+                connectTrackerTimes.erase(ret.first->second.second);
+                connectTracker.erase(ret.first);
             }

The first hunk lets the periodic sweep run as intended, which also acts as a backstop for any other path that leaves entries behind. The second removes the orphan at its source. The patch has not been built or tested yet.

Workaround

Cap the service with a systemd drop-in, for example MemoryMax=512M with the unit's existing Restart=on-failure, plus StartLimitIntervalSec=0. The restart limit matters because a restarted sysmon can fail with libbpf: failed to mmap perf buffer: -ENOMEM on a fragmented host and would otherwise give up after five attempts.

Possibly related: #211 (memory growth, though its Valgrind output points at ProcessCache).

Vorherrschende Sprache
C
Sterne
2.2k
Forks
224
Ø Merge
4 Std. 50 Min.
Gemergte PRs (30 T.)
3

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/Sysmon-Linux

Alle Issues in microsoft/Sysmon-Linux

Ähnliche Issues

Weitere Issues zu C

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.