NetworkTracker leaks a connectTrackerTimes node per outbound TCP connection, and PurgeStale never runs on busy hosts

Ouverte
  • Quelque chose ne va pas ?
#247 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
55/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
c
Domaine
networking

Piste de recherche

Start by reading networkTracker.cpp, especially NetworkTracker::PurgeStale and NetworkTracker::SeenConnect; the issue identifies both faulty paths and includes a proposed fix. Build the project and run a reproduction with a NetworkConnect rule and repeated loopback TCP connections. Done when the tracker removes established connections using the stored timestamp and stale entries are swept on busy hosts; the issue says the proposed patch has not been built or tested.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Version: sysmonforlinux 1.5.3 (networkTracker.cpp is unchanged on main at b2f11cc07a). Seen on Ubuntu 22.04 (kernel 5.15) and Ubuntu with kernel 7.0.

Symptom

The sysmon heap grows linearly with the host's rate of outbound TCP connections and is only freed by a restart. On a 4 GB nginx reverse proxy making about 11 outbound connections per second, the sysmon heap (VmData) reached 2.6 GB after 36 days (about 70 MB/day), leaving the host with 60 MB available. Hosts with few outbound connections grow only a few MB per day, which hides the bug.

Any configuration with a NetworkConnect rule triggers it, because processNetworkEvent feeds the tracker before rule matching. An include rule that matches only wget/curl, plus a 127. exclude, still leaks on every connection.

Reproduction

With any NetworkConnect include rule loaded, open 20,000 loopback TCP connections (connect, then close). The sysmon RSS grows by about 2 MB (roughly 100 bytes per connection) and never shrinks.

Evidence

A sample of the long-running process's heap was almost entirely 64-byte malloc chunks, each a red-black tree node whose value is (LONGLONG time in 100 ns ticks, kernel socket pointer). That is the node layout of std::map<LONGLONG, const void *>, i.e. connectTrackerTimes. The heap held about 42 million of them, with timestamps three weeks old, although staleDuration is 15 minutes.

Cause

Two bugs in networkTracker.cpp:

  1. Wrong erase key on TCP_ESTABLISHED. In SeenConnect, the branch for an already-tracked socket reaching TCP_ESTABLISHED calls connectTrackerTimes.erase(event->m_EventTime.QuadPart). That is the time of the ESTABLISHED event, but the entry was inserted under the time of the earlier SYN_SENT event, which is stored in ret.first->second.second. The erase misses, so every successful outbound connection orphans one connectTrackerTimes node. The TCP_CLOSE branch already erases by the stored time correctly.

  2. PurgeStale never runs on a busy host. PurgeStale refreshes timeLastChecked at the end of every call, not only after a sweep. It is called on every connect and accept, so curTime - timeLastChecked > checkDuration (60 s) holds only after a full minute with no TCP connect or accept events at all. On a server that never happens, so the 15-minute stale sweep that would collect the orphans never runs. This also stops acceptTrackerTimes from ever being swept.

Proposed fix

--- a/networkTracker.cpp
+++ b/networkTracker.cpp
@@ void NetworkTracker::PurgeStale()
                 break; // as this is an ordered map, we have exhausted all stale items
             }
         }
-    }
-
-    GetSystemTimeAsLargeInteger(&timeLastChecked);
+        timeLastChecked = curTime;
+    }
 }
@@ pid_t NetworkTracker::SeenConnect(PSYSMON_EVENT_HEADER eventHdr)
             if (event->m_NewState == TCP_ESTABLISHED) {
                 // connection is established, so remove it
-                connectTracker.erase(event->m_SockId);
-                connectTrackerTimes.erase(event->m_EventTime.QuadPart);
+                connectTrackerTimes.erase(ret.first->second.second);
+                connectTracker.erase(ret.first);
             }

The first hunk lets the periodic sweep run as intended, which also acts as a backstop for any other path that leaves entries behind. The second removes the orphan at its source. The patch has not been built or tested yet.

Workaround

Cap the service with a systemd drop-in, for example MemoryMax=512M with the unit's existing Restart=on-failure, plus StartLimitIntervalSec=0. The restart limit matters because a restarted sysmon can fail with libbpf: failed to mmap perf buffer: -ENOMEM on a fragmented host and would otherwise give up after five attempts.

Possibly related: #211 (memory growth, though its Valgrind output points at ProcessCache).

Langage dominant
C
Étoiles
2.2k
Forks
224
Merge moyen
4 h 50 min
PR mergées (30 j)
3

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de microsoft/Sysmon-Linux

Toutes les issues de microsoft/Sysmon-Linux

Issues similaires

Plus d'issues C

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.