[BUG] Streamable HTTP server leaks ~10KB of heap per MCP initialize request, never freed (mcp-go v0.43.2 predates the session cleanup fix)

Start in cmd/main.go at the server.NewStreamableHTTPServer wiring and inspect the pinned mcp-go version in go.mod. Compare the v0.43.2 streamable_http.go session cleanup with the upstream fix described in the issue, then use the provided initialize and DELETE reproduction against the published image. Done means terminated and abandoned sessions no longer cause cumulative heap growth.

📋 Prerequisites

• I have searched the existing issues to avoid creating a duplicate
• By submitting this issue, you agree to follow our Code of Conduct
• I am using the latest version of the software (kagent-tools 0.2.1)
• I can consistently reproduce this issue

🎯 Affected Component(s)

cmd/main.go — the server.NewStreamableHTTPServer(...) wiring, and the pinned github.com/mark3labs/mcp-go v0.43.2 dependency in go.mod.

🚦 Impact/Severity

Blocker for long-running deployments — the server is OOMKilled on a fixed schedule determined only by its memory limit.

🐛 Bug Description

The kagent-tools HTTP server leaks roughly 10 KB of Go heap per MCP initialize request, and the memory is never released — not on client disconnect, and not even when the client sends a well-formed DELETE to terminate the session.

In a production EKS cluster this shows up as perfectly linear heap growth of +12.4 MiB/day until the container hits its memory limit and is OOMKilled. With a 256Mi limit the pod survives about 17 days, then restarts and the cycle repeats.

The growth is time-driven, not workload-driven. Over the same period the server recorded only ~8 tool invocations per day (kagent_tools_mcp_invocations_total), while go_goroutines stayed flat at 22 and the number of exported Prometheus series stayed flat at 130–133. So this is neither a goroutine leak nor metric-cardinality growth — it is retained live objects that scale with the number of MCP sessions ever created.

Production heap growth (16 days, single pod)

xychart-beta
    title "go_memstats_heap_inuse_bytes (MiB) - kagent-tools 0.2.1, production"
    x-axis "days since pod start" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
    y-axis "heap in use (MiB)" 0 --> 260
    line [18.5, 30.9, 42.2, 56.4, 68.2, 80.5, 93.3, 105.4, 118.2, 130.3, 142.8, 155.2, 167.4, 180.0, 192.5, 204.8]

day	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
heap in use (MiB)	18.5	30.9	42.2	56.4	68.2	80.5	93.3	105.4	118.2	130.3	142.8	155.2	167.4	180.0	192.5	204.8

Slope is +12.4 MiB/day with no plateau and no sawtooth — GC never reclaims any of it. process_resident_memory_bytes tracks the same slope and reached 261 MiB right before the OOMKill.

Other counters over the same window:

metric	day 3	day 16	change
go_memstats_heap_inuse_bytes	43 MiB	214 MiB	+12.3 MiB/day
go_memstats_heap_objects	101k	484k	+27k/day
go_goroutines	22	22	flat
exported Prometheus series	130	133	flat

🔄 Steps To Reproduce

Everything below runs against the published image, no cluster required.

1. Start the server:

   podman run -d --rm --name kt -p 18084:8084 -p 18085:8085 \
     ghcr.io/kagent-dev/kagent/tools:0.2.1 \
     --read-only --port 8084 --metrics-port 8085
   

2. Record the live heap. Sampling /metrics repeatedly and taking the minimum approximates the post-GC live set:

   probe() { for i in $(seq 20); do
     curl -s localhost:18085/metrics | awk '/^go_memstats_heap_alloc_bytes /{print $2}'
   done | sort -n | head -1; }
   probe
   

3. Send MCP initialize requests in batches of 500, probing after each batch:

   for batch in 1 2 3 4; do
     for i in $(seq 500); do
       curl -s -o /dev/null -X POST localhost:18084/mcp \
         -H 'Content-Type: application/json' \
         -H 'Accept: application/json, text/event-stream' \
         -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'
     done
     echo "$((batch*500)) sessions: $(probe)"
   done
   

4. Observe that the live heap grows linearly and never comes back down, no matter how long you wait.

5. Repeat the run, but this time capture Mcp-Session-Id from each response and send DELETE /mcp with that header. The per-session cost is unchanged.

🤔 Expected Behavior

Per-session state should be released when a session ends:

• A DELETE with a valid Mcp-Session-Id should free everything associated with that session.
• Sessions abandoned without a DELETE (client crash, pod restart, connection reset, LB timeout) should eventually be reclaimed.

Steady-state memory should be a function of concurrent sessions, not of cumulative sessions.

📱 Actual Behavior

Measured on a fresh container (heap_alloc_bytes, post-GC trough):

xychart-beta
    title "Live heap vs cumulative initialize requests (local, kagent-tools 0.2.1)"
    x-axis "cumulative initialize requests" [0, 500, 1000, 1500, 2000]
    y-axis "live heap (MB)" 0 --> 26
    line [3.28, 7.84, 13.43, 18.99, 23.35]

cumulative initialize requests	0	500	1000	1500	2000
live heap (MB)	3.28	7.84	13.43	18.99	23.35

That is +20.1 MB over 2000 sessions ≈ 10 KB retained per session, linear, permanent.

Control runs isolating the other traffic on the same ports:

traffic	requests	live heap delta	per request
GET /health	5000	+0.05 MB	~11 bytes (noise)
POST /mcp initialize, no DELETE	1000	+8.66 MB	~8.7 KB
POST /mcp initialize + DELETE /mcp	500	+4.40 MB	~8.8 KB

The last row is the important one: sending DELETE frees nothing.

🔍 Root Cause Analysis

cmd/main.go builds a stateful streamable HTTP server:

sseServer := server.NewStreamableHTTPServer(mcp,
    server.WithHeartbeatInterval(30*time.Second),
)

In mcp-go v0.43.2 (the version pinned in go.mod), StreamableHTTPServer registers a session on initialize over POST:

// server/streamable_http.go, handlePost
if isInitializeRequest && sessionID != "" {
    if _, exists := s.server.sessions.Load(sessionID); !exists {
        s.activeSessions.Store(sessionID, session)
        if err := s.server.RegisterSession(ctx, session); err != nil { ... }
    }
}

but handleDelete in that version never undoes it:

// server/streamable_http.go, handleDelete (v0.43.2)
s.sessionTools.delete(sessionID)
s.sessionResources.delete(sessionID)
s.sessionResourceTemplates.delete(sessionID)
s.sessionLogLevels.delete(sessionID)
s.sessionRequestIDs.Delete(sessionID)
// no s.server.UnregisterSession(...)
// no s.activeSessions.Delete(...)

UnregisterSession + activeSessions.Delete only appear on the handleGet path (as deferred calls when the SSE stream ends). A client that speaks POST-only streamable HTTP — which is the normal request/response mode — therefore leaves one entry in activeSessions and one in server.sessions forever, for every initialize it ever sent. That matches the measured ~10 KB per session exactly, and it explains why DELETE changes nothing.

Upstream has since fixed this. mark3labs/mcp-go PR #724 ("fix: add session idle TTL sweeper to prevent transport state leak") landed in v0.44.1 and introduced cleanupSessionState, which handleDelete now calls:

func (s *StreamableHTTPServer) cleanupSessionState(ctx context.Context, sessionID string) {
    s.server.UnregisterSession(ctx, sessionID)
    s.activeSessions.Delete(sessionID)
    s.sessionTools.delete(sessionID)
    ...
}

plus a background sweeper that reclaims sessions idle longer than a configurable TTL.

kagent-tools 0.2.1 pins mcp-go v0.43.2, which is the release immediately before that fix.

Production arithmetic cross-check

10 KB per session against the observed +12.4 MiB/day implies roughly 1,300 leaked sessions per day, i.e. about one new session per minute. That is consistent with a controller that re-initializes its MCP connection on a short interval, and it is consistent with the leak being independent of the ~8 tool invocations actually served per day.

💻 Environment

• Kubernetes: EKS 1.36
• Kubernetes provider: AWS
• kagent-tools version: 0.2.1 (image ghcr.io/kagent-dev/kagent/tools:0.2.1, git commit 9d18d83)
• Deployment: upstream kagent-tools Helm chart 0.2.1, --read-only, single replica, metrics on a separate port
• Local reproduction: podman 6.1.2 on macOS 26 (arm64), same image

🛠 Proposed Fix

1. Bump github.com/mark3labs/mcp-go to ≥ v0.44.1 (current is v1.1.0) so handleDelete actually unregisters the session.

2. Enable the sweeper explicitly in cmd/main.go. It is opt-in even on v1.1.0 — WithSessionIdleTTL defaults to zero, which disables it — so a dependency bump alone is not sufficient for clients that never send DELETE:

   sseServer := server.NewStreamableHTTPServer(mcp,
       server.WithHeartbeatInterval(30*time.Second),
       server.WithSessionIdleTTL(10*time.Minute), // value up for discussion
   )
   

3. Optional but valuable for future diagnosis: expose net/http/pprof on the metrics port (behind a flag). The image is distroless and has no shell, so there is currently no way to pull a heap profile from a running pod, which is why this issue leans on go_memstats_* and a local reproduction instead of a pprof dump.

🔍 Additional Context

• Workaround in use today is purely palliative: raise resources.limits.memory to stretch the OOMKill interval. At the measured 12.4 MiB/day, 256Mi buys ~17 days, 384Mi ~27 days, 512Mi ~37 days. Nothing bounds it.
• With replicaCount: 1 each OOMKill is a short outage for every agent that depends on this MCP server.
• Did it ever work? Not on 0.2.1. The pinned mcp-go v0.43.2 predates the upstream fix, so any release using that pin should be affected.

🙋 Are you willing to contribute?

• I am willing to submit a PR to fix this issue