[BUG] Streamable HTTP server leaks ~10KB of heap per MCP initialize request, never freed (mcp-go v0.43.2 predates the session cleanup fix)

cmd/main.go の server.NewStreamableHTTPServer の配線から始め、go.mod に固定されている mcp-go のバージョンを確認します。v0.43.2 streamable_http.go のセッションクリーンアップを、issue で説明されている upstream の修正と比較し、その後、提供されている initialize と DELETE の再現手順を公開済みイメージに対して実行します。終了および放棄されたセッションによってヒープが累積的に増加しなくなれば完了です。

📋 Prerequisites

• I have searched the existing issues to avoid creating a duplicate
• By submitting this issue, you agree to follow our Code of Conduct
• I am using the latest version of the software (kagent-tools 0.2.1)
• I can consistently reproduce this issue

🎯 Affected Component(s)

cmd/main.go — the server.NewStreamableHTTPServer(...) wiring, and the pinned github.com/mark3labs/mcp-go v0.43.2 dependency in go.mod.

🚦 Impact/Severity

Blocker for long-running deployments — the server is OOMKilled on a fixed schedule determined only by its memory limit.

🐛 Bug Description

The kagent-tools HTTP server leaks roughly 10 KB of Go heap per MCP initialize request, and the memory is never released — not on client disconnect, and not even when the client sends a well-formed DELETE to terminate the session.

In a production EKS cluster this shows up as perfectly linear heap growth of +12.4 MiB/day until the container hits its memory limit and is OOMKilled. With a 256Mi limit the pod survives about 17 days, then restarts and the cycle repeats.

The growth is time-driven, not workload-driven. Over the same period the server recorded only ~8 tool invocations per day (kagent_tools_mcp_invocations_total), while go_goroutines stayed flat at 22 and the number of exported Prometheus series stayed flat at 130–133. So this is neither a goroutine leak nor metric-cardinality growth — it is retained live objects that scale with the number of MCP sessions ever created.

Production heap growth (16 days, single pod)

xychart-beta
    title "go_memstats_heap_inuse_bytes (MiB) - kagent-tools 0.2.1, production"
    x-axis "days since pod start" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
    y-axis "heap in use (MiB)" 0 --> 260
    line [18.5, 30.9, 42.2, 56.4, 68.2, 80.5, 93.3, 105.4, 118.2, 130.3, 142.8, 155.2, 167.4, 180.0, 192.5, 204.8]

day	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
heap in use (MiB)	18.5	30.9	42.2	56.4	68.2	80.5	93.3	105.4	118.2	130.3	142.8	155.2	167.4	180.0	192.5	204.8

Slope is +12.4 MiB/day with no plateau and no sawtooth — GC never reclaims any of it. process_resident_memory_bytes tracks the same slope and reached 261 MiB right before the OOMKill.

Other counters over the same window:

metric	day 3	day 16	change
go_memstats_heap_inuse_bytes	43 MiB	214 MiB	+12.3 MiB/day
go_memstats_heap_objects	101k	484k	+27k/day
go_goroutines	22	22	flat
exported Prometheus series	130	133	flat

🔄 Steps To Reproduce

Everything below runs against the published image, no cluster required.

1. Start the server:

   podman run -d --rm --name kt -p 18084:8084 -p 18085:8085 \
     ghcr.io/kagent-dev/kagent/tools:0.2.1 \
     --read-only --port 8084 --metrics-port 8085
   

2. Record the live heap. Sampling /metrics repeatedly and taking the minimum approximates the post-GC live set:

   probe() { for i in $(seq 20); do
     curl -s localhost:18085/metrics | awk '/^go_memstats_heap_alloc_bytes /{print $2}'
   done | sort -n | head -1; }
   probe
   

3. Send MCP initialize requests in batches of 500, probing after each batch:

   for batch in 1 2 3 4; do
     for i in $(seq 500); do
       curl -s -o /dev/null -X POST localhost:18084/mcp \
         -H 'Content-Type: application/json' \
         -H 'Accept: application/json, text/event-stream' \
         -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'
     done
     echo "$((batch*500)) sessions: $(probe)"
   done
   

4. Observe that the live heap grows linearly and never comes back down, no matter how long you wait.

5. Repeat the run, but this time capture Mcp-Session-Id from each response and send DELETE /mcp with that header. The per-session cost is unchanged.

🤔 Expected Behavior

Per-session state should be released when a session ends:

• A DELETE with a valid Mcp-Session-Id should free everything associated with that session.
• Sessions abandoned without a DELETE (client crash, pod restart, connection reset, LB timeout) should eventually be reclaimed.

Steady-state memory should be a function of concurrent sessions, not of cumulative sessions.

📱 Actual Behavior

Measured on a fresh container (heap_alloc_bytes, post-GC trough):

xychart-beta
    title "Live heap vs cumulative initialize requests (local, kagent-tools 0.2.1)"
    x-axis "cumulative initialize requests" [0, 500, 1000, 1500, 2000]
    y-axis "live heap (MB)" 0 --> 26
    line [3.28, 7.84, 13.43, 18.99, 23.35]

cumulative initialize requests	0	500	1000	1500	2000
live heap (MB)	3.28	7.84	13.43	18.99	23.35

That is +20.1 MB over 2000 sessions ≈ 10 KB retained per session, linear, permanent.

Control runs isolating the other traffic on the same ports:

traffic	requests	live heap delta	per request
GET /health	5000	+0.05 MB	~11 bytes (noise)
POST /mcp initialize, no DELETE	1000	+8.66 MB	~8.7 KB
POST /mcp initialize + DELETE /mcp	500	+4.40 MB	~8.8 KB

The last row is the important one: sending DELETE frees nothing.

🔍 Root Cause Analysis

cmd/main.go builds a stateful streamable HTTP server:

sseServer := server.NewStreamableHTTPServer(mcp,
    server.WithHeartbeatInterval(30*time.Second),
)

In mcp-go v0.43.2 (the version pinned in go.mod), StreamableHTTPServer registers a session on initialize over POST:

// server/streamable_http.go, handlePost
if isInitializeRequest && sessionID != "" {
    if _, exists := s.server.sessions.Load(sessionID); !exists {
        s.activeSessions.Store(sessionID, session)
        if err := s.server.RegisterSession(ctx, session); err != nil { ... }
    }
}

but handleDelete in that version never undoes it:

// server/streamable_http.go, handleDelete (v0.43.2)
s.sessionTools.delete(sessionID)
s.sessionResources.delete(sessionID)
s.sessionResourceTemplates.delete(sessionID)
s.sessionLogLevels.delete(sessionID)
s.sessionRequestIDs.Delete(sessionID)
// no s.server.UnregisterSession(...)
// no s.activeSessions.Delete(...)

UnregisterSession + activeSessions.Delete only appear on the handleGet path (as deferred calls when the SSE stream ends). A client that speaks POST-only streamable HTTP — which is the normal request/response mode — therefore leaves one entry in activeSessions and one in server.sessions forever, for every initialize it ever sent. That matches the measured ~10 KB per session exactly, and it explains why DELETE changes nothing.

Upstream has since fixed this. mark3labs/mcp-go PR #724 ("fix: add session idle TTL sweeper to prevent transport state leak") landed in v0.44.1 and introduced cleanupSessionState, which handleDelete now calls:

func (s *StreamableHTTPServer) cleanupSessionState(ctx context.Context, sessionID string) {
    s.server.UnregisterSession(ctx, sessionID)
    s.activeSessions.Delete(sessionID)
    s.sessionTools.delete(sessionID)
    ...
}

plus a background sweeper that reclaims sessions idle longer than a configurable TTL.

kagent-tools 0.2.1 pins mcp-go v0.43.2, which is the release immediately before that fix.

Production arithmetic cross-check

10 KB per session against the observed +12.4 MiB/day implies roughly 1,300 leaked sessions per day, i.e. about one new session per minute. That is consistent with a controller that re-initializes its MCP connection on a short interval, and it is consistent with the leak being independent of the ~8 tool invocations actually served per day.

💻 Environment

• Kubernetes: EKS 1.36
• Kubernetes provider: AWS
• kagent-tools version: 0.2.1 (image ghcr.io/kagent-dev/kagent/tools:0.2.1, git commit 9d18d83)
• Deployment: upstream kagent-tools Helm chart 0.2.1, --read-only, single replica, metrics on a separate port
• Local reproduction: podman 6.1.2 on macOS 26 (arm64), same image

🛠 Proposed Fix

1. Bump github.com/mark3labs/mcp-go to ≥ v0.44.1 (current is v1.1.0) so handleDelete actually unregisters the session.

2. Enable the sweeper explicitly in cmd/main.go. It is opt-in even on v1.1.0 — WithSessionIdleTTL defaults to zero, which disables it — so a dependency bump alone is not sufficient for clients that never send DELETE:

   sseServer := server.NewStreamableHTTPServer(mcp,
       server.WithHeartbeatInterval(30*time.Second),
       server.WithSessionIdleTTL(10*time.Minute), // value up for discussion
   )
   

3. Optional but valuable for future diagnosis: expose net/http/pprof on the metrics port (behind a flag). The image is distroless and has no shell, so there is currently no way to pull a heap profile from a running pod, which is why this issue leans on go_memstats_* and a local reproduction instead of a pprof dump.

🔍 Additional Context

• Workaround in use today is purely palliative: raise resources.limits.memory to stretch the OOMKill interval. At the measured 12.4 MiB/day, 256Mi buys ~17 days, 384Mi ~27 days, 512Mi ~37 days. Nothing bounds it.
• With replicaCount: 1 each OOMKill is a short outage for every agent that depends on this MCP server.
• Did it ever work? Not on 0.2.1. The pinned mcp-go v0.43.2 predates the upstream fix, so any release using that pin should be affected.

🙋 Are you willing to contribute?

• I am willing to submit a PR to fix this issue