Greenstand/treetracker-infrastructure

Set resource requests and limits on treetracker-web-map-client to prevent node memory pressure

Open

#306 aperta il 6 mag 2026

Vedi su GitHub
 (5 commenti) (0 reazioni) (1 assegnatario)HCL (40 fork)auto 404
cloud-engineeringgood first issue

Metriche repository

Star
 (23 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

No resource requests or limits set on treetracker-web-map-client

Context

This was identified while investigating periodic memory pressure on the microservices node (microservices-node-pool-d1tvb) that hosts treetracker-web-map-client. The node had MemoryPressure: True as recently as Apr 22, 2026.


Background: Kubernetes QoS classes and why they matter

Kubernetes assigns every pod a Quality of Service (QoS) class based on whether it has resource requests and limits configured. There are three classes:

  • Guaranteed — requests and limits are both set and equal. The pod gets exactly what it asks for and is the last to be evicted.
  • Burstable — requests are set but limits are higher (or only one is set). The pod can burst above its request up to its limit. Evicted after BestEffort under pressure.
  • BestEffort — no requests or limits set at all. The kubelet makes no scheduling guarantees and evicts these pods first when the node runs low on memory.

treetracker-web-map-client is currently BestEffort. So is every other pod on this node.


Current state

kubectl top shows current live memory consumption:

NAME                                                     CPU(cores)   MEMORY(bytes)
treetracker-web-map-client-deploy-854c4fb7bd-lxt2g       47m          1008Mi

The deployment has no resource configuration:

CPU Requests:    0 (0%)
CPU Limits:      0 (0%)
Memory Requests: 0 (0%)
Memory Limits:   0 (0%)

The node it runs on:

Node:               microservices-node-pool-d1tvb
Instance type:      s-2vcpu-4gb
Allocatable memory: 3110Mi
Total pods:         12

All 12 pods on this node have zero resource requests and limits. web-map-client at ~1008Mi currently occupies ~32% of allocatable memory with no ceiling.


Why web-map-client consumes significant memory

treetracker-web-map-client is a Next.js 12 server-side rendered application. When running in a container, its memory footprint comes from several sources:

  1. Node.js V8 heap — React component trees, MUI/emotion style caches, Leaflet map state, and D3 data structures are all allocated in the V8 heap during SSR. V8's garbage collector is deliberately lazy — it defers collection until heap pressure forces it, so RSS stays at its high-water mark even after requests complete.

  2. Native memory outside the heap — Node.js buffers, HTTP parser state, libuv event loop structures, and OpenSSL all allocate outside the V8 heap and are not subject to V8 garbage collection. kubectl top reports RSS, which includes all of these.

  3. Per-request overheadgetInitialProps in _app.js makes an outbound HTTP call to fetch map configuration on every page request. Under concurrent load, multiple in-flight requests each hold their own connection, response buffer, and parsed JSON payload simultaneously.

  4. devDependencies in the production image — The Dockerfile is single-stage, so npm ci installs everything including Cypress (~200MB), webpack, jest, and other build tooling into the production container. While not all of these are loaded into memory at runtime, they are present in the container filesystem and can be touched by Node's module resolver at startup.


The problem: no ceiling means unbounded growth

With no memory limit set, web-map-client can grow beyond its current ~1008Mi baseline during traffic spikes. Because the node has only 3110Mi allocatable and all pods are BestEffort, the kubelet has no contract to enforce. When memory pressure builds:

  • The Linux kernel's OOM killer or the kubelet's eviction manager selects BestEffort pods for termination
  • The selection is arbitrary — whichever pod the eviction manager targets first gets killed, regardless of whether it caused the pressure
  • Neighboring pods like treetracker-query-api (currently at only 68Mi, well-behaved) are just as vulnerable as web-map-client itself
  • A killed query-api pod means map search and tree lookup return errors to users until the pod reschedules and warms up

This is the structural condition that makes the node fragile. It does not require a bug or a leak to trigger — normal traffic growth is sufficient.


Proposed fix

Step 1: Set resource requests and limits on the deployment

Suggested starting values based on current observed usage:

resources:
  requests:
    memory: "512Mi"
    cpu: "50m"
  limits:
    memory: "1200Mi"
    cpu: "200m"

What each value does:

  • requests.memory: 512Mi — tells the Kubernetes scheduler to only place this pod on a node with at least 512Mi free. Promotes the pod from BestEffort to Burstable QoS, giving it eviction protection over unmanaged pods.
  • limits.memory: 1200Mi — hard ceiling enforced by the Linux kernel cgroup. If the container exceeds this, it receives an OOMKill rather than growing unchecked and pressuring the node. The 1200Mi value gives ~200Mi headroom above the current ~1008Mi baseline.
  • requests.cpu: 50m / limits.cpu: 200m — reasonable bounds based on observed ~47m CPU usage. CPU limits are less critical than memory limits for stability, but help the scheduler make better placement decisions.

Step 2: Add NODE_OPTIONS to align V8 heap with the container limit

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=896"

By default, Node.js V8 does not aggressively garbage collect until it approaches a self-determined heap ceiling, which can be much larger than the container memory limit. Setting --max-old-space-size to approximately 75% of the container memory limit (75% of 1200Mi ≈ 896Mi) tells V8 to GC more frequently as it approaches that threshold. This leaves headroom for native off-heap memory so the container limit is not breached before V8 has had a chance to recover memory.

Without this, even with a container limit set, Node.js may hit the cgroup ceiling before V8 GCs, resulting in OOMKill rather than graceful memory recovery.


Expected outcome

  • Pod is promoted to Burstable QoS — protected from arbitrary eviction under moderate memory pressure
  • Memory growth is bounded — if web-map-client exceeds 1200Mi it receives a clean OOMKill and reschedules, rather than silently evicting neighbors
  • V8 GCs more aggressively — reducing the likelihood of hitting the container limit under normal load
  • treetracker-query-api and other neighboring pods are insulated from web-map-client memory spikes

Notes

  • The proposed values are starting points. After deployment, memory usage should be monitored over several days to determine whether the limit needs adjustment.
  • A separate issue tracks the single-stage Dockerfile and devDependencies in the production image, which contributes to the overall container footprint.
  • A PR will follow this issue.

Guida contributor