Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Steady ~15 MiB/day memory growth on an idle gateway (2026.2.4, Linux container)

Open
  • Is something wrong?
#1,973 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
backend, networking, performance

Research direction

Start by reproducing the idle process with the Kubernetes probes polling GET /jet/health every 20 seconds and 60 seconds, while sampling container_memory_rss hourly for 48 hours. Compare runs with the probes disabled, then use a diagnostic build or heap profile if available. Done means identifying the retaining path and confirming that RSS no longer grows linearly during an idle run.

Written by the indexing model from the issue text.

Description

Summary

devolutions-gateway 2026.2.4 grows its RSS by a constant ~15.3 MiB/day while completely idle. The growth is linear, does not depend on traffic, and does not level off. In a container with a 256 MiB memory limit this ends in an OOM kill after about 16 days; on a host without a limit it would just keep climbing.

Environment
  • Image: docker.io/devolutions/devolutions-gateway:2026.2.4 (sha256:f9ea0f4d…54887), amd64
  • Runtime: Kubernetes 1.34 / containerd, Talos Linux, cgroup v2, memory limit 256 MiB
  • Started with the packaged binary directly rather than the pwsh entrypoint:
    /opt/devolutions/gateway/devolutions-gateway --config-path /var/lib/devolutions-gateway
  • Single instance, one HTTP listener, web app enabled with "Authentication": "None" (an external reverse proxy does the authentication)
{
  "Listeners": [{ "InternalUrl": "http://*:7171", "ExternalUrl": "https://*:443" }],
  "WebApp": { "Enabled": true, "Authentication": "None" },
  "ProvisionerPublicKeyFile": "…/provisioner.pub",
  "ProvisionerPrivateKeyFile": "…/provisioner.key",
  "JrlFile": "…/jrl.json",
  "TlsVerifyStrict": true
}
Observed behaviour

container_memory_rss, one sample per day, over a single 15.7-day process lifetime:

08-22  38.4 MiB
08-23  53.5 MiB   (+15.1)
08-24  68.6 MiB   (+15.1)
08-25  83.8 MiB   (+15.2)
08-26  99.1 MiB   (+15.3)
08-27 114.5 MiB   (+15.4)
08-28 129.9 MiB   (+15.4)
08-29 145.3 MiB   (+15.4)
08-30 160.5 MiB   (+15.2)
08-31 175.8 MiB   (+15.3)
09-01 191.6 MiB   (+15.8)
09-02 207.7 MiB   (+16.1)
09-03 223.5 MiB   (+15.9)
09-04 240.1 MiB   (+16.5)
09-05          -> OOMKilled at 253.9 MiB against a 256 MiB limit

Page cache is not involved: container_memory_cache was 0.2 MiB at the time of the kill, so all of this is anonymous memory in the process.

Why it looks usage-independent
  • Between process start and the first user request there were 12 days with zero traffic — the slope over that stretch is identical to the rest.
  • The last log line before the OOM was written ~40 h earlier (the end of an RDP session). Nothing was logged in those 40 h, and the curve keeps rising at the same rate.
  • The replacement process, which nobody has used at all, is leaking at 15.26 MiB/day — measured as deriv(container_memory_rss[90m]).

So something on a timer or on a periodically-repeated request path accumulates memory.

Hypothesis

The only recurring input to the idle process is the health endpoint: a readiness probe every 20 s and a liveness probe every 60 s, i.e. ~4320 requests/day to GET /jet/health. That works out to roughly 3.7 KB retained per health request, which is in the right ballpark for a per-request leak (a retained span, connection record, or similar).

This is a hypothesis, not a proof — I have not yet run the process with the probes disabled to confirm. One detail that may matter: requests to /jet/health do not appear in the devolutions_gateway::middleware::log output, while every other request does, so they seem to take a different path through the stack.

Steps to reproduce (expected)
  1. Run 2026.2.4 in a container with the config above and no user traffic.
  2. Poll GET /jet/health every 20 s.
  3. Sample RSS hourly for 48 h.

Expected: a straight line at roughly +0.64 MiB/h.

Questions
  • Is a background task (job queue, traffic audit, JRL refresh, mDNS/network-scanner daemon) known to retain memory when idle?
  • Is there a build or an env var that would let me hand you a heap profile from the running container?

I can run any diagnostic build you point me at — the instance is easy to redeploy.

Dominant language
Rust
Stars
161
Forks
30
Avg merge
2d 10m
Merged PRs (30d)
22

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Devolutions/devolutions-gateway

All issues in Devolutions/devolutions-gateway

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.