jetstack/version-checker

version-checker seemingly leaks memory and gets oom-killed

Open

#76 opened on Feb 20, 2021

View on GitHub
 (8 comments) (5 reactions) (2 assignees)Go (80 forks)auto 404
enhancementhelp wanted

Repository metrics

Stars
 (771 stars)
PR merge metrics
 (Avg merge 1m) (2 merged PRs in 30d)

Description

I am running version-checker on a single node, quite small cluster with ~60 pods. So far it is working nicely, but I do not understand the memory behavior it has.

I'm basically running the sample deployment file, plus the --test-all-containers flag and some cpu limits:

        resources:
          requests:
            cpu: 10m
            memory: 32M
          limits:
            cpu: 50m
            memory: 128M

kubectl get pod -o yaml

Over time, I see that version-checker approaches the memory limit and then stays near ~99% for a while. After some time, the kernel kills the ct due to OOM and k8s restarts the pod.

Memory chart

However, I do not see anything alarming in the logs, other than some failures and expected permission errors.

This doesn't seem to have any functional impact, but does fire some alerts and doesn't look good on my dashboards :)

Is this behavior intended, and/or is there any way to prevent it?

Contributor guide

Research direction
Investigate memory allocation patterns in the main loop, possibly using Go's pprof. Check for any resource leaks in HTTP requests to registry or Kubernetes API. Look at the deployment configuration to see if limits are appropriate, and review the issue comments for any suggested investigation by maintainers.
Tech stack
gokubernetes
Domain
backenddevopsobservability
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Go programming basicsKubernetes fundamentalsmemory profiling with pprof
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15