Scan container images for misplaced secrets

Summary

The ROADMAP lists "Scanning for misplaced secrets in container images (Monitoring phase)" under Planning, but I couldn't find a tracking issue for it. This issue proposes an approach, shares some measurements from a prototype, and asks a few design questions. If the direction looks right, I'll write a design doc in designs-and-proposals before any implementation.

Background

• kubescape/kubevuln#206 (@slashben, Jan 2024) prototyped regex-based detection in kubevuln. @matthyx commented: "this will be implemented in node-agent".
• kubescape/kubevuln#296 (Dive + TruffleHog) was closed. TruffleHog is AGPL-3.0, the PR pulled the image a second time, and it stored raw secret values.
• node-agent already has every image layer on disk (overlay lowerdir) when it builds SBOMs, so scanning here needs no registry pull and no credentials.

Problem

Credentials often end up baked into images: a copied .env, an SSH key used during the build, or ENV API_TOKEN=... in a Dockerfile. Anyone who can pull the image can read them, and deleting the file in a later layer doesn't remove it, because the earlier layer still contains it.

Kubescape doesn't detect any of this today:

• C-0012 (Applications credentials in configuration files) only checks pod-spec env vars and ConfigMaps. It never looks inside the image.
• The SBOM pipeline already reads every file in every image layer, but only to catalogue packages.

This proposal follows @matthyx's direction (node-agent) and builds on the #206 rule set.

Why node-agent

With nodeSbomGeneration enabled (the chart default), node-agent already has the image layers on disk when it builds the SBOM:

• getMountedVolumes reads the container's overlay lowerdir list, which is one directory per image layer.
• NodeResolver exposes those layers to Syft, and each file location carries its layer diff-ID.

So a secret scan here needs no registry pull, no pull credentials, and no second copy of the image. It's one extra pass over files node-agent already reads.

Measurements

I wrote a throwaway spike that walks every layer of an image, runs the #206 rules on regular text files (≤10 MB, NUL-byte binaries skipped) and prints only rule/path/line (never values). I ran it once more with the gitleaks v8 library (default config) for comparison.

Image	Text scanned	#206 rules	#206 rules + keyword prefilter	gitleaks defaults
python:3.12-slim	33 MB	10.5 s, 0 hits	—	12.6 s, 0 hits
nginx:1.27	15 MB	4.7 s, 0 hits	—	6.3 s, 0 hits
node:22-slim	36 MB	12.0 s, 3 hits	3.0 s, 3 hits	30.7 s, 10 hits
postgres:16	66 MB	22.6 s, 1 hit	4.7 s, 1 hit	22.9 s, 2 hits
golang:1.23-alpine	108 MB	34.9 s, 4 hits	5.2 s, 4 hits	40.7 s, 61 hits
quay.io/kubescape/node-agent	1 MB	0.4 s, 0 hits	—	—

Observations:

• No hit was a real leak. Every hit was a false positive (npm docs, Go stdlib test keys, entropy matches in source) or intentional (/etc/ssl/private/ssl-cert-snakeoil.key from Debian's ssl-cert package). False-positive control has to be part of the core design, not a later add-on.
• Regex is the whole cost. Reading the files takes under 0.6 s even for the Go image. A simple per-rule keyword check before running each regex gave identical findings 4–7× faster.
• gitleaks' defaults are tuned for git repositories. Its generic-api-key rule produced 57 of the 61 hits on the Go image.

I also built a small fixture image with crane that has a private key added in one layer and whited-out in the next, a .env overwritten in a later layer, a directory hidden by an opaque whiteout, and ENV GH_TOKEN=ghp_... in the image config.

• The layer-aware scan found the removed key, the overwritten .env and the hidden file, and labelled them as not visible at runtime.
• It missed the ENV token, because image config isn't a file. This is a separate, cheap check.

How the spike ran

• Images were pulled with syft.GetSource (kubevuln's registry path), linux/amd64, on a laptop. node-agent would read the same layers from overlay directories instead, so pull time isn't relevant there. The "walk" times above exclude the pull.
• Layer visibility was taken from Syft's SquashedScope vs AllLayersScope.
• Happy to share the spike code if useful. It's a measurement tool, not a proposed implementation.

Proposal (phase 1)

1. Detector package (secretscan): a rule set with per-rule keywords (prefilter), path allowlists for known test/doc locations, stop-words for documented example values (e.g. AKIAIOSFODNN7EXAMPLE), size and binary limits, and streaming reads.
2. Image config scan: Env, History[].created_by and Labels. node-agent already has these from the CRI ImageStatus.
3. node-agent integration next to SbomManager (in-process and sidecar):
   • layer-aware visibility: "visible at runtime" vs "only in an earlier layer"
   • its own scanner-version key, so images that already have an SBOM still get scanned, and rules can be updated later
   • CPU, byte and time budgets; a partial scan is reported as incomplete, never as clean
4. Storage: findings stored per image (like the SBOM, once per image rather than per node).
5. Exceptions: a spec.secrets[] section in SecurityException so users can accept known findings such as the snakeoil key. match.images already covers image selection.
6. Operations: a Helm capability flag (off by default at first) and metrics (files scanned, files skipped as binary or too large, findings by rule, duration).

Findings never contain the secret value, not even a partial one. Only rule, path, layer, line and a fingerprint that can't be reversed. A plain hash of a weak password can be brute-forced, so the fingerprint should be either rule + path + layer + line, or an HMAC with a per-cluster key.

Illustrative finding (the shape is up for discussion):

- ruleID: github-pat
  severity: Critical
  source: file            # file | config.env | config.history | config.label
  path: /app/config/settings.py
  line: 12
  layer: sha256:ed87f7…   # diff-ID of the layer that contains it
  visibleAtRuntime: false # removed or overwritten by a later layer
  fingerprint: 3f9c…      # not derived from the value alone

Edge cases the design needs to cover

Layers

• A file deleted by a whiteout, overwritten by a later layer, or hidden by an opaque directory: still reported, marked not visible at runtime.
• On the node, whiteouts are 0/0 character devices and opaque directories are marked with trusted.overlay.opaque / user.overlay.opaque xattrs. Nothing in node-agent handles these today, so visibility has to be computed explicitly. NodeResolver.AllLocations does return files from every layer, which is what the scan needs.
• The same file in several layers: one finding with a list of layers.
• Symlinks and hardlinks: scan regular files only, and don't count them twice.

Content

• Binaries, files over the size limit, archives (.tgz, .jar, .whl): skipped in phase 1 and counted in metrics.
• Documented example values, runtime test data (/usr/local/go/src/**/testdata, Python tests/) and CA bundles and public certificates: allowlisted or never matched. The path allowlist must be configurable, because users' own applications can also live under paths like site-packages.
• Intentional keys such as ssl-cert-snakeoil.key: reported, but can be accepted with SecurityException.

Node and runtime

• Without overlayfs, node-agent falls back to /proc/<pid>/root, which is the merged live filesystem. It has no layer history and likely includes the container's mounted volumes (Secret volumes, service-account tokens). Syft's mount skipping uses node-agent's own mount table, not the container's. I haven't verified this on a ZFS/btrfs node yet (see question 4).
• The scan must stay within node-agent's resource budget and must not delay SBOM generation.

Test plan

• A fixture image built in CI with crane, with known answers for each case above: whiteout, overwrite, opaque directory, config ENV, test-fixture path, binary, large file, symlink.
• Unit tests for each rule (true and false examples), the prefilter and the visibility logic, including whiteout and opaque cases on a directory tree that mimics overlay lowerdirs.
• A false-positive check against a few popular images (like the table above) to catch rule regressions.
• A node-agent component test on kind: deploy the fixture image, then assert the stored findings and that no secret value appears in storage, logs or metrics.

Later (not phase 1)

• Runtime relevancy: join findings with ContainerProfile.spec.opens to flag secrets that the container actually reads, similar to how relevancy works for CVEs.
• Reuse the package in kubevuln (registry scans) and kubescape scan image.
• prometheus-exporter counters.

Out of scope: archives and binaries, host filesystem scanning, verifying secrets against live APIs, and secret rotation or revocation.

Questions

1. Should the detector live only in node-agent, or in a shared module so kubevuln and the CLI can reuse it later?
2. Storage: a new kind, or an extension of an existing per-image object?
3. Rules: evolve the #206 set (Apache-2.0-compatible, no new dependencies), or use the gitleaks library (MIT, but it brings a wasm-based RE2 dependency and needed image-specific tuning)?
4. On nodes without overlayfs (/proc/<pid>/root fallback): skip secret scanning, or filter out the container's mounts?
5. Default on or off for the first release?

Next steps

If this direction makes sense, I'll write the design doc in designs-and-proposals based on the answers here, and I'm happy to take on the implementation afterwards. I can also present the measurements at a community meeting.