Support rotating certs in running pods instead of restarting them

Open
  • Is something wrong?
#769 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes, rust
Domain
infrastructure, security

Research direction

Start by tracing the CSIDriver configuration and the NodePublishVolume implementation, then find where the replace time is computed for the restarter annotation. The issue proposes opt-in per-volume certificate renewal, safe replacement of files in mounted volumes, and retaining restart behavior as a fallback. Done would include renewal on republish without exposing partially written certificates; the issue names no files or tests.

Written by the indexing model from the issue text.

Description

customer-request

Description

Right now, when a cert from the secret-operator gets close to expiring, the restarter just kills the pod based on the annotation set by secret-op (I think). That works, but it’s annoying for things that are slow to come back up or don't handle restarts well (looking at you, NiFi!), and it means short cert lifetimes come with a lot of restarts.

Some products can already reload certs from disk on their own. For example NiFi does this with nifi.security.autoreload.enabled. For products like that, it’d be nice if we could swap the cert in the volume and let the product take it from there, without forcing an unneeded restart on the pod.

The CSI lifecyle in Kubernetes has a hook for this: if we set requiresRepublish: true on our CSIDriver, the kubelet keeps calling NodePublishVolume for volumes that are already mounted. We can use that to check whether the cert is past its replace time and, if so, issue a new one and write it into the existing volume. We already compute that time for the restarter annotation, so we’d mainly need to keep it somewhere cheap to check, for example in memory with a small file in the volume as a fallback after the driver restarts.

This would have to be opt-in per volume, since products that don’t reload would just keep using the old cert until it expires. We’d probably also keep the restarter annotation as a safety net in case reissuing fails.

Bit of file handling to investigate since we don't want half-written certs being read, so probably need to come up with some symlink or mv magic, but that should be doable.

Dominant language
Rust
Stars
14
Forks
8
Avg merge
1d 12h
Merged PRs (30d)
9

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stackabletech/secret-operator

All issues in stackabletech/secret-operator

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.