Kafka sidecar: a volume mounted at a parent of a run.kafka.tls cert path is wrongly removed from the function container
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
Research direction
Start with kafkaSidecarVolumeMounts in pkg/k8s/deployer.go and compare its path matching with findSecretForPath in pkg/keda/kafka_scaling.go. Trace how the selected mounts are passed to excludeVolumeMounts, then run the relevant deployer tests or add a case for a PVC at /data and a Kafka Secret at /data/certs. Done when the PVC remains mounted in the function and only the Secret backing the certificate is moved to the sidecar.
Written by the indexing model from the issue text.
Description
What happens
When a function consumes from Kafka with TLS (run.kafka.tls.*), the Kafka
runtime sidecar is given the volume mount(s) that back the configured certificate
paths, and those volumes are then excluded from the function (user) container so
the private key never lands in application code's filesystem.
The mount-selection logic over-matches: it keeps every volume mount whose path
is an ancestor directory of a cert path, not just the volume that actually backs
the certificate. If an unrelated volume (a PVC, ConfigMap, emptyDir, or another
Secret) is mounted at a parent directory of a Kafka cert path, that volume is:
- mounted into the sidecar (unnecessary), and
- removed from the function container (by
excludeVolumeMounts, which drops
by volume name).
The function therefore silently loses a volume mount it depends on.
Reproduce
run:
volumes:
- persistentVolumeClaim:
claimName: data-pvc
path: /data
- secret: kafka-cert
path: /data/certs
kafka:
brokers: kafka:9092
topic: orders
consumerGroup: my-func
securityProtocol: SSL
tls:
caCert: /data/certs/ca.crt
Deploy with the raw or keda deployer. The kafka-cert secret at /data/certs
legitimately backs the CA cert, so it is moved to the sidecar. But because
/data (the PVC) is also a parent directory of /data/certs/ca.crt, the PVC mount
is swept into the same set and then excluded from the function container. The
function comes up without /data.
Root cause
kafkaSidecarVolumeMounts in pkg/k8s/deployer.go matches a mount when the cert
path equals the mount path or is nested under it:
for _, vm := range volumeMounts {
mountPath := path.Clean(vm.MountPath)
for _, cp := range certPaths {
if cp == mountPath || strings.HasPrefix(cp, mountPath+"/") {
mounts = append(mounts, vm)
break
}
}
}
This has two problems:
- It considers all volume types, not only Secrets. A Kafka TLS certificate can
only come from a Secret, so a PVC/ConfigMap/emptyDir at an ancestor path should
never be treated as backing the cert. - It keeps every ancestor mount rather than the single most-specific (longest
mount path) one that actually provides the file.
The sibling resolver on the keda path, findSecretForPath
(pkg/keda/kafka_scaling.go), already skips non-Secret volumes
(if v.Secret == nil || v.Path == nil { continue }), so the KEDA
TriggerAuthentication wiring resolves the correct secret in this scenario. The
two code paths are not actually at parity even though the comment on
kafkaSidecarVolumeMounts says they use "the same path-to-volume matching."
Impact
- Silent: no error; the function just comes up missing a volume, which typically
surfaces later as a confusing runtime failure. - Scope: only functions that both (a) configure
run.kafka.tls.*and (b) mount an
unrelated volume at a parent directory of a cert path. Mounting the cert Secret
at a path that is not nested under another volume avoids it.
Workaround
Mount the Kafka TLS Secret at a path that is not nested under another volume's
mount path (e.g. /etc/kafka/certs instead of /data/certs when /data is a
PVC).
Suggested fix
Make kafkaSidecarVolumeMounts mirror findSecretForPath: only consider Secret
volumes as cert backers, and select the single most-specific matching mount (or
resolve the exact backing Secret via findSecretForPath and mount only that
volume into the sidecar). That keeps the sidecar/user-container split correct and
brings the two code paths genuinely into parity.
Follow-up to #4060 (language-agnostic Kafka runtime sidecar).
/kind bug
- Dominant language
- Go
- Stars
- 366
- Forks
- 226
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 20
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from knative/func
-
Kafka sidecar: add a startupProbe so slow-starting functions aren't restart-loopedPossibly taken @aliok claimed this 1 day ago. Openkind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
knative/func#4097 · 1 assignee ·
Maintainers usually reply within 1 day
-
KEDA deployer: default minReplicaCount prevents scale-to-zeroPossibly taken @Tushar-Khandelwal-2004 claimed this 114 days ago. Openlifecycle/stale
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
knative/func#3924 · 1 comment ·
Maintainers usually reply within 1 day
-
git/gitlab: default project webhooks to SSL verification when creating hooksPossibly taken @Vi-shub claimed this 160 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
mcp: config add/remove handlers bypass readonly guardPossibly taken @Elvand-Lie claimed this 163 days ago. Openkind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
knative/func#3704 · 1 comment ·
Maintainers usually reply within 1 day
-
kind/documentation
Difficulty 3/5 1-2 days Newbie friendliness 72/100
Maintainers usually reply within 1 day
Similar issues
-
native
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
prime-radiant-inc/evener#4839 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
521xueweihan/HelloGitHub#4073 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
goharbor/terraform-provider-harbor#655 · 1 comment ·
-
bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
syncthing/syncthing#10935 · 1 comment ·
Maintainers usually reply within 1 day