K8S: Truncating lease name causes uniqueness issues

Open
  • Is something wrong?
#702 6 comments 14 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
kubernetes, scala
Domain
distributed-systems, infrastructure

Research direction

Start in lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/AbstractKubernetesLease.scala, especially the lease-name normalization around the referenced line. Check how the CRD and Native implementations use this shared name handling and look for related tests. Done means long lease names remain distinct while satisfying Kubernetes resource-name limits, with the behavior covered by tests.

Written by the indexing model from the issue text.

Description

bug

Hello everyone.
First of all thank you for the great work maintaining this project !

After using the K8S lease in production environments, it has come to my attention that an issue arises when using singleton lease with long actor system names. This also can affect other leases but is less visible.

Issue

In pekko, the singleton lease name is configured as:

private val singletonLeaseName = s"${context.system.name}-singleton-${self.path}"

In the AbstractKubernetesLease implementation, this name above is truncated to 63 characters to avoid the 64 character limit of K8S API resource names.

The issue here, is that if the actor system name is long (anonymous actor system in Kubernetes for example, that retrieves the deployment name, which can be very long), it can very easily exceeds 64 characters.

Here is an example leaseName computed by Pekko:

test-a05b99e1-3377-464b-95e3-ef17b07027b7-singleton-pekko://test-a05b99e1-3377-464b-95e3-ef17b07027b7/path/to/actor

After treatment by the AbstractKubernetesLease, the computed lease name is

test-a05b99e1-3377-464b-95e3-ef17b07027b7-singleton-pekkoans-a0

From there, the problem is pretty obvious, each singleton actor tries to access the same lease resource !

This affects both K8S Lease implementations, the CRD and the Native one, as it is defined in the AbstractKubernetesLease that is the base for both implementations.

Proposed solution

From my perspective, I think the normalization that takes place in the AbstractKubernetesLease should not trim, as this can hinder unicity.

A possible solution would be to hash the input leaseName, thus reducing length but conserving unicity.

An extension to the resources metadata in form of labels would allow to keep the full lease name for easy audit of the cluster state.

If this solution meets your requirements, I can happily open a PR.

Thank you for your time !

Dominant language
Scala
Stars
34
Forks
20
Avg merge
19h 10m
Merged PRs (30d)
31

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/pekko-management

All issues in apache/pekko-management

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.