Listeners do not re-attach post deletion without podrestart

Open
#445 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes, python

Research direction

Start by tracing how listener UIDs are assigned and how pod labels are matched to recreated listener services; the issue does not name specific files or tests. Reproduce the deletion and recreation scenario with a HiveCluster and a ClusterInternal listener, then check that the recreated service can reach the original ready pods without a pod restart.

Written by the indexing model from the issue text.

Description

type/bug
Affected Stackable version

all

Current and expected behavior

Current behaviour

Assume a HiveCluster using a listener class e.g. ClusterInternal. A HiveCluster get's created with a listener discovered by label and IP provisioned by a volumeMount at the STS level.

If a Listener service gets deleted e.g. by hand or by e.g. only updating listener-operator, a new listener service gets created, the products operator does not reconcile and thus although pods are ready it fails to be reached via the fresh listener service.

Cause

Whenever a listener is recreated, it comes with a new UID. Pod's are labeled with something like listener.stackable.tech/mnt.<uid> referencing the listener UID created for this pod. Those UIDs change when a listener restarts and the pod don't. This mismatch will cause the pod being unavailable through it's listener address while becoming healthy and available

Expected behaviour

Listeners should reattach to its original pod after restart.

Possible solution

Probably something like a label which doesn't contain the UID.

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

Dominant language
Python
Stars
9
Forks
3
Avg merge
1d 6h
Merged PRs (30d)
8

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stackabletech/listener-operator

All issues in stackabletech/listener-operator

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.