KRaft: a new controller can bootstrap against itself and never join the quorum
Maintainers usually reply within 1 day
Assessment
This issue has not been assessed yet.
Description
Description
A controller that starts without metadata (a new pod, e.g. after scaling up a controller role group) finds the quorum leader through controller.quorum.bootstrap.servers. Outside Kerberos, that list has one entry per controller role group: the role group's headless Service. That Service sets publishNotReadyAddresses: true, so the name also resolves to the starting pod's own IP.
Kafka treats each entry in controller.quorum.bootstrap.servers as one bootstrap node and resolves its name when it connects. If the starting pod's lookup returns its own IP, the connection succeeds, so Kafka never tries another address. The pod asks itself for the leader, gets no answer, and repeats this forever.
This is a race in the quorum management added in #1010.
Symptoms
- The new controller stays in Raft state
unattached(epoch 0) and never becomes an observer. The quorum's replication status doesn't list it at all. - Its
quorum-managersidecar logsLocal Raft state is 'unattached', nothing to doindefinitely, so it never callsadd-controller. - The
kafkacontainer never becomes ready, and the StatefulSet rollout stalls. Because the StatefulSet usesOrderedReady, later replicas are not created either. - The pod's only connections on the controller port go to its own IP.
- Deleting the pod usually clears it: the replacement resolves another IP and joins normally.
- Dominant language
- Rust
- Stars
- 31
- Forks
- 9
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 14
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/kafka-operator
-
Use mTLS instead of Kerberos for KRaft inter-node trafficPossibly taken @razvan claimed this 1 day ago. Opentype/internal-improvement
stackabletech/kafka-operator#1066 · 1 assignee ·
Maintainers usually reply within 1 day
-
KRaft: adding a controller role group can bootstrap a second quorumPossibly taken @razvan claimed this 1 day ago. Opentype/bug
stackabletech/kafka-operator#1060 · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
stackabletech/kafka-operator#1059 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
stackabletech/kafka-operator#1058 ·
Maintainers usually reply within 1 day
-
type/bug
Difficulty 5/5 Over a week Newbie friendliness 35/100
stackabletech/kafka-operator#955 · 1 comment ·
Maintainers usually reply within 1 day
All issues in stackabletech/kafka-operator
Similar issues
-
[Bug]: opencode_go chat route drops reasoning_effort (even 'none'), while the same endpoint honors it via custom_llm_provider=openaiPossibly taken @arvindkumar-ship-it claimed this today. Openllm translation
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
BerriAI/litellm#51871 · 1 comment ·
Maintainers usually reply within 1 day
-
comp-formats
Difficulty 2/5 Half a day Newbie friendliness 64/100
ClickHouse/ClickHouse#131694 ·
Maintainers usually reply within 1 day
-
cli: `execCargoMetadata` decodes stdout per chunk, corrupting non-ASCII paths across pipe boundariesOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
C-Bug S-Needs-Triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
bevyengine/bevy#26558 ·
Maintainers usually reply within 2 days