Helix ZkClient times out after successful ZooKeeper SASL DIGEST-MD5 authentication

Open
  • Is something wrong?
#3,184 0 comments 0 reactions 0 assignees View on GitHub

@adityaanikam is already working on this.

Since Sep 29, 2026.

  • #3185 by @adityaanikam — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java
Domain
authentication, distributed-systems, testing

Research direction

Start at ZkClient.waitUntilConnected and the isKerberosAuthEnabled check described in the issue. Reproduce the controller or participant timeout with the provided DIGEST-MD5 JAAS configuration, then add the requested regression test covering successful authentication while preserving Kerberos behavior. Done means the authenticated session is treated as connected and startup succeeds.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

When a ZooKeeper server requires SASL DIGEST-MD5 authentication, an Apache Helix controller and participant establish ZooKeeper sessions and receive SaslAuthenticated, but Helix subsequently reports that its ZooKeeper client is not connected and times out.

I reproduced the controller failure with unmodified Helix 1.4.3 and 2.0.1, and the participant failure with Helix 1.4.3. The same DIGEST-MD5 credentials work for a ZooKeeper client and for bootstrapping the Helix cluster.

To Reproduce
  1. Start ZooKeeper with org.apache.zookeeper.server.auth.SASLAuthenticationProvider and sessionRequireClientSASLAuth=true.
  2. Configure the Helix JVM with a JAAS Client entry using org.apache.zookeeper.server.auth.DigestLoginModule and set -Djava.security.auth.login.config=<JAAS file> and -Dzookeeper.sasl.client=true.
  3. Bootstrap a Helix cluster using the same ZooKeeper client configuration
  4. Start org.apache.helix.controller.HelixControllerMain with that cluster, or connect a participant through HelixManagerFactory.getZKHelixManager(..., InstanceType.PARTICIPANT, ...).

The controller's ZooKeeper events reach SyncConnected and then SaslAuthenticated, followed by:

org.apache.helix.zookeeper.zkclient.exception.ZkTimeoutException:
Waiting to be connected to ZK server has timed out.

The Helix 1.4.3 participant shows the same state transition, then:

zkClient is not connected, wait for 10000ms.
zkClient is not connected after waiting 10000ms.
org.apache.helix.HelixException: HelixManager is not connected within retry timeout
Expected behavior

Helix should treat a successful SASL DIGEST-MD5 authentication as a connected session and allow the controller and participant to start.

Additional context

In Helix 2.0.1, ZkClient.waitUntilConnected expects SaslAuthenticated only when isKerberosAuthEnabled() finds a JAAS Krb5LoginModule. A DigestLoginModule client instead waits for SyncConnected. After successful DIGEST-MD5 authentication, the current state becomes SaslAuthenticated, and the exact-state wait can time out.

As a local proof of concept, extending that JAAS module check to include DigestLoginModule allowed the controller to authenticate and become leader. This was only a diagnostic class override, not a proposed production patch. A fix should include a regression test for DIGEST-MD5 and preserve Kerberos behavior.

This appears related to #3101 (Kerberos connection timeout). The Kerberos fix does not cover DIGEST-MD5.

Dominant language
Java
Stars
505
Forks
258
Avg merge
4d 20h
Merged PRs (30d)
3

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/helix

All issues in apache/helix

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.