[Bug] Lookup retries use a fixed backoff that doubles to 60 s, so a reconnect during a broker outage can sleep through the failover

Open
#1,554 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@EdwardShen125 is already working on this.

Since Oct 27, 2026.

  • #1560 by @EdwardShen125 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Start with pulsar/internal/rpc_client.go, especially requestToHost around lines 121–147, and read pulsar/backoff/backoff.go to understand the current retry policy. Trace how NewRPCClient receives client options and inspect the existing backoff policy tests. Done means a client option can control request retry delays while nil preserves the current behavior, with a unit test that needs no broker.

Written by the indexing model from the issue text.

Description

Expected behavior

A lookup that starts while a broker is away succeeds soon after the broker answers again. The application can set that pace, as it can for reconnects with ProducerOptions.BackOffPolicyFunc and ConsumerOptions.BackOffPolicyFunc.

Actual behavior

Each reconnect attempt of a producer or consumer starts a topic lookup, and the lookup retries with its own backoff, which no option reaches. If the lookup starts while the broker is still away, its retries grow to steps of 6.4 s and then 12.8 s. When the topic gets a new owner, the lookup is often asleep in one of those steps. Or it runs out its OperationTimeout (30 s by default) and the reconnect starts over. The producer's own capped backoff only paces the attempts between lookups.

In a broker-kill test with many producers, each producer had a reconnect backoff capped at 2 s. One producer per run came back 14–15 s after its topic had a new owner, while every other producer came back within 2.6 s. Publishes to its topic stalled for that long.

Why. rpcClient.requestToHost (pulsar/internal/rpc_client.go#L121-L147) builds a new backoff for every request:

bo := backoff.NewDefaultBackoffWithInitialBackOff(100 * time.Millisecond)

DefaultBackoff doubles from 100 ms with up to 20 % jitter, capped at 60 s (pulsar/backoff/backoff.go#L57). Only OperationTimeout bounds the retries. With the default 30 s, the retries come about 0.1, 0.3, 0.7, 1.5, 3.1, 6.3, 12.7 and 25.5 s after the first attempt, or later with jitter.

requestToHost serves the binary lookup service's Lookup (through RequestToHost), and its partitioned-topic metadata, schema and namespace-topics requests (through RequestToAnyBroker). Neither a producer's or consumer's BackOffPolicyFunc nor any ClientOptions field reaches it.

Observed on v0.21.0. Unchanged on master (3bab464).

Steps to reproduce
  1. Put the broker behind a TCP proxy that can either refuse or forward connections, and point the client's service URL at the proxy.
  2. Make the proxy refuse every connection, and at the same moment create a producer, which starts a lookup.
  3. After 16 s, let connections through again. That is past the default backoff's seventh retry (12.7–15.2 s in, with jitter), so its next retry comes 25.5 s in or later.
  4. Measure how long after the proxy forwards again the producer is created.
lookup backoff runs producer up after the broker answered
default (v0.21.0) 3 10.2 s, 12.2 s, 13.7 s
a policy capped at 2 s (the fix below) 3 0.20 s, 0.56 s, 0.86 s
Proposed fix

A client option, passed through NewRPCClient and used between requestToHost's retries. Nil keeps today's behaviour:

// ClientOptions
	// LookupBackoffPolicyFunc gives the backoff between the retries of a request
	// the client may send to any broker of the service URL, a lookup or a topic's
	// partition metadata, within OperationTimeout. Without it the delay doubles
	// from 100ms to 60s, so a lookup that starts while a broker is away can
	// sleep through the moment its topic has a new owner.
	LookupBackoffPolicyFunc func() backoff.Policy

// rpcClient.requestToHost
	bo := c.newRetryBackoff()

// newRetryBackoff is the backoff of one request's retries: the client's
// policy, else one doubling from 100ms.
func (c *rpcClient) newRetryBackoff() backoff.Policy {
	if c.retryBackoff != nil {
		return c.retryBackoff()
	}
	return backoff.NewDefaultBackoffWithInitialBackOff(100 * time.Millisecond)
}

It mirrors the producer's and consumer's BackOffPolicyFunc. Those already let an application shorten its reconnect backoff, but today the lookup inside each reconnect attempt can undo that.

We have this change with a unit test that needs no broker: a connection pool that refuses every connection, with a counting policy against the default. We're happy to open a PR. We're also happy to change the shape if you prefer, for example a maximum delay instead of a policy func, or a floor on the delay.

Related: #1270 (the reconnect backoff against the send timeout) and #454 (a configurable backoff, closed over the risk of clients that retry too fast; here the default keeps today's pacing).

System configuration

Pulsar version: broker 5.0.0; pulsar-client-go v0.21.0 (same code on master at 3bab464)

Dominant language
Go
Stars
746
Forks
394
Avg merge
6d 13h
Merged PRs (30d)
7

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/pulsar-client-go

All issues in apache/pulsar-client-go

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.