[Bug] Lookup retries use a fixed backoff that doubles to 60 s, so a reconnect during a broker outage can sleep through the failover
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 54/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
Research direction
Start with pulsar/internal/rpc_client.go, especially requestToHost around lines 121–147, and read pulsar/backoff/backoff.go to understand the current retry policy. Trace how NewRPCClient receives client options and inspect the existing backoff policy tests. Done means a client option can control request retry delays while nil preserves the current behavior, with a unit test that needs no broker.
Written by the indexing model from the issue text.
Description
Expected behavior
A lookup that starts while a broker is away succeeds soon after the broker answers again. The application can set that pace, as it can for reconnects with ProducerOptions.BackOffPolicyFunc and ConsumerOptions.BackOffPolicyFunc.
Actual behavior
Each reconnect attempt of a producer or consumer starts a topic lookup, and the lookup retries with its own backoff, which no option reaches. If the lookup starts while the broker is still away, its retries grow to steps of 6.4 s and then 12.8 s. When the topic gets a new owner, the lookup is often asleep in one of those steps. Or it runs out its OperationTimeout (30 s by default) and the reconnect starts over. The producer's own capped backoff only paces the attempts between lookups.
In a broker-kill test with many producers, each producer had a reconnect backoff capped at 2 s. One producer per run came back 14–15 s after its topic had a new owner, while every other producer came back within 2.6 s. Publishes to its topic stalled for that long.
Why. rpcClient.requestToHost (pulsar/internal/rpc_client.go#L121-L147) builds a new backoff for every request:
bo := backoff.NewDefaultBackoffWithInitialBackOff(100 * time.Millisecond)
DefaultBackoff doubles from 100 ms with up to 20 % jitter, capped at 60 s (pulsar/backoff/backoff.go#L57). Only OperationTimeout bounds the retries. With the default 30 s, the retries come about 0.1, 0.3, 0.7, 1.5, 3.1, 6.3, 12.7 and 25.5 s after the first attempt, or later with jitter.
requestToHost serves the binary lookup service's Lookup (through RequestToHost), and its partitioned-topic metadata, schema and namespace-topics requests (through RequestToAnyBroker). Neither a producer's or consumer's BackOffPolicyFunc nor any ClientOptions field reaches it.
Observed on v0.21.0. Unchanged on master (3bab464).
Steps to reproduce
- Put the broker behind a TCP proxy that can either refuse or forward connections, and point the client's service URL at the proxy.
- Make the proxy refuse every connection, and at the same moment create a producer, which starts a lookup.
- After 16 s, let connections through again. That is past the default backoff's seventh retry (12.7–15.2 s in, with jitter), so its next retry comes 25.5 s in or later.
- Measure how long after the proxy forwards again the producer is created.
| lookup backoff | runs | producer up after the broker answered |
|---|---|---|
| default (v0.21.0) | 3 | 10.2 s, 12.2 s, 13.7 s |
| a policy capped at 2 s (the fix below) | 3 | 0.20 s, 0.56 s, 0.86 s |
Proposed fix
A client option, passed through NewRPCClient and used between requestToHost's retries. Nil keeps today's behaviour:
// ClientOptions
// LookupBackoffPolicyFunc gives the backoff between the retries of a request
// the client may send to any broker of the service URL, a lookup or a topic's
// partition metadata, within OperationTimeout. Without it the delay doubles
// from 100ms to 60s, so a lookup that starts while a broker is away can
// sleep through the moment its topic has a new owner.
LookupBackoffPolicyFunc func() backoff.Policy
// rpcClient.requestToHost
bo := c.newRetryBackoff()
// newRetryBackoff is the backoff of one request's retries: the client's
// policy, else one doubling from 100ms.
func (c *rpcClient) newRetryBackoff() backoff.Policy {
if c.retryBackoff != nil {
return c.retryBackoff()
}
return backoff.NewDefaultBackoffWithInitialBackOff(100 * time.Millisecond)
}
It mirrors the producer's and consumer's BackOffPolicyFunc. Those already let an application shorten its reconnect backoff, but today the lookup inside each reconnect attempt can undo that.
We have this change with a unit test that needs no broker: a connection pool that refuses every connection, with a counting policy against the default. We're happy to open a PR. We're also happy to change the shape if you prefer, for example a maximum delay instead of a policy func, or a floor on the delay.
Related: #1270 (the reconnect backoff against the send timeout) and #454 (a configurable backoff, closed over the risk of clients that retry too fast; here the default keeps today's pacing).
System configuration
Pulsar version: broker 5.0.0; pulsar-client-go v0.21.0 (same code on master at 3bab464)
- Dominant language
- Go
- Stars
- 746
- Forks
- 394
- Avg merge
- 6d 13h
- Merged PRs (30d)
- 7
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/pulsar-client-go
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPossibly taken @0z5a claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/pulsar-client-go#1575 ·
Maintainers usually reply within 2 days
-
[Bug] Concurrent transactional ACKs can crash in registerAckTopicPossibly taken @Ebispongebob claimed this 73 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/pulsar-client-go#1536 ·
Maintainers usually reply within 2 days
-
Nil pointer panic in `sendRequest.done()` during producer reconnection via `FailTimeoutMessages`Possibly taken @nodece claimed this 186 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/pulsar-client-go#1497 · 1 comment ·
Maintainers usually reply within 2 days
-
[Bug][Consumer] Throughput collapse and permanent deadlock on large payloads since v0.20.0Possibly taken @HrithikShet claimed this 2 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 32/100
apache/pulsar-client-go#1573 ·
Maintainers usually reply within 2 days
-
[Bug][Producer] A send error or an out-of-order receipt for one producer closes the shared connection, reconnecting every producer and consumer on itPossibly taken @diegofernandes claimed this 26 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
apache/pulsar-client-go#1553 · 1 comment · 1 assignee ·
Maintainers usually reply within 2 days
All issues in apache/pulsar-client-go
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
seaweedfs/seaweed-block#443 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
version-fox/vfox#825 ·
Maintainers usually reply within 1 day
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodePossibly taken @Indian-boult claimed this today. Openbug queries
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ClickHouse/clickhouse-go#2155 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
transparency-dev/tessera#1286 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100