[Bug][Producer] A send error or an out-of-order receipt for one producer closes the shared connection, reconnecting every producer and consumer on it
Maintainers usually reply within 2 days
@diegofernandes is already working on this.
Since Oct 27, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- distributed-systems
Research direction
Start with pulsar/internal/connection.go at handleSendError and pulsar/producer_partition.go at partitionProducer.ReceivedSendReceipt; compare both paths with handleCloseProducer. The issue includes a proposed producer-only reconnect and reports three broker-free unit tests, but does not name their files. Add tests for both error paths and run ./pulsar/internal/...; done means only the affected producer reconnects while other listeners stay connected.
Written by the indexing model from the issue text.
Description
Expected behavior
A send error, or an unexpected send receipt, that concerns one producer makes that producer reconnect. The other producers and consumers sharing the connection stay connected.
That is what the client already does on CLOSE_PRODUCER (handleCloseProducer). It is also what the Java client does on a send error since apache/pulsar#23038:
default:
// don't close this ctx, otherwise it will close all consumers and producers which use this ctx
producer.connectionClosed(this, Optional.empty(), Optional.empty());
Actual behavior
Two code paths close the whole connection because of one producer. With MaxConnectionsPerBroker at its default of 1, every producer and consumer the client has on that broker shares that connection, so all of them reconnect at once:
- producers re-send their pending messages, some of which the broker had already persisted, so those are written twice when deduplication is off;
- acknowledgements in flight with
AckWithResponsefail with "connection closed".
We hit it by deleting a topic with force=true while another client was still sending to it. In a load test, each of 3 such deletes closed about 920 producers and consumers at once. Together they wrote 2,619 messages twice and failed 266 acknowledgements, all on topics unrelated to the deleted ones.
Observed on v0.21.0. Both paths are unchanged on master (3bab464).
1. connection.handleSendError (pulsar/internal/connection.go#L918-L921)
default:
// By default, for transient error, let the reconnection logic
// to take place and re-establish the produce again
c.Close()
The broker answers a send that meets a forced delete with PersistenceError ("ManagedLedger … is fenced", or "fenced" once a refused write has fenced the topic). Every error other than NotAllowedError and TopicTerminatedError takes this branch. The comment's intent is to re-establish the producer. Closing the connection does that, but for every listener on the connection.
Terminating the topic before deleting it does not avoid it. The first write the terminated ledger refuses fences the topic on the broker, and the other sends in flight then get PersistenceError: fenced.
2. partitionProducer.ReceivedSendReceipt (pulsar/producer_partition.go#L1426-L1431)
if pi.sequenceID < response.GetSequenceId() {
// Force connection closing so that messages can be re-transmitted in a new connection
p.log.Warnf("Received ack for %v on sequenceId %v - expected: %v, local < remote, closing connection",
response.GetMessageId(), response.GetSequenceId(), pi.sequenceID)
p._getConn().Close()
return
After the broker closes a producer (CLOSE_PRODUCER, as a forced delete does), the producer reconnects on the same connection. The receipts of its earlier sends can then arrive ahead of its oldest pending message, and this closes the connection. In our runs it showed up once the first path no longer closed the connection (with the fix below, or when the topic was terminated before the delete). Otherwise the send error had already closed it. The Java client still closes the channel here (ProducerImpl.ackReceived). #650 raised this path in 2021, before it was narrowed to the local < remote case.
Steps to reproduce
- One client with
DisableBatching: true, and broker deduplication off. Two topics, A and B, owned by the same broker. - Send to A from 8 goroutines in a loop. On B, send in a loop and run a
Sharedconsumer withAckWithResponse: truethat acks everything it receives. - From another client, delete A:
DELETE /admin/v2/persistent/<tenant>/<namespace>/A?force=true. - The client logs
Received send error from server: [PersistenceError], thenConnection was closedfor every producer and consumer on the connection. B's consumer receives some messages twice, and acks in flight fail with "connection closed".
Measured on a local 3-broker cluster with that setup, 3 deletes of A per run:
| client | runs | runs where B got duplicates or failed acks | PersistenceError send errors |
out-of-order receipts |
|---|---|---|---|---|
| v0.21.0 | 10 | 5 (2–6 duplicates; once 6 failed acks) | 41 | 0 |
| v0.21.0 + the fix below | 12 | 0 | 44 | 2 |
Proposed fix
In both places, reconnect only the producer, as handleCloseProducer already does:
// connection.handleSendError, default branch
producer, ok := c.deletePendingProducers(producerID)
if !ok {
c.log.Warnf("Received send error for unknown producer %d: [%v] : [%s]",
producerID, sendError.GetError(), sendError.GetMessage())
return
}
producer.ConnectionClosed(nil)
// partitionProducer.ReceivedSendReceipt, local < remote
p._getConn().UnregisterListener(p.producerID)
p.ConnectionClosed(nil)
The producer leaves the connection's listeners, reconnects, and re-sends what it had pending. Every other producer and consumer stays connected. A producer whose topic was deleted fails to reconnect with TopicNotFound, closes, and fails its pending sends with ErrTopicNotfound, as it does today.
We have this change with three unit tests that need no broker. They fail on v0.21.0 and pass with the change, and ./pulsar/internal/... passes. We're happy to open a PR.
System configuration
Pulsar version: broker 5.0.0; pulsar-client-go v0.21.0 (same code on master at 3bab464)
- Dominant language
- Go
- Stars
- 746
- Forks
- 394
- Avg merge
- 6d 13h
- Merged PRs (30d)
- 7
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/pulsar-client-go
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPossibly taken @0z5a claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/pulsar-client-go#1575 ·
Maintainers usually reply within 2 days
-
[Bug] Concurrent transactional ACKs can crash in registerAckTopicPossibly taken @Ebispongebob claimed this 73 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/pulsar-client-go#1536 ·
Maintainers usually reply within 2 days
-
Nil pointer panic in `sendRequest.done()` during producer reconnection via `FailTimeoutMessages`Possibly taken @nodece claimed this 186 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/pulsar-client-go#1497 · 1 comment ·
Maintainers usually reply within 2 days
-
[Bug][Consumer] Throughput collapse and permanent deadlock on large payloads since v0.20.0Possibly taken @HrithikShet claimed this 2 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 32/100
apache/pulsar-client-go#1573 ·
Maintainers usually reply within 2 days
-
[Bug] Lookup retries use a fixed backoff that doubles to 60 s, so a reconnect during a broker outage can sleep through the failoverPossibly taken @EdwardShen125 claimed this 26 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 54/100
apache/pulsar-client-go#1554 ·
Maintainers usually reply within 2 days
All issues in apache/pulsar-client-go
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
seaweedfs/seaweed-block#443 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
version-fox/vfox#825 ·
Maintainers usually reply within 1 day
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodePossibly taken @Indian-boult claimed this today. Openbug queries
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ClickHouse/clickhouse-go#2155 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
transparency-dev/tessera#1286 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100