[Bug][Producer] A send error or an out-of-order receipt for one producer closes the shared connection, reconnecting every producer and consumer on it

Open
#1,553 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@diegofernandes is already working on this.

Since Oct 27, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Start with pulsar/internal/connection.go at handleSendError and pulsar/producer_partition.go at partitionProducer.ReceivedSendReceipt; compare both paths with handleCloseProducer. The issue includes a proposed producer-only reconnect and reports three broker-free unit tests, but does not name their files. Add tests for both error paths and run ./pulsar/internal/...; done means only the affected producer reconnects while other listeners stay connected.

Written by the indexing model from the issue text.

Description

Expected behavior

A send error, or an unexpected send receipt, that concerns one producer makes that producer reconnect. The other producers and consumers sharing the connection stay connected.

That is what the client already does on CLOSE_PRODUCER (handleCloseProducer). It is also what the Java client does on a send error since apache/pulsar#23038:

default:
    // don't close this ctx, otherwise it will close all consumers and producers which use this ctx
    producer.connectionClosed(this, Optional.empty(), Optional.empty());
Actual behavior

Two code paths close the whole connection because of one producer. With MaxConnectionsPerBroker at its default of 1, every producer and consumer the client has on that broker shares that connection, so all of them reconnect at once:

  • producers re-send their pending messages, some of which the broker had already persisted, so those are written twice when deduplication is off;
  • acknowledgements in flight with AckWithResponse fail with "connection closed".

We hit it by deleting a topic with force=true while another client was still sending to it. In a load test, each of 3 such deletes closed about 920 producers and consumers at once. Together they wrote 2,619 messages twice and failed 266 acknowledgements, all on topics unrelated to the deleted ones.

Observed on v0.21.0. Both paths are unchanged on master (3bab464).

1. connection.handleSendError (pulsar/internal/connection.go#L918-L921)

default:
	// By default, for transient error, let the reconnection logic
	// to take place and re-establish the produce again
	c.Close()

The broker answers a send that meets a forced delete with PersistenceError ("ManagedLedger … is fenced", or "fenced" once a refused write has fenced the topic). Every error other than NotAllowedError and TopicTerminatedError takes this branch. The comment's intent is to re-establish the producer. Closing the connection does that, but for every listener on the connection.

Terminating the topic before deleting it does not avoid it. The first write the terminated ledger refuses fences the topic on the broker, and the other sends in flight then get PersistenceError: fenced.

2. partitionProducer.ReceivedSendReceipt (pulsar/producer_partition.go#L1426-L1431)

if pi.sequenceID < response.GetSequenceId() {
	// Force connection closing so that messages can be re-transmitted in a new connection
	p.log.Warnf("Received ack for %v on sequenceId %v - expected: %v, local < remote, closing connection",
		response.GetMessageId(), response.GetSequenceId(), pi.sequenceID)
	p._getConn().Close()
	return

After the broker closes a producer (CLOSE_PRODUCER, as a forced delete does), the producer reconnects on the same connection. The receipts of its earlier sends can then arrive ahead of its oldest pending message, and this closes the connection. In our runs it showed up once the first path no longer closed the connection (with the fix below, or when the topic was terminated before the delete). Otherwise the send error had already closed it. The Java client still closes the channel here (ProducerImpl.ackReceived). #650 raised this path in 2021, before it was narrowed to the local < remote case.

Steps to reproduce
  1. One client with DisableBatching: true, and broker deduplication off. Two topics, A and B, owned by the same broker.
  2. Send to A from 8 goroutines in a loop. On B, send in a loop and run a Shared consumer with AckWithResponse: true that acks everything it receives.
  3. From another client, delete A: DELETE /admin/v2/persistent/<tenant>/<namespace>/A?force=true.
  4. The client logs Received send error from server: [PersistenceError], then Connection was closed for every producer and consumer on the connection. B's consumer receives some messages twice, and acks in flight fail with "connection closed".

Measured on a local 3-broker cluster with that setup, 3 deletes of A per run:

client runs runs where B got duplicates or failed acks PersistenceError send errors out-of-order receipts
v0.21.0 10 5 (2–6 duplicates; once 6 failed acks) 41 0
v0.21.0 + the fix below 12 0 44 2
Proposed fix

In both places, reconnect only the producer, as handleCloseProducer already does:

// connection.handleSendError, default branch
producer, ok := c.deletePendingProducers(producerID)
if !ok {
	c.log.Warnf("Received send error for unknown producer %d: [%v] : [%s]",
		producerID, sendError.GetError(), sendError.GetMessage())
	return
}
producer.ConnectionClosed(nil)

// partitionProducer.ReceivedSendReceipt, local < remote
p._getConn().UnregisterListener(p.producerID)
p.ConnectionClosed(nil)

The producer leaves the connection's listeners, reconnects, and re-sends what it had pending. Every other producer and consumer stays connected. A producer whose topic was deleted fails to reconnect with TopicNotFound, closes, and fails its pending sends with ErrTopicNotfound, as it does today.

We have this change with three unit tests that need no broker. They fail on v0.21.0 and pass with the change, and ./pulsar/internal/... passes. We're happy to open a PR.

System configuration

Pulsar version: broker 5.0.0; pulsar-client-go v0.21.0 (same code on master at 3bab464)

Dominant language
Go
Stars
746
Forks
394
Avg merge
6d 13h
Merged PRs (30d)
7

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/pulsar-client-go

All issues in apache/pulsar-client-go

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.