[Bug][Consumer] Throughput collapse and permanent deadlock on large payloads since v0.20.0
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 32/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- backend, performance
Research direction
Start by reading commit 1796b8a / PR #1503, which removed default: return true from chCond.waitWithContext, then trace shrinkReceiverQueueSize() and the availablePermits arithmetic in the consumer implementation. Follow the reader path in readAtLeast() where r.buffer.Resize(totalFrameSize * 2) grows the buffer, and the clearQueueCh drain path that skips memLimit.ReleaseMemory(). Done means permits never go negative under downsizing, drained reader buffers are shrunk, and drained messages release their memory quota, verified with a large-payload consumer test under a bounded MemoryLimitBytes.
Written by the indexing model from the issue text.
Description
Expected behavior
When consuming large message payloads (e.g. 256 KB – 4 MB), the client should maintain continuous throughput, adaptively scale receiver queue size under memory pressure without stalling, and release TCP connection reader buffers when drained.
Actual behavior
Starting in v0.20.0 and v0.21.0 (following commit 1796b8a / PR #1503 which removed default: return true in chCond.waitWithContext), consuming large payloads causes consumers in production pods to freeze completely at 0 msgs/s until the pod is restarted.
Three interconnected root causes were identified:
- Flow-Control Permit Underflow Deadlock:
Under memory pressure withEnableAutoScaledReceiverQueueSize,shrinkReceiverQueueSize()reduces receiver queue size (e.g. 100 -> 50) and calls:
Becausepc.availablePermits.add(newSize - oldSize) // e.g. -50availablePermitsis an unbounded atomic integer, it drops below zero (e.g. -45). Even when all buffered messages are consumed,availablePermitsremains negative and never reachesflowThreshold. The consumer permanently ceases requesting flow permits from the broker (CommandFlow), deadlocking the consumer until pod restart. - Double-Allocation and Permanent Heap Buffer Retention in Connection Reader:
InreadAtLeast(),r.buffer.Resize(totalFrameSize * 2)unconditionally doubles frame capacity (allocating 8 MB for a 4 MB frame). When the buffer is drained (ReadableBytes() == 0),r.bufferis never shrunk back. Long-lived TCP connection objects permanently retain multi-megabyte buffers on the heap, triggering frequent GC stop-the-world pauses. - Memory Limiter Quota Leak on Channel Drain:
InclearQueueCh, messages drained fromqueueChandmessageson seek/reconnect are discarded without callingpc.client.memLimit.ReleaseMemory(), permanently leaking global memory quota.
Steps to reproduce
- Run a consumer against an Apache Pulsar broker with
EnableAutoScaledReceiverQueueSize: trueand boundedMemoryLimitBytes(e.g. 64 MB). - Produce large payloads (>= 256 KB or 2 MB – 4 MB).
- Observe that when receiver queue downsizing is triggered under memory pressure,
availablePermitsunderflows negative and the consumer stalls permanently at 0 msgs/s.
System configuration
- Pulsar Client Go Version: v0.20.0, v0.21.0, and master
- OS: Linux / macOS
- Dominant language
- Go
- Stars
- 746
- Forks
- 394
- Avg merge
- 6d 13h
- Merged PRs (30d)
- 7
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/pulsar-client-go
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPossibly taken @0z5a claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/pulsar-client-go#1575 ·
Maintainers usually reply within 2 days
-
[Bug] Concurrent transactional ACKs can crash in registerAckTopicPossibly taken @Ebispongebob claimed this 73 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/pulsar-client-go#1536 ·
Maintainers usually reply within 2 days
-
Nil pointer panic in `sendRequest.done()` during producer reconnection via `FailTimeoutMessages`Possibly taken @nodece claimed this 186 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/pulsar-client-go#1497 · 1 comment ·
Maintainers usually reply within 2 days
-
[Bug] Lookup retries use a fixed backoff that doubles to 60 s, so a reconnect during a broker outage can sleep through the failoverPossibly taken @EdwardShen125 claimed this 26 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 54/100
apache/pulsar-client-go#1554 ·
Maintainers usually reply within 2 days
-
[Bug][Producer] A send error or an out-of-order receipt for one producer closes the shared connection, reconnecting every producer and consumer on itPossibly taken @diegofernandes claimed this 26 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
apache/pulsar-client-go#1553 · 1 comment · 1 assignee ·
Maintainers usually reply within 2 days
All issues in apache/pulsar-client-go
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
seaweedfs/seaweed-block#443 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
version-fox/vfox#825 ·
Maintainers usually reply within 1 day
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodePossibly taken @Indian-boult claimed this today. Openbug queries
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ClickHouse/clickhouse-go#2155 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
transparency-dev/tessera#1286 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100