[Bug][Consumer] Throughput collapse and permanent deadlock on large payloads since v0.20.0

Open
#1,573 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@HrithikShet is already working on this.

Since Nov 20, 2026.

  • #1574 by @HrithikShet — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Start by reading commit 1796b8a / PR #1503, which removed default: return true from chCond.waitWithContext, then trace shrinkReceiverQueueSize() and the availablePermits arithmetic in the consumer implementation. Follow the reader path in readAtLeast() where r.buffer.Resize(totalFrameSize * 2) grows the buffer, and the clearQueueCh drain path that skips memLimit.ReleaseMemory(). Done means permits never go negative under downsizing, drained reader buffers are shrunk, and drained messages release their memory quota, verified with a large-payload consumer test under a bounded MemoryLimitBytes.

Written by the indexing model from the issue text.

Description

Expected behavior

When consuming large message payloads (e.g. 256 KB – 4 MB), the client should maintain continuous throughput, adaptively scale receiver queue size under memory pressure without stalling, and release TCP connection reader buffers when drained.

Actual behavior

Starting in v0.20.0 and v0.21.0 (following commit 1796b8a / PR #1503 which removed default: return true in chCond.waitWithContext), consuming large payloads causes consumers in production pods to freeze completely at 0 msgs/s until the pod is restarted.

Three interconnected root causes were identified:

  1. Flow-Control Permit Underflow Deadlock:
    Under memory pressure with EnableAutoScaledReceiverQueueSize, shrinkReceiverQueueSize() reduces receiver queue size (e.g. 100 -> 50) and calls:
    pc.availablePermits.add(newSize - oldSize) // e.g. -50
    
    Because availablePermits is an unbounded atomic integer, it drops below zero (e.g. -45). Even when all buffered messages are consumed, availablePermits remains negative and never reaches flowThreshold. The consumer permanently ceases requesting flow permits from the broker (CommandFlow), deadlocking the consumer until pod restart.
  2. Double-Allocation and Permanent Heap Buffer Retention in Connection Reader:
    In readAtLeast(), r.buffer.Resize(totalFrameSize * 2) unconditionally doubles frame capacity (allocating 8 MB for a 4 MB frame). When the buffer is drained (ReadableBytes() == 0), r.buffer is never shrunk back. Long-lived TCP connection objects permanently retain multi-megabyte buffers on the heap, triggering frequent GC stop-the-world pauses.
  3. Memory Limiter Quota Leak on Channel Drain:
    In clearQueueCh, messages drained from queueCh and messages on seek/reconnect are discarded without calling pc.client.memLimit.ReleaseMemory(), permanently leaking global memory quota.
Steps to reproduce
  1. Run a consumer against an Apache Pulsar broker with EnableAutoScaledReceiverQueueSize: true and bounded MemoryLimitBytes (e.g. 64 MB).
  2. Produce large payloads (>= 256 KB or 2 MB – 4 MB).
  3. Observe that when receiver queue downsizing is triggered under memory pressure, availablePermits underflows negative and the consumer stalls permanently at 0 msgs/s.
System configuration
  • Pulsar Client Go Version: v0.20.0, v0.21.0, and master
  • OS: Linux / macOS
Dominant language
Go
Stars
746
Forks
394
Avg merge
6d 13h
Merged PRs (30d)
7

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/pulsar-client-go

All issues in apache/pulsar-client-go

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.