The consumer does not consume all partitions fairly
#808 opened on 2020年6月18日
説明
Environment Information
- OS [CentOS Linux]:
- Node Version [12 and 14]:
- NPM Version [6.14.5]:
- node-rdkafka version [2.9.0]:
node-rdkafka Configuration Settings I use non-following mode consumer and each batch contains 2000 records, consumer configuration:
{
groupId: 'frg',
bootstrapServer: '10.120.1.20:9092',
topic: 'rdo-stocks-v1',
clientId: 'rdo-stock-quantity',
enableAutoCommit: true,
enableAutoOffsetStore: true,
socketKeepaliveEnable: 'true',
securityProtocol: 'SASL_PLAINTEXT',
saslMechanisms: 'SCRAM-SHA-512',
saslUsername: 'rdUser',
saslPassword: 'secret',
consumerMaxPollInterval: '200000',
consumerAutoOffsetReset: 'earliest',
consumerCommitInterval: 500,
consumerPartitionAssignmentStrategy: 'roundrobin',
debug: 'fetch',
consumerHeartbeatInternalMs: 10000,
consumerSessionTimeout: 20000,
consumerFetchMessageSizeByte: 1048576,
messageMaxBytes: 1548576,
consumerQueuedMaxMessagesKbytes: 1048576,
consumerMaxPollRecords: 2000,
consumerDefaultTimeout: 100
}
UV_THREADPOOL_SIZE is set to 30. The consumer run on in a docker container and each consumer runs on a different host (8 cores).
Additional context
The consumer consumes messages well, but as we can see in the picture below, each Kafka broker is the leader of two partitions and the consumer consumes only 1 partition:

Initially, the consumer commit messages manually, I changed it to auto-commit and i have the same result. #180 Note We have a Kafka cluster with 4 brokers, and each one has only 2 cores(to this consumer we add a sink and source connector), is that enough?
Any help is welcome
Thank's Samir