Blizzard/node-rdkafka

The consumer does not consume all partitions fairly

Open

#808 opened on 2020年6月18日

GitHub で見る
 (2 comments) (0 reactions) (0 assignees)JavaScript (1,979 stars) (385 forks)batch import
help wanted

説明

Environment Information

  • OS [CentOS Linux]:
  • Node Version [12 and 14]:
  • NPM Version [6.14.5]:
  • node-rdkafka version [2.9.0]:

node-rdkafka Configuration Settings I use non-following mode consumer and each batch contains 2000 records, consumer configuration:

{
  groupId: 'frg',
  bootstrapServer: '10.120.1.20:9092',
  topic: 'rdo-stocks-v1',
  clientId: 'rdo-stock-quantity',
  enableAutoCommit: true,
  enableAutoOffsetStore: true,
  socketKeepaliveEnable: 'true',
  securityProtocol: 'SASL_PLAINTEXT',
  saslMechanisms: 'SCRAM-SHA-512',
  saslUsername: 'rdUser',
  saslPassword: 'secret',
  consumerMaxPollInterval: '200000',
  consumerAutoOffsetReset: 'earliest',
  consumerCommitInterval: 500,
  consumerPartitionAssignmentStrategy: 'roundrobin',
  debug: 'fetch',
  consumerHeartbeatInternalMs: 10000,
  consumerSessionTimeout: 20000,
  consumerFetchMessageSizeByte: 1048576,
  messageMaxBytes: 1548576,
  consumerQueuedMaxMessagesKbytes: 1048576,
  consumerMaxPollRecords: 2000,
  consumerDefaultTimeout: 100
}

UV_THREADPOOL_SIZE is set to 30. The consumer run on in a docker container and each consumer runs on a different host (8 cores).

Additional context

The consumer consumes messages well, but as we can see in the picture below, each Kafka broker is the leader of two partitions and the consumer consumes only 1 partition:

image

Initially, the consumer commit messages manually, I changed it to auto-commit and i have the same result. #180 Note We have a Kafka cluster with 4 brokers, and each one has only 2 cores(to this consumer we add a sink and source connector), is that enough?

Any help is welcome

Thank's Samir

コントリビューターガイド