linkedin/cruise-control

Redundant set/unset throttling during the rebalance

Open

#1972 aperta il 2 mar 2023

Vedi su GitHub
 (6 commenti) (0 reazioni) (1 assegnatario)Java (649 fork)auto 404
functionalitygood first issue

Metriche repository

Star
 (3036 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Today, Cruise Control are setting/resetting replication rate throttles on a per-task base. E.g. it set throttling before a task executes:

https://github.com/linkedin/cruise-control/blob/155bf3c2f7fe9e7049173403b61b02d9fee8379f/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/Executor.java#L1415

Then clear the throttling for the broker after task finished and if there is no ongoing-task on the same broker: https://github.com/linkedin/cruise-control/blob/155bf3c2f7fe9e7049173403b61b02d9fee8379f/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/Executor.java#L1437

Note that each set/clear throttle will be an individual Kafka admin request, will may take a couple of seconds.

The problem is: there can be a lot of tasks in the backlog that is involving the same broker. It doesn't make senses to set/clear the throttling for the same broker again and again on every task execution batch. From local unit test running, it takes ~20 seconds for adminClient to handle each set/clear throttle request.

If we set the throttling for broker at the beginning of the execution, and only clean up the throttling after all tasks finished execution, that should speed up the rebalance.

Note: this is not the concurrency throttling, but the replication bytes throttling. This throttle is set by Cruise Control sending request to kafka brokers to cap the max replication bytes rate.

Guida contributor