Feature discussion - a performance metrics prioritized router that supports cluster.

Incentive

The main incentive is actually not about the performance of the system. It's to mitigate the situation when a portion of routees start to become irresponsive (not processing messages) due to some anomalies. In such a case, especially when a cluster aware router is involved, although other routees are still working perfectly fine, the router will continue forward the same portion of traffic to the problematic ones. Ideally we want the problematic routees to be avoided or de-prioritized, and just let the working routees taking more work.

Proposed Solution

A router that, like the (ScatterGatherFirstCompleted)[https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/routing/ScatterGatherFirstCompleted.scala], uses ask which requires the routee to reply a message. This way the router can keep track of the routees' response time and thus start to de-prioritize the slow ones. The relationship between the priority and performance is up for discussion, but I speculate that it should come with some randomization and be non-linear recipes.

Other alternatives

As @johanandren suggested, another solution would be implementing work pulling. We do have a work pulling library, kanaloa that we developed and are using. Making kanaloa cluster aware is going to solve our problem for us (not without significant overhaul though) but a router solution would be beneficial to other Akka users without kanaloa or implementing the pulling pattern.