MoP Proxy seems redundant, introduces unnecessary complexity

Start by tracing the MoP proxy's client and broker connection lifecycle, including the keep-alive behavior described in the issue. Then compare the proposed local-topic lookup and PulsarClient forwarding with the existing Pulsar Proxy approach; done would require an agreed architecture and implementation plan rather than another isolated proxy fix.

Is your enhancement request related to a problem? Please describe.

In several attempts to get the MoP proxy to work properly we run across constant stability problems, and bugs related to the MoP proxy. The current design is that the MoP broker only serves the subset of topics that Pulsar broker serves, and the MoP proxy handles routing connections to the designated broker. This gets complicated by the fact that the proxy does a lot of mediation, but ultimately has to handle the lifecycle of n number of MQTT connections to each broker. The current keep alive mechanism is flawed in the fact that keep alive from the client may never happen if the client is communicating frequently enough, and that means that the keep alive to certain brokers might not be maintained. Introducing keep alive between client and proxy, and proxy and each broker isn't too difficult, but there this only increases the complexity of lifecycle management of the proxy and broker connections in conjunction with the client and proxy connection.

Describe the solution you'd like

Omit the need for a MoP proxy at all. Terminate MQTT as soon as possible and translate into Pulsar on the same broker the connection from the client is made. Instead of having each MoP broker handle the subset of topics on that broker, instead have the MoP broker do a lookup, if it resolves the lookup is local then handle the operation similar to what's being done now, if the lookup resolves to a different broker then create a PulsarClient to that broker to handle the operation. Alternatively a lot of work can possibly be saved but just having a PulsarClient which is already connected to all brokers that you forward operations through and rely on it's own mechanism to route to the correct broker.

I get the concern of not wanting to handle the routing on the Pulsar broker, as Pulsar has obviously made the decision not to do this themselves, and Pulsar itself has a Pulsar Proxy to do something very similar. The reality is that the MoP proxy already runs on the broker, and rather than using Pulsar to forward the requests, MQTT is forwarded as MQTT messages, which creates a lot of complicated problems, especially because the way that the MoP proxy forwards requests is not implemented in a way where the MoP proxy is a proper MQTT client. The reality is forwarding MQTT the way the proxy does takes special consideration, because that's not really the way MoP was designed.

It really seems like the benefits of a MoP proxy are almost entirely lost, but the complexity remains. I know there is a proposal to move the proxy onto the Pulsar Proxy, but there's not really a current effort to do that. Even if the effort was made, the problem of complexity persists. It seems that moving to the Pulsar Proxy finally achieves the intended design, but as it stands the complexity of having the MoP proxy doesn't seem to add any benefit in any regard I can think of. If load balance is an issue, I'd simply recommend informing the users to run MoP behind a load balancer.

Describe alternatives you've considered
Continuously patching issues in the MoP proxy.