OPENSSL_internal:WRONG_VERSION_NUMBER excpetion when using MOP TLS Proxy

Start with MQTTProxyExchanger and MqttMessageUtils, especially MQTTProxyExchanger$ExchangerHandler.channelRead and the logged connection to port 8883. Reproduce with MOP Proxy 2.9.4.3 and Pulsar 2.9.4 while checking the TLS endpoint and protocol expectations. Done means identifying the cause of the handshake and CONNECT-header errors and verifying a successful publish/subscribe path.

Hi Streamnative team,

I'm using MOP Proxy v 2.9.4.3 with pulsar v2.9.4 , I'm getting below error when trying to pub/sub to a topic :

2023-02-16T06:52:38,092-0500 [mqtt-redirect-io-57-3] INFO io.streamnative.pulsar.handlers.mqtt.support.handler.AbstractAckHandler - current connection state : ESTABLISHED
2023-02-16T06:52:39,389-0500 [mqtt-redirect-io-57-3] INFO io.streamnative.pulsar.handlers.mqtt.proxy.MQTTProxyExchanger - connected to broker: dcwidavgds0267.epg.nam.gm.com:8883
2023-02-16T06:52:39,391-0500 [pulsar-io-4-10] WARN io.streamnative.pulsar.handlers.mqtt.MQTTCommonInboundHandler - An unexpected exception was caught while processing MQTT message. Closing Netty channel [id: 0x2a9f8118, L:/10.127.57.165:8883 ! R:/10.127.57.165:12354]. connection = null
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.needWrapAgain(ReferenceCountedOpenSslEngine.java:1343) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1360) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:221) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
... 19 more
2023-02-16T06:52:39,391-0500 [mqtt-redirect-io-57-3] ERROR io.streamnative.pulsar.handlers.mqtt.proxy.MQTTProxyExchanger - Exception was caught while processing MQTT broker message
java.lang.IllegalStateException: Illegal BIT 2 or 1 in fixed header of CONNECT message, must be 0, found 2
at io.streamnative.pulsar.handlers.mqtt.utils.MqttMessageUtils.checkState(MqttMessageUtils.java:45) ~[0WHqhBuH5S_IR4ygx_uSTw/:?]
at io.streamnative.pulsar.handlers.mqtt.proxy.MQTTProxyExchanger$ExchangerHandler.channelRead(MQTTProxyExchanger.java:97) ~[0WHqhBuH5S_IR4ygx_uSTw/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar