Socket-mode application become a zombie when unhandled exception occurs in single-threaded thread pool

In our application, we use Bolt Socket-mode SDK for the bot. And we've found some strange behavior. Sometimes (looks like it doesn't depend on anything) app becomes a "zombie", meaning that it continues to work but the bot isn't in a working state. For example, let's say for the message with text ping it will respond with a pong message, and in that "non-working" state bot won't react to the ping message. But at the same time app isn't "dead" or crashed, we can successfully "stop" SocketModeApp later. According to our observations, this always occurs after javax.websocket.DeploymentException

(we use SocketModeClientTyrusImpl)

I suspect (but still can be wrong 😅) this is due to the facts that:

• we have a single-threaded ThreadPool-s for com.slack.api.socket_mode.SocketModeClient#initializeSessionMonitorExecutor and com.slack.api.socket_mode.SocketModeClient#initializeMessageProcessorExecutor. Both create TP-s by Executors.newSingleThreadScheduledExecutor
• here are two exceptions logs that show unhandled exception occurrences:

  • [2021-08-27T16:43:47,600Z](socket-mode-session-monitor-worker-36)([]) trace_id= ERROR - SocketModeClient - Failed to reconnect to Socket Mode server: javax.websocket.DeploymentException: Handshake error.
    ! org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 408.
    ! at org.glassfish.tyrus.client.TyrusClientEngine.processResponse(TyrusClientEngine.java:299)
    ! at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleHandshake(GrizzlyClientFilter.java:322)
    ! at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleRead(GrizzlyClientFilter.java:291)
    ! at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    ! at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
    ! at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
    ! at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
    ! at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
    ! at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    ! at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
    ! at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    ! at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    ! at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    ! at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    ! at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
    ! at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
    ! ... 1 common frames omitted
    ! Causing: javax.websocket.DeploymentException: Handshake error.
    ! at org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:658)
    ! at org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:696)
    ! at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    ! at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ! at org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:849)
    ! at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
    ! at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:493)
    ! at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
    ! at com.slack.api.socket_mode.impl.SocketModeClientTyrusImpl.connect(SocketModeClientTyrusImpl.java:170)
    ! ... 9 common frames omitted
    ! Causing: java.io.IOException: javax.websocket.DeploymentException: Handshake error.
    ! at com.slack.api.socket_mode.impl.SocketModeClientTyrusImpl.connect(SocketModeClientTyrusImpl.java:173)
    ! at com.slack.api.socket_mode.SocketModeClient.connectToNewEndpoint(SocketModeClient.java:67)
    ! at com.slack.api.socket_mode.impl.SocketModeClientTyrusImpl.maintainCurrentSession(SocketModeClientTyrusImpl.java:137)
    ! at com.slack.api.socket_mode.SocketModeClient.lambda$initializeSessionMonitorExecutor$1(SocketModeClient.java:180)
    ! at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    ! at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    ! at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    ! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    ! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ! at java.base/java.lang.Thread.run(Thread.java:834)
    

  • [2021-08-27T17:06:35,818Z](socket-mode-message-processor-worker-37)([]) trace_id= ERROR - SocketModeClient - Failed to reconnect to Socket Mode server: javax.websocket.DeploymentException: Handshake response not received.
    ! javax.websocket.DeploymentException: Handshake response not received.
    ! at org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:678)
    ! at org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:696)
    ! at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    ! at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ! at org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:849)
    ! at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
    ! at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:493)
    ! at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
    ! at com.slack.api.socket_mode.impl.SocketModeClientTyrusImpl.connect(SocketModeClientTyrusImpl.java:170)
    ! ... 9 common frames omitted
    ! Causing: java.io.IOException: javax.websocket.DeploymentException: Handshake response not received.
    ! at com.slack.api.socket_mode.impl.SocketModeClientTyrusImpl.connect(SocketModeClientTyrusImpl.java:173)
    ! at com.slack.api.socket_mode.SocketModeClient.connectToNewEndpoint(SocketModeClient.java:67)
    ! at com.slack.api.socket_mode.SocketModeClient.processMessage(SocketModeClient.java:350)
    ! at com.slack.api.socket_mode.SocketModeClient.lambda$initializeMessageProcessorExecutor$0(SocketModeClient.java:157)
    ! at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    ! at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    ! at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    ! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    ! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ! at java.base/java.lang.Thread.run(Thread.java:834)
    

• after an unhandled exception occurred it kills one thread in TP and we lose one important functionality
  • recreate a new session for the first presented log case
  • consume\process messages for the second presented log case

The Slack SDK version

[INFO] +- com.slack.api:bolt:jar:1.8.1:compile
[INFO] |  +- com.slack.api:slack-api-model:jar:1.8.1:compile
[INFO] |  +- com.slack.api:slack-api-client:jar:1.8.1:compile
[INFO] |  \- com.slack.api:slack-app-backend:jar:1.8.1:compile
[INFO] +- com.slack.api:bolt-socket-mode:jar:1.8.1:compile
[INFO] |  +- com.slack.api:bolt:jar:1.8.1:runtime
[INFO] |  |  +- com.slack.api:slack-api-model:jar:1.8.1:runtime
[INFO] |  |  +- com.slack.api:slack-api-client:jar:1.8.1:runtime
[INFO] |  |  \- com.slack.api:slack-app-backend:jar:1.8.1:runtime
[INFO] |  +- com.slack.api:bolt-socket-mode:jar:1.8.1:runtime
[INFO]    +- com.slack.api:bolt:jar:1.8.1:compile
[INFO]    |  +- com.slack.api:slack-api-model:jar:1.8.1:compile
[INFO]    |  +- com.slack.api:slack-api-client:jar:1.8.1:compile
[INFO]    |  \- com.slack.api:slack-app-backend:jar:1.8.1:compile
[INFO]    +- com.slack.api:bolt-socket-mode:jar:1.8.1:compile
[INFO] |  |  +- com.slack.api:bolt:jar:1.8.1:runtime
[INFO] |  |  |  +- com.slack.api:slack-api-model:jar:1.8.1:runtime
[INFO] |  |  |  +- com.slack.api:slack-api-client:jar:1.8.1:runtime
[INFO] |  |  |  \- com.slack.api:slack-app-backend:jar:1.8.1:runtime
[INFO] |  |  +- com.slack.api:bolt-socket-mode:jar:1.8.1:runtime

OS info

Inside the openjdk11:jdk-11.0.7_10-alpine-slim

Steps to reproduce:

• can't say for sure. It just happens after javax.websocket.DeploymentExceptions. Please see my assumptions and presented logs.

Expected result:

The app is "usable", able to react to events in a common way after occurred javax.websocket.DeploymentExceptions

Actual result:

The app is "unusable". Like a zombie, unable to react to events in a common way after occurred javax.websocket.DeploymentExceptions, but isn't totally crashed.