cachedHostConnectionPoolHttps keeps failing with "Connection was shutdown"

Akka http 10.1.1

We had a unplanned upgrade of our kubernetes cluster causing some services to get isolated for a while (dns resolving, hosts not reachable)

One of our connection pools ended up in a state where all requests kept failing with the below exception (no stack trace) This was only resolved after restarting the service

The docs mention this:

The returned [[akka.stream.scaladsl.Flow]] instances therefore remain valid throughout the lifetime of the application.

Our logs contain:

akka.http.impl.engine.client.pool.SlotState$BusyState$$anon$1: Connection was shutdown.  

...
INFO akka.actor.LocalActorRef - Message [akka.io.Dns$Resolved] from Actor[akka://rest-runtime/system/IO-DNS/inet-address/$a#1575798958] to Actor[akka://rest-runtime/system/IO-TCP/selectors/$a/4324#1643062019] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

This originates from https://github.com/akka/akka-http/blob/37b79e30bbd62508028fef63c1d83b67e41d8355/akka-http-core/src/main/scala/akka/http/impl/engine/client/pool/SlotState.scala#L132

private val HOST = "logging.googleapis.com"
private val poolClientFlow = Http().cachedHostConnectionPoolHttps[Promise[HttpResponse]](HOST)
  private val QueueSize = 10
  private val queueSource =
    Source.queue[(HttpRequest, Promise[HttpResponse])](QueueSize, OverflowStrategy.dropNew)
      .via(poolClientFlow)
      .toMat(Sink.foreach({
        case (Success(resp), p) => p.success(resp)
        case (Failure(e), p)    => p.failure(e)
      }))(Keep.left)
      .run()

Any hints would be appreciated, we have not been able to reproduce this issue yet.