AWS Kinesis/CloudWatch error is filling up the filesystem

まず、PrefetchRecordsPublisher を呼び出している Kinesis 統合コードを特定し、この例外がどのようにログ出力され、再試行されるかを確認します。Issue で説明されているログの繰り返しを再現または観察し、その後、エラーが無限にループしたりファイルシステムをいっぱいにしたりしなくなったことを確認します。

For the second time in 2 weeks i had the following error in my graylog-server log file:

ERROR [PrefetchRecordsPublisher] shardId-000000000000 : Exception thrown while fetching records from Kinesis software.amazon.awssdk.core.exception.SdkClientException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[?:?]
        at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[?:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125) ~[?:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:263) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:140) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:501) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:277) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:262) ~[?:?]
        ... 11 more
Caused by: java.nio.channels.ClosedChannelException
        at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.findCtx(Http2StreamChannelBootstrap.java:156) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:120) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:110) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$acquireClaimedStream$1(MultiplexedChannelRecord.java:89) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:139) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireClaimedStream(MultiplexedChannelRecord.java:82) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireStream(MultiplexedChannelRecord.java:75) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquireStreamOnInitializedConnection(Http2MultiplexedChannelPool.java:221) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquire(Http2MultiplexedChannelPool.java:115) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire0(BetterFixedChannelPool.java:164) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire(BetterFixedChannelPool.java:138) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.acquire0(HttpOrHttp2ChannelPool.java:80) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.lambda$acquire$0(HttpOrHttp2ChannelPool.java:69) ~[?:?]
        at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.lambda$doInEventLoop$3(NettyUtils.java:143) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[?:?]
        at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73) ~[?:?]
        ... 6 more

Expected Behavior

Graylog should handle the exception and not repeat the error indefinitely

Current Behavior

This error is logged every x msec until the filesystem is full (~ 1GB every 10 minutes)

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Ec2 instance inside a vpc using nat gateways and the default kinesis endpoint.

• Graylog Version: 3.2.4
• Elasticsearch Version: 6.8.8
• MongoDB Version: 4.0.14
• Operating System: Centos 7.7
• Browser version: Firefox