AWS Kinesis/Cloudwatch scheduler errors

El payload no identifica un archivo fuente ni una prueba. Reproduce el problema creando una entrada de AWS Kinesis/CloudWatch e inspecciona el punto de entrada KinesisConsumer junto con los logs de timeout de CloudWatchMetricsPublisher indicados. Se considera terminado cuando la entrada ya no produce los errores del servidor indicados ni el consumo elevado de memoria asociado, con cobertura de regresión cuando el código relevante lo permita.

Description

when creating an AWS Kinesis/Cloudwatch input, the input may shows the following server errors logs
this may be causing a high volume of memory consumption

Steps To Reproduce

1. create a AWS Kinesis/Cloudwatch input
2. observe server logs

Environment

• Graylog Version: 3.3.6
• Elasticsearch Version: 7.9.1
• MongoDB Version: 3.6
• Browser Version: firefox 81

2020-10-05 11:53:10,501 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STARTING
2020-10-05 11:53:10,510 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now RUNNING
2020-10-05 11:53:10,958 INFO : software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
2020-10-05 11:53:10,975 INFO : software.amazon.kinesis.coordinator.Scheduler - Initialization attempt 1
2020-10-05 11:53:10,975 INFO : software.amazon.kinesis.coordinator.Scheduler - Initializing LeaseCoordinator
2020-10-05 11:53:14,390 INFO : software.amazon.kinesis.coordinator.Scheduler - Syncing Kinesis shard info
2020-10-05 11:53:18,300 INFO : software.amazon.kinesis.coordinator.Scheduler - Starting LeaseCoordinator
2020-10-05 11:53:18,449 INFO : software.amazon.kinesis.coordinator.Scheduler - Initialization complete. Starting worker loop.
2020-10-05 11:53:41,496 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:54:41,611 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:55:11,684 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:55:12,686 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:55:12,686 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:55:16,704 WARN : software.amazon.kinesis.metrics.CloudWatchMetricsPublisher - Could not publish 14 datums to CloudWatch
java.util.concurrent.TimeoutException: null
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_191]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_191]
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.blockingExecute(CloudWatchMetricsPublisher.java:87) ~[amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.publishMetrics(CloudWatchMetricsPublisher.java:74) [amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.runOnce(CloudWatchPublisherRunnable.java:138) [amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.run(CloudWatchPublisherRunnable.java:84) [amazon-kinesis-client-2.2.10.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2020-10-05 11:55:41,747 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:11,809 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:13,810 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:56:13,810 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:56:28,235 WARN : software.amazon.kinesis.metrics.CloudWatchMetricsPublisher - Could not publish 4 datums to CloudWatch
java.util.concurrent.TimeoutException: null
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_191]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_191]
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.blockingExecute(CloudWatchMetricsPublisher.java:87) ~[amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.publishMetrics(CloudWatchMetricsPublisher.java:74) [amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.runOnce(CloudWatchPublisherRunnable.java:138) [amazon-kinesis-client-2.2.10.jar:?]
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.run(CloudWatchPublisherRunnable.java:84) [amazon-kinesis-client-2.2.10.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2020-10-05 11:56:41,871 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:53,373 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STOPPING
2020-10-05 11:56:53,373 INFO : org.graylog.integrations.aws.transports.KinesisConsumer - Waiting up to 20 seconds for Kinesis Consumer shutdown to complete.
2020-10-05 11:56:53,374 INFO : software.amazon.kinesis.coordinator.Scheduler - Worker shutdown requested.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.coordinator.Scheduler - All record processors have been shutdown successfully.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.coordinator.Scheduler - Starting worker's final shutdown.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Shutting down CWPublication thread.
2020-10-05 11:56:54,249 INFO : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - CWPublication thread finished.
2020-10-05 11:56:54,249 INFO : software.amazon.kinesis.coordinator.Scheduler - Worker loop is complete. Exiting from worker.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@5a342571, software.amazon.kinesis.metrics.MetricDatumWithKey@67c5fd31, software.amazon.kinesis.metrics.MetricDatumWithKey@6f661af1, software.amazon.kinesis.metrics.MetricDatumWithKey@cdbb7a51, software.amazon.kinesis.metrics.MetricDatumWithKey@1f63051] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@af256011, software.amazon.kinesis.metrics.MetricDatumWithKey@38d8b02e, software.amazon.kinesis.metrics.MetricDatumWithKey@e79e7291] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@d733be4e, software.amazon.kinesis.metrics.MetricDatumWithKey@d73c1ff1] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,808 INFO : software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - Worker graylog-node-a06f8344bfa040ceb90e6360f1deddcff126a8f5ca6e1d706652f4b2560fd070 has successfully stopped lease-tracking threads
2020-10-05 11:56:54,812 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STOPPED
2020-10-05 11:56:54,815 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now TERMINATED```