Messages not being flushed from queue on subsequent runs

Abierto

#40 abierto el 14/3/2023 1 comentario 2 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad: 4/5
Tiempo estimado: 3-5 días
Aptitud para principiantes: 32/100

Tipo de issue: Error
Claridad: Necesita aclaración
Estado de actividad: Estancado
Stack tecnológico: aws, java
Área: backend, cloud, observability-sre

Línea de trabajo

Comienza con S3EventProcessor y sigue su ciclo de vida de transporte a través de GelfSenderThread y GelfTcpTransport, usando los logs de Lambda proporcionados para comparar la primera invocación con las posteriores. Revisa el comportamiento de la cola y del apagado en torno a TCP_QUEUE_SIZE y SHUTDOWN_FLUSH_RETRIES; se considera completado cuando los mensajes se entregan a Graylog en las ejecuciones posteriores sin perder el ciclo de vida del transporte.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug triaged

We are encountering an issue where log messages are not being sent to Graylog. When the lambda first starts, it's able to process one message but it fails for all subsequent runs without any apparent reason (or log message).

Runtime: Java 8 on Amazon Linux 2
Memory: 512 MB

I've tried tweaking the following but nothing seems to help

TCP_QUEUE_SIZE
SHUTDOWN_FLUSH_RETRIES

Logs

message
INIT_START Runtime Version: java:8.al2.v18	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:xxx
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (lambdainternal.CustomerClassLoader@37bba400)
START RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a Version: $LATEST
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
"2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-44-54-814351.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776303.922"" short_message=""[{'time': '2023-03-14T06:43:54.697Z', 'type': 'platform.start', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'version': '8'}}, {'time': '2023-03-14T06:43:54.697Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:43:54.697837+00:00""}
'}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.end', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22'}}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.report', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'metrics': {'durationMs': 331.874, 'billedDurationMs': 332, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:160 - Successfully flushed messages. Shutting down now.
2023-03-14 06:45:04  INFO  GelfTcpTransport:115 - Channel disconnected!
2023-03-14 06:45:04  DEBUG GelfSenderThread:123 - GelfSenderThread exiting!
2023-03-14 06:45:06  DEBUG PoolThreadCache:224 - Freed 1 thread-local buffer(s) from thread: gelfTcpTransport-1-1
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a
REPORT RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a	Duration: 6418.68 ms	Billed Duration: 6419 ms	Memory Size: 512 MB	Max Memory Used: 116 MB
START RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3 Version: $LATEST
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.buffer.AbstractByteBuf
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.util.Recycler
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.util.ReferenceCountUtil
"
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
"io.netty.util.ResourceLeakDetectorFactory
"
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-45-54-692438.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776356.731"" short_message=""[{'time': '2023-03-14T06:44:54.813Z', 'type': 'platform.start', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'version': '8'}}, {'time': '2023-03-14T06:44:54.828Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:44:54.814409+00:00""}
'}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.end', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997'}}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.report', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'metrics': {'durationMs': 284.761, 'billedDurationMs': 285, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 ERROR GelfSenderThread:175 - Failed to flush messages in [10] attempts. Shutting down anyway.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3
REPORT RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3	Duration: 1882.47 ms	Billed Duration: 1883 ms	Memory Size: 512 MB	Max Memory Used: 116 MB

Lenguaje dominante: Java
Estrellas: 13
Forks: 6
Métricas de merge de PR: Sin PR fusionados en 30 d

Preparar el entorno

Leer la guía de contribución

Primeros pasos

Lee el issue completo y luego la guía de contribución del proyecto.
Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
Haz un fork del repositorio y trabaja en una rama.
Abre un pull request que haga referencia al número del issue.

Más de Graylog2/graylog-s3-lambda

Safer - Compatible Updates to Fix Vulnerable Dependencies
Abierto
- ¿Algo no está bien?
Dificultad 5/5 Más de una semana Aptitud para principiantes 18/100

Graylog2/graylog-s3-lambda#44 · Abierto el 16/7/2025
Current release causes excessive timeouts
Abierto
- ¿Algo no está bien?
triaged

Dificultad 4/5 3-5 días Aptitud para principiantes 25/100

Graylog2/graylog-s3-lambda#38 · 20 comentarios · Abierto el 19/1/2022
Automate graylog-s3-lambda releases
Abierto
- ¿Algo no está bien?
triaged

Dificultad 4/5 3-5 días Aptitud para principiantes 35/100

Graylog2/graylog-s3-lambda#28 · 1 comentario · Abierto el 13/12/2021
Increased memory usage on the Amazon Corretto runtime
Abierto
- ¿Algo no está bien?
triaged

Dificultad 4/5 3-5 días Aptitud para principiantes 35/100

Graylog2/graylog-s3-lambda#23 · 3 comentarios · Abierto el 7/7/2021
Cloudflare IP fields are set as string instead of IP
Abierto
- ¿Algo no está bien?
feature triaged

Dificultad 3/5 1-2 días Aptitud para principiantes 42/100

Graylog2/graylog-s3-lambda#21 · 1 comentario · Abierto el 17/6/2021

Todos los issues de Graylog2/graylog-s3-lambda

Issues similares

AI chat SQL block buttons (Execute/Copy/Open in editor) silently do nothing on Linux
Abierto
- ¿Algo no está bien?
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100

dbeaver/dbeaver#42255 · Abierto el 29/9/2026

Los mantenedores suelen responder en 1 día
[BUGFIX]: Schedule 9 Contractual Work Report panels show the database record id (e.g. 3752, 3759) instead of the report number (1, 2) — legacy numbered them 1, 2, 3…
Abierto
- ¿Algo no está bien?
bug

Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100

bcgov/nr-ilcr#528 · Abierto el 29/9/2026

Los mantenedores suelen responder en 1 día
rust: in-process transport frees the connection callback state immediately when `copilot_runtime_connection_open` returns 0 (possible use-after-free)
Abierto
- ¿Algo no está bien?
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100

github/copilot-sdk#2793 · Abierto el 29/9/2026

Los mantenedores suelen responder en 1 día
Unify jpa4 into orm8
Abierto
- ¿Algo no está bien?
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100

hibernate/hibernate-migration-recipes#21 · Abierto el 29/9/2026
[ENHANCEMENT] Minesweeper entry on Wiki
Abierto
- ¿Algo no está bien?
enhancement good first issue

Dificultad 1/5 1-3 horas Aptitud para principiantes 88/100

Bram-Hub/LEGUP#1000 · 1 comentario · Abierto el 29/9/2026

Más issues de Java

Messages not being flushed from queue on subsequent runs

¿Algo no está bien?

Evaluación

Línea de trabajo

Descripción

Preparar el entorno

Primeros pasos

Más de Graylog2/graylog-s3-lambda

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

Issues similares

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

¿Algo no está bien?

Recibe los nuevos issues en tu correo