Messages not being flushed from queue on subsequent runs

未关闭

#40 创建于 2023年3月14日 1 条评论 2 个 reaction 已指派 0 人在 GitHub 查看

还没有人认领这个 Issue。

评估

难度: 4/5
预计耗时: 3-5 天
新手友好度: 32/100

Issue 类型: 缺陷
描述清晰度: 需要澄清
活跃度: 停滞
技术栈: aws, java
领域: backend, cloud, observability-sre

调研方向

从 S3EventProcessor 开始，沿着其传输生命周期跟踪到 GelfSenderThread 和 GelfTcpTransport，并使用提供的 Lambda 日志比较第一次调用与后续调用。检查 TCP_QUEUE_SIZE 和 SHUTDOWN_FLUSH_RETRIES 相关的队列与关闭行为；当消息在后续运行中送达 Graylog，且传输生命周期未丢失时，即表示完成。

由索引模型根据 Issue 内容生成。

描述

bug triaged

We are encountering an issue where log messages are not being sent to Graylog. When the lambda first starts, it's able to process one message but it fails for all subsequent runs without any apparent reason (or log message).

Runtime: Java 8 on Amazon Linux 2
Memory: 512 MB

I've tried tweaking the following but nothing seems to help

TCP_QUEUE_SIZE
SHUTDOWN_FLUSH_RETRIES

Logs

message
INIT_START Runtime Version: java:8.al2.v18	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:xxx
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (lambdainternal.CustomerClassLoader@37bba400)
START RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a Version: $LATEST
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
"2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-44-54-814351.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776303.922"" short_message=""[{'time': '2023-03-14T06:43:54.697Z', 'type': 'platform.start', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'version': '8'}}, {'time': '2023-03-14T06:43:54.697Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:43:54.697837+00:00""}
'}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.end', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22'}}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.report', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'metrics': {'durationMs': 331.874, 'billedDurationMs': 332, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:160 - Successfully flushed messages. Shutting down now.
2023-03-14 06:45:04  INFO  GelfTcpTransport:115 - Channel disconnected!
2023-03-14 06:45:04  DEBUG GelfSenderThread:123 - GelfSenderThread exiting!
2023-03-14 06:45:06  DEBUG PoolThreadCache:224 - Freed 1 thread-local buffer(s) from thread: gelfTcpTransport-1-1
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a
REPORT RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a	Duration: 6418.68 ms	Billed Duration: 6419 ms	Memory Size: 512 MB	Max Memory Used: 116 MB
START RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3 Version: $LATEST
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.buffer.AbstractByteBuf
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.util.Recycler
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.util.ReferenceCountUtil
"
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
"io.netty.util.ResourceLeakDetectorFactory
"
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-45-54-692438.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776356.731"" short_message=""[{'time': '2023-03-14T06:44:54.813Z', 'type': 'platform.start', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'version': '8'}}, {'time': '2023-03-14T06:44:54.828Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:44:54.814409+00:00""}
'}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.end', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997'}}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.report', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'metrics': {'durationMs': 284.761, 'billedDurationMs': 285, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 ERROR GelfSenderThread:175 - Failed to flush messages in [10] attempts. Shutting down anyway.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3
REPORT RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3	Duration: 1882.47 ms	Billed Duration: 1883 ms	Memory Size: 512 MB	Max Memory Used: 116 MB

主要语言: Java
星标: 13
派生: 6
PR 合并指标: 30 天内没有已合并 PR

环境准备

没有 Dockerfile 或 Docker Compose 文件
没有 Pull Request 模板
阅读贡献指南

从这里开始

先读完整个 Issue，再读项目的贡献指南。
在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
Fork 仓库，在一个分支上完成修改。
提交 Pull Request，并在描述里引用这个 Issue 编号。

Graylog2/graylog-s3-lambda 的其他 Issue

Safer - Compatible Updates to Fix Vulnerable Dependencies
未关闭
- 哪里不对？
难度 5/5 一周以上新手友好度 18/100

Graylog2/graylog-s3-lambda#44 · 2025年7月16日创建
Current release causes excessive timeouts
未关闭
- 哪里不对？
triaged

难度 4/5 3-5 天新手友好度 25/100

Graylog2/graylog-s3-lambda#38 · 20 条评论 · 2022年1月19日创建
Automate graylog-s3-lambda releases
未关闭
- 哪里不对？
triaged

难度 4/5 3-5 天新手友好度 35/100

Graylog2/graylog-s3-lambda#28 · 1 条评论 · 2021年12月13日创建
Increased memory usage on the Amazon Corretto runtime
未关闭
- 哪里不对？
triaged

难度 4/5 3-5 天新手友好度 35/100

Graylog2/graylog-s3-lambda#23 · 3 条评论 · 2021年7月7日创建
Cloudflare IP fields are set as string instead of IP
未关闭
- 哪里不对？
feature triaged

难度 3/5 1-2 天新手友好度 42/100

Graylog2/graylog-s3-lambda#21 · 1 条评论 · 2021年6月17日创建

查看 Graylog2/graylog-s3-lambda 的全部 Issue

相似的 Issue

[BugFix] Lake scan node omits partition_order_hint, so DESC top-N scans the whole time range (shared-data)
未关闭
- 哪里不对？
type/bug

难度 2/5 1-3 小时新手友好度 88/100

StarRocks/starrocks#80033 · 2026年10月1日创建

维护者通常 1 天内回复
[BUGFIX]: The navigation pop ups for the following schedules should list the screen the user is navigating away from
未关闭
- 哪里不对？
bug

难度 2/5 1-3 小时新手友好度 68/100

bcgov/nr-ilcr#539 · 2026年10月1日创建

维护者通常 1 天内回复
`+unlint` validation crashes when a rule name contains regex metacharacters
未关闭
- 哪里不对？
难度 2/5 1-3 小时新手友好度 82/100

objectionary/lints#1520 · 2026年10月1日创建

维护者通常 1 天内回复
[BUG] ECS ListTasks returns no tasks when serviceName is a service ARN
未关闭
- 哪里不对？
难度 2/5 1-3 小时新手友好度 78/100

floci-io/floci#4858 · 2026年10月1日创建

维护者通常 1 天内回复
Use the CSS constants of ph-css instead of CssProp
未关闭
- 哪里不对？
难度 2/5 1-3 小时新手友好度 74/100

SchweizerischeBundesbahnen/ch.sbb.polarion.extension.pdf-exporter#1109 · 2026年10月1日创建

维护者通常 1 天内回复

更多 Java Issue

Messages not being flushed from queue on subsequent runs

哪里不对？

评估

调研方向

描述

环境准备

从这里开始

Graylog2/graylog-s3-lambda 的其他 Issue

哪里不对？

哪里不对？

哪里不对？

哪里不对？

哪里不对？

相似的 Issue

哪里不对？

哪里不对？

哪里不对？

哪里不对？

哪里不对？

把新 issue 发到你的邮箱