out_s3: stalled TLS request-body writes are not bounded by net.io_timeout (5.0.4)

Aperta
  • Qualcosa non va?
#12,742 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
aws, c, linux
Ambito
backend, networking

Direzione di ricerca

Start by tracing how the S3 output creates its HTTPS connection and writes the PutObject body, then check where net.io_timeout is applied to the request. Compare the stalled-body case with the full-body/no-response control described here; the issue does not name source files or tests. Done means establishing whether a supported setting or fix bounds stalled writes and documenting or implementing a validated path that allows retry.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug Report

Describe the bug

With Fluent Bit 5.0.4, an S3 PutObject upload can remain blocked while writing the HTTPS request body when the peer stops acknowledging data. In a controlled local test, net.io_timeout=3 did not interrupt the stalled write during 20 seconds of observation.

The local baseline was observed for 20 seconds only; the experiment did not measure the eventual default OS TCP timeout.

Environment
  • Official image: cr.fluentbit.io/fluent/fluent-bit:5.0.4; reported commit 1e69f3726f.
  • Local reproduction: Linux arm64, kernel 6.8.0-117-generic, Docker 29.5.2 in Colima.
  • Host/VM TCP settings unchanged; tcp_retries2=15.
  • tail input, filesystem storage, S3 PutObject/gzip, one S3 worker.
  • Not yet reproduced on the latest release/master.
Configuration used in the local test

Relevant S3 output settings (endpoint, certificate and credentials are local test fixtures, not AWS):

[OUTPUT]
    Name s3
    Match *
    bucket local-test
    region us-east-1
    endpoint https://127.0.0.1:8443
    tls On
    tls.verify Off
    use_put_object On
    compression gzip
    total_file_size 250M
    upload_timeout 2s
    store_dir /work/s3
    store_dir_limit_size 256M
    preserve_data_ordering On
    Retry_Limit False
    workers 1
    net.keepalive Off
    net.io_timeout 3

TLS verification is disabled ONLY for the local self-signed mock.

Steps to reproduce
  1. Tail approximately 8 MiB of random JSON records into the output above.
  2. Use a local HTTPS PutObject receiver in an isolated container network namespace.
  3. After the first TLS handshake and HTTP headers, stop reading the body and drop server-to-client packets for that established connection only. In the receiver namespace:
iptables -I OUTPUT -p tcp --sport 8443 --dport <client-ephemeral-port> -j DROP
  1. Allow subsequent connections to succeed. Observe the pending client send queue and packet-drop counters.
  2. Compare with TCP_USER_TIMEOUT applied to S3 sockets using a test-only LD_PRELOAD connect shim. Verify the effective value with getsockopt.

Both controls and treatments used the same small socket buffers (SO_SNDBUF requested 32768 bytes; receiver SO_RCVBUF 32768). No real AWS resources were used.

Results
Failure TCP_USER_TIMEOUT net.io_timeout Observation
Stalled body write 0 0s No recovery during 20s
Stalled body write 0 3s No recovery during 20s
Stalled body write 5000ms 0s First retry after 5.43s
Stalled body write 60000ms 0s First retry after 60.44s
Full body received/ACKed, no HTTP reply 5000ms 0s No recovery during 20s
Full body received/ACKed, no HTTP reply 5000ms 3s First retry after 3.01s

The write-timeout treatments logged errno=110 (Connection timed out), retried without restarting Fluent Bit, and delivered all 528 test IDs. All IDs reached the parallel TCP sink too. This small workload does not reproduce high-volume ingestion backpressure.

The no-response control intentionally distinguishes a body-write stall from response waiting: TCP_USER_TIMEOUT alone does not bound the entire HTTP request. Complete bodies were retransmitted in the response-timeout case, so this is not an exactly-once delivery claim.

Expected behavior / question

A stalled synchronous S3 body write should have an effective configurable timeout, allowing retry without waiting for the default OS TCP timeout.

Is there an existing supported configuration or fix for this path? Otherwise, could synchronous writes be bounded with a write deadline or an opt-in per-output TCP_USER_TIMEOUT on Linux? The latter is an experimental candidate, not a universal HTTP timeout or a validated fix.

Related architectural discussion: https://github.com/fluent/fluent-bit/issues/7466

Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.