out_azure_blob: append is sent without the SAS signature after the blob is created (regression since 3.0.7)

Aperta Adatta ai principianti
  • Qualcosa non va?
#12,636 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
79/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
azure, c
Ambito
backend, cloud

Direzione di ricerca

Inizia da plugins/out_azure_blob/azure_blob.c e segui come http_send_blob() gestisce la risposta 404 prima del nuovo tentativo dopo la creazione del blob. Controlla la redazione dell’URI nel ramo 404 e i rami correlati 201 e 202; il report dice che l’URI della richiesta viene riutilizzato dopo essere stato modificato. Il lavoro è completato quando la redazione della firma non altera l’URI usato per l’append successivo, che dovrebbe riuscire al primo tentativo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug Report

Describe the bug

With auth_type sas and the default append blob type, the first append to every new blob goes out without the sig parameter of the SAS token, so the service rejects it (403 on Azurite, 401 NoAuthenticationInformation on Azure Storage). The chunk is then retried and the retry succeeds, so data normally arrives, but every new blob costs one failed request, an [error] line and a delay of one retry interval.

It is a regression: 3.0.7 is not affected, 5.1.4 is, and the code involved is unchanged on master.

To Reproduce

  1. Start Azurite and create an account SAS signed with its well-known development key:
docker network create azt
docker run -d --name azurite --network azt mcr.microsoft.com/azure-storage/azurite \
  azurite-blob --blobHost 0.0.0.0 --skipApiVersionCheck --loose

# sas.py: prints the SAS token
import base64, hmac, hashlib, urllib.parse
KEY = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
acct, sv, sp, ss, srt, se, spr = "devstoreaccount1", "2021-08-06", "racwdl", "b", "sco", "2031-01-01T00:00:00Z", "https,http"
sts = "\n".join([acct, sp, ss, srt, "", se, "", spr, sv, ""]) + "\n"
sig = base64.b64encode(hmac.new(base64.b64decode(KEY), sts.encode(), hashlib.sha256).digest()).decode()
print(urllib.parse.urlencode({"sv": sv, "ss": ss, "srt": srt, "sp": sp, "se": se, "spr": spr, "sig": sig}, quote_via=urllib.parse.quote))
  1. Run Fluent Bit with this configuration (sas_token is the output of sas.py):
[SERVICE]
    Flush 1
    Log_Level info

[INPUT]
    Name  dummy
    Dummy {"message":"hello"}
    Rate  1

[OUTPUT]
    Name                  azure_blob
    Match                 *
    account_name          devstoreaccount1
    auth_type             sas
    sas_token             <output of sas.py>
    container_name        test
    path                  logs
    auto_create_container on
    emulator_mode         on
    endpoint              http://azurite:10000
    tls                   off

docker run --network azt -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:5.1.4

Expected behavior

The append that follows "blob created successfully" carries the full SAS token and succeeds.

Actual behavior

[ info] [output:azure_blob:azure_blob.0] container 'test' created sucessfully
[ info] [output:azure_blob:azure_blob.0] blob not found: /devstoreaccount1/test/logs/dummy.0?comp=appendblock&sv=2021-08-06&...
[ info] [output:azure_blob:azure_blob.0] blob created successfully: /devstoreaccount1/test/logs/dummy.0?sv=2021-08-06&...
[error] [output:azure_blob:azure_blob.0] http_status=403 cannot append content to blob
[ warn] [engine] failed to flush chunk '1-1792673448.64011297.flb', retry in 7 seconds: ...

The requests Azurite received for the first blob, showing only the parameter names:

PUT dummy.0  comp&sv&ss&srt&sp&se&spr&sig   -> 404
PUT dummy.0  sv&ss&srt&sp&se&spr&sig        -> 201   (create blob)
PUT dummy.0  comp&sv&ss&srt&sp&se&spr       -> 403   (append: no sig)
PUT dummy.0  comp&sv&ss&srt&sp&se&spr&sig   -> 201   (the retry, a fresh URI)

With fluent/fluent-bit:3.0.7 the same configuration produces no error, and the append after the create is signed. With auth_type key there is nothing to truncate and it does not happen.

Cause

http_send_blob() hides the signature from the log line by writing a NUL into the request URI (plugins/out_azure_blob/azure_blob.c, v5.1.4 line 1030, in the 404 branch):

/* delete "&sig=..." in the c->uri for security */
char *p = strstr(c->uri, "&sig=");
if (p) {
    *p = '\0';
}
flb_plg_info(ctx->ins, "blob not found: %s", c->uri);

flb_http_client() does not copy the URI (src/flb_http_client.c: c->uri = uri;), so c->uri is the buffer send_blob() allocated, and that string is now cut before &sig=. send_blob() then creates the blob and calls http_send_blob() again with the same uri (line 1177 onwards):

else if (ret == CREATE_BLOB) {
    ret = create_blob(ctx, path_prefix, name);
    if (ret == FLB_OK) {
        ret = http_send_blob(config, ctx, ref_name, uri, block_id, event_type, data, bytes);

In 3.0.7 the equivalent code called send_blob() again, which rebuilt the URI, so the redaction was harmless there.

Suggested fix

Redact a copy instead of the live request, e.g. log with a length: flb_plg_info(ctx->ins, "blob not found: %.*s", p ? (int) (p - c->uri) : (int) strlen(c->uri), c->uri);, or duplicate the string first. The same strstr(c->uri, "&sig=") write appears in the 201 and 202 branches (lines 755 and 834); they do not reuse the buffer afterwards today, but they would break the same way if they did, so they are worth changing too. I am happy to send a PR.

Your Environment

  • Version used: 5.1.4 (official image); the code is the same on master (checked 2026-10-22)
  • Configuration: above
  • Environment name and version: Docker; Azurite for the reproduction, and the same 401 NoAuthenticationInformation seen against Azure Storage
  • Filters and plugins: dummy input, azure_blob output
Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.