A record whose body fails to build leaves its action line in the bulk request, and the whole chunk is rejected

Aperta Adatta ai principianti
#208 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
76/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
ruby
Ambito
backend, databases

Direzione di ricerca

Inizia da append_record_to_messages in lib/fluent/plugin/out_opensearch.rb, intorno alle righe 817–840, poi leggi la gestione degli errori in write, intorno alle righe 1023–1027. Riproduci il problema con la configurazione e i record forniti, quindi controlla i test esistenti in test_out_opensearch.rb e test_out_opensearch_data_stream.rb. Il lavoro è completo quando un errore nella creazione del body invia solo quel record a @ERROR, mentre i record validi nel chunk vengono scritti senza riprovare la richiesta bulk malformata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Describe the bug

append_record_to_messages adds the action line of a record to the bulk request before it builds the body. When building the body raises, the action line stays in the request without a body. write sends the record to @ERROR, but does not remove the line.

OpenSearch then reads the next action line as the body, and rejects the whole bulk request with 400. None of the other records in the chunk are written. The retry builds the same request, so the chunk fails until the retry limit and is dropped, and the bad record goes to @ERROR on every try.

https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L817-L840
https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L1023-L1027

I found two ways for the body to raise:

  • With remove_keys_on_update_key, the record has a String there instead of an Array, such as "msg, foo". remove_keys raises NoMethodError: undefined method 'any?' for an instance of String.
    https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L861
  • Without the oj gem, the plugin uses JSON.generate, which raises JSON::GeneratorError on a value that is not valid UTF-8. This also happens with write_operation index. The response is Malformed action/metadata line [5], expected one of [create, delete, index, update] but found [id].

When the bad record is the last one in the request, OpenSearch ignores the trailing action line, and the other records are written.

To Reproduce

Run Fluentd with the configuration below and an OpenSearch at localhost:19201. /tmp/records.jsonl has three records, and the second one has a String in skip:

{"id":1,"msg":"a"}
{"id":2,"msg":"b","foo":"F","skip":"msg, foo"}
{"id":3,"msg":"c"}

The bulk request (from @log_level trace):

{"update":{"_index":"keystr-fluentd","_id":1}}
{"doc":{"id":1,"msg":"a"},"doc_as_upsert":true}
{"update":{"_index":"keystr-fluentd","_id":2}}
{"update":{"_index":"keystr-fluentd","_id":3}}
{"doc":{"id":3,"msg":"c"},"doc_as_upsert":true}

OpenSearch rejects it with [400] x_content_parse_exception: [1:2] [UpdateRequest] unknown field [update]. The first try and the three retries all fail the same way. Record 2 is sent to @ERROR four times. Then the chunk with the three records is dropped, and no record is written.

Expected behavior

Only the bad record goes to @ERROR. The other records in the chunk are written, and the chunk is not retried because of it.

Your Environment
- Fluentd version: 1.20.1
- TD Agent version: -
- fluent-plugin-opensearch version: main (76c177e). v1.1.8 has the same code.
- OpenSearch version (optional): 2.19.1
- OpenSearch template(s) (optional): -
- Operating system: Ubuntu 24.04
- Kernel version: -
Your Configuration
<source>
  @type tail
  path /tmp/records.jsonl
  pos_file /tmp/records.pos
  read_from_head true
  tag repro
  <parse>
    @type json
  </parse>
</source>
<match repro>
  @type opensearch
  @log_level trace
  host localhost
  port 19201
  index_name keystr-fluentd
  write_operation upsert
  id_key id
  remove_keys_on_update_key skip
  <buffer>
    @type file
    path /tmp/buffer
    flush_mode interval
    flush_interval 1s
    retry_type periodic
    retry_wait 2s
    retry_max_times 3
  </buffer>
</match>
<label @ERROR>
  <match **>
    @type stdout
  </match>
</label>
Your Error Log
2026-11-17 05:11:48 +0000 [warn]: #0 send an error event to @ERROR: error_class=NoMethodError error="undefined method 'any?' for an instance of String" location=".../lib/fluent/plugin/out_opensearch.rb:861:in 'Fluent::Plugin::OpenSearchOutput#remove_keys'" tag="repro" time=2026-11-17 05:11:46.009996892 +0000
2026-11-17 05:11:48 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2026-11-17 05:11:50 +0000 chunk="66071ee4f4fed3ab81b7d212ccc1377d" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({host: \"localhost\", port: 19201, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"},\"status\":400}"
2026-11-17 05:11:54 +0000 [error]: #0 Hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=3 error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({host: \"localhost\", port: 19201, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"},\"status\":400}"
Additional context

I tried building the body first and adding the action line and the body together in append_record_to_messages. With that change, records 1 and 3 are written in one request, and only record 2 goes to @ERROR, once. The invalid UTF-8 case is fixed the same way. The existing tests in test_out_opensearch.rb and test_out_opensearch_data_stream.rb still pass with it (checked on 278e101, the commit before #206).

Lingua principale
Ruby
Stelle
64
Fork
25
Merge medio
1g 16h
PR unite (30g)
8

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-plugin-opensearch

Tutte le issue di fluent/fluent-plugin-opensearch

Issue simili

Altre issue su Ruby

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.