A record whose body fails to build leaves its action line in the bulk request, and the whole chunk is rejected
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Idoneità per principianti
- 76/100
Direzione di ricerca
Inizia da append_record_to_messages in lib/fluent/plugin/out_opensearch.rb, intorno alle righe 817–840, poi leggi la gestione degli errori in write, intorno alle righe 1023–1027. Riproduci il problema con la configurazione e i record forniti, quindi controlla i test esistenti in test_out_opensearch.rb e test_out_opensearch_data_stream.rb. Il lavoro è completo quando un errore nella creazione del body invia solo quel record a @ERROR, mentre i record validi nel chunk vengono scritti senza riprovare la richiesta bulk malformata.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Describe the bug
append_record_to_messages adds the action line of a record to the bulk request before it builds the body. When building the body raises, the action line stays in the request without a body. write sends the record to @ERROR, but does not remove the line.
OpenSearch then reads the next action line as the body, and rejects the whole bulk request with 400. None of the other records in the chunk are written. The retry builds the same request, so the chunk fails until the retry limit and is dropped, and the bad record goes to @ERROR on every try.
https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L817-L840
https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L1023-L1027
I found two ways for the body to raise:
- With
remove_keys_on_update_key, the record has a String there instead of an Array, such as"msg, foo".remove_keysraisesNoMethodError: undefined method 'any?' for an instance of String.
https://github.com/fluent/fluent-plugin-opensearch/blob/76c177e/lib/fluent/plugin/out_opensearch.rb#L861 - Without the oj gem, the plugin uses
JSON.generate, which raisesJSON::GeneratorErroron a value that is not valid UTF-8. This also happens withwrite_operation index. The response isMalformed action/metadata line [5], expected one of [create, delete, index, update] but found [id].
When the bad record is the last one in the request, OpenSearch ignores the trailing action line, and the other records are written.
To Reproduce
Run Fluentd with the configuration below and an OpenSearch at localhost:19201. /tmp/records.jsonl has three records, and the second one has a String in skip:
{"id":1,"msg":"a"}
{"id":2,"msg":"b","foo":"F","skip":"msg, foo"}
{"id":3,"msg":"c"}
The bulk request (from @log_level trace):
{"update":{"_index":"keystr-fluentd","_id":1}}
{"doc":{"id":1,"msg":"a"},"doc_as_upsert":true}
{"update":{"_index":"keystr-fluentd","_id":2}}
{"update":{"_index":"keystr-fluentd","_id":3}}
{"doc":{"id":3,"msg":"c"},"doc_as_upsert":true}
OpenSearch rejects it with [400] x_content_parse_exception: [1:2] [UpdateRequest] unknown field [update]. The first try and the three retries all fail the same way. Record 2 is sent to @ERROR four times. Then the chunk with the three records is dropped, and no record is written.
Expected behavior
Only the bad record goes to @ERROR. The other records in the chunk are written, and the chunk is not retried because of it.
Your Environment
- Fluentd version: 1.20.1
- TD Agent version: -
- fluent-plugin-opensearch version: main (76c177e). v1.1.8 has the same code.
- OpenSearch version (optional): 2.19.1
- OpenSearch template(s) (optional): -
- Operating system: Ubuntu 24.04
- Kernel version: -
Your Configuration
<source>
@type tail
path /tmp/records.jsonl
pos_file /tmp/records.pos
read_from_head true
tag repro
<parse>
@type json
</parse>
</source>
<match repro>
@type opensearch
@log_level trace
host localhost
port 19201
index_name keystr-fluentd
write_operation upsert
id_key id
remove_keys_on_update_key skip
<buffer>
@type file
path /tmp/buffer
flush_mode interval
flush_interval 1s
retry_type periodic
retry_wait 2s
retry_max_times 3
</buffer>
</match>
<label @ERROR>
<match **>
@type stdout
</match>
</label>
Your Error Log
2026-11-17 05:11:48 +0000 [warn]: #0 send an error event to @ERROR: error_class=NoMethodError error="undefined method 'any?' for an instance of String" location=".../lib/fluent/plugin/out_opensearch.rb:861:in 'Fluent::Plugin::OpenSearchOutput#remove_keys'" tag="repro" time=2026-11-17 05:11:46.009996892 +0000
2026-11-17 05:11:48 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2026-11-17 05:11:50 +0000 chunk="66071ee4f4fed3ab81b7d212ccc1377d" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({host: \"localhost\", port: 19201, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"},\"status\":400}"
2026-11-17 05:11:54 +0000 [error]: #0 Hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=3 error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({host: \"localhost\", port: 19201, scheme: \"http\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [UpdateRequest] unknown field [update]\"},\"status\":400}"
Additional context
I tried building the body first and adding the action line and the body together in append_record_to_messages. With that change, records 1 and 3 are written in one request, and only record 2 goes to @ERROR, once. The invalid UTF-8 case is fixed the same way. The existing tests in test_out_opensearch.rb and test_out_opensearch_data_stream.rb still pass with it (checked on 278e101, the commit before #206).
- Lingua principale
- Ruby
- Stelle
- 64
- Fork
- 25
- Merge medio
- 1g 16h
- PR unite (30g)
- 8
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di fluent/fluent-plugin-opensearch
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
The plugin is not retrying on specific errors and dropping the data on error 400Forse già presa @dazoakley l’ha presa 720 giorni fa. Apertaenhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
fluent/fluent-plugin-opensearch#134 · 5 commenti ·
-
enhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 50/100
fluent/fluent-plugin-opensearch#123 · 1 commento ·
-
bug enhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 38/100
fluent/fluent-plugin-opensearch#122 · 1 commento ·
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 35/100
fluent/fluent-plugin-opensearch#100 · 1 commento · 2 reazioni ·
Tutte le issue di fluent/fluent-plugin-opensearch
Issue simili
-
Payment details dialog shows "Did not pass" for AVS/CVV results that mean the check wasn't performedAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
autonome
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
demarche-numerique/demarche.numerique.gouv.fr#14566 ·
I maintainer di solito rispondono entro 1 giorno
-
Home link is now tinyApertaarea/web interface
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
mastodon/mastodon#41723 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 85/100
akitaonrails/ai-book-2026#15 ·