elastic/elasticsearch-py

Helpers for `bulk` method such as `async_bulk` sleep in blocking manner, preventing graceful shutdown

Open

Aperta il 22 mar 2024

Vedi su GitHub
 (5 commenti) (0 reazioni) (0 assegnatari)Python (4060 star) (1206 fork)batch import
Area: HelpersCategory: Enhancementgood first issue

Descrizione

We tried using async_bulk and async_streaming_bulk helpers to ingest data into Elasticsearch and they work great, but we've found out that they prevent our code from gracefully shutting down when CTRL+C is pressed.

Example code that sleeps: https://github.com/elastic/elasticsearch-py/blob/e1603f4f25888ca7fee03ee977abafb00d512601/elasticsearch/_async/helpers.py#L249-L251

It would be great to have a way to:

  • Either define how the sleep happens by passing sleep function into the client
  • Make Elasticsearch client internally cancel all sleeps when the client is closed

Guida contributor

Tech stack
python
Dominio
backenddata
Tipo issue
bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
half day
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
active
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
Python async programmingElasticsearch client helpers
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
65
Direzione di ricerca
Examine the helpers.py file at the linked line (L249 L251) to understand the blocking sleep call. Consider implementing one of the proposed solutions: either allow passing a custom sleep function to the helpers, or make the client cancel all in progress sleeps when closed. Review the existing async helpers for bulk operations and test the fix with a simulated shutdown scenario.