Six concurrent autocomplete lookups for "vitamin a" each blocked on Solr for 4-5 minutes

Aperta
#306 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Start by tracing the autocomplete request path and the Solr query construction for short queries with limit=100 and a biolink_types filter. Compare behavior for concurrent identical requests with the per-request concurrency bound discussed in #273; use the linked log analysis if available. Done means identifying and documenting or fixing the cause of the multi-minute latency under a concurrent burst, with a reproducible check.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

While characterizing NameRes autocomplete traffic from CloudWatch (translator-eks-ci-blue-cluster application logs), the six slowest lookups in a 29-day window were all the same query, issued within 180 ms of each other.

2026-08-07 19:17:26.612  "vitamin a"   321253.90 ms waiting for Solr
2026-08-07 19:17:26.647  "Vitamin A "  268760.21 ms
2026-08-07 19:17:26.682  "vitamin A"   313424.73 ms
2026-08-07 19:17:26.719  "Vitamin A"   264460.04 ms
2026-08-07 19:17:26.756  "Vitamin A"   275157.98 ms
2026-08-07 19:17:26.791  "vitamin A"   308571.10 ms

All six: autocomplete=True, highlighting=False, offset=0, limit=100, biolink_types=['SmallMolecule'], only_prefixes=, exclude_prefixes=, only_taxa=None, pod name-lookup-web-server-dep-648d67989c-288dr, image translator-name-lookup:v1.5.2. All six logged SLOW QUERY.

Two things stand out:

  1. The absolute latency. 4.4 to 5.4 minutes of Solr wait for a two-word common query with a single Biolink type filter. took exceeds the Solr wait by only ~25 ms in every case, so essentially all of it is Solr, not the NameRes app layer.
  2. The arrival pattern. These are ~35 ms apart, so they were in flight concurrently rather than issued one after another — a caller firing six near-identical requests in under 200 ms, differing only in capitalization and a trailing space. Whether that is a retry loop, a type-ahead widget without debouncing, or six real users is not something the logs can distinguish (there is no session or request ID). But six concurrent limit=100 autocomplete queries appear to be enough to put Solr into a state where each takes minutes.

For scale: this is 6 of the 7 lookups in the whole 607-row autocomplete export that exceeded 60 s of Solr wait. Median autocomplete Solr wait in the same export is 48.5 ms. So this is not a slow tail — it is a distinct failure mode that only appears under a concurrent burst.

Worth investigating whether short-ish autocomplete queries at limit=100 with a biolink_types filter have a pathological Solr plan that only bites under concurrency, and whether NameRes should be collapsing or bounding identical in-flight lookups. Possibly related to the per-request concurrency bound discussed in #273.

Happy to share the parsed log export or the analysis notebook if useful — it lives in TranslatorSRI/babel-validation#107.

Lingua principale
Python
Stelle
5
Fork
4
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NCATSTranslator/NameResolution

Tutte le issue di NCATSTranslator/NameResolution

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.