Heap corruption when a compiled NumExpr object is called from several threads with NUMEXPR_NUM_THREADS=1
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 50/100
Direzione di ricerca
Iniziare eseguendo il reproducer fornito con NUMEXPR_NUM_THREADS=1 e ispezionare il percorso single-thread di NumExpr_run, confrontandolo con numexpr.evaluate(). Determinare se le chiamate concorrenti riutilizzano i buffer di lavoro per oggetto. Il lavoro è completato quando l’oggetto compilato condiviso non corrompe più l’heap con il reproducer, con una copertura di regressione o una documentazione chiara della condivisione tra thread supportata.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Hi,
I noticed random crashes of python when parallelizing test execution in pyFAI, noticeable when tuning the number of worker and the number of threads per worker:
https://github.com/silx-kit/pyFAI/issues/2948
I had an AI-agent investigating the issue and it arrived to numexpr with a fairly comprehensive bug-report with a reproducer without pyFAI or anything else than numpy. So I believe this is a valid bug:
Summary
Calling a single pre-compiled numexpr.NumExpr object concurrently from several Python threads corrupts the heap when numexpr
runs single-threaded (NUMEXPR_NUM_THREADS=1). The process dies with Segmentation fault, double free or corruption (out),
corrupted size vs. prev_size or malloc(): invalid size (unsorted), the exact symptom varying from run to run.
Setting NUMEXPR_NUM_THREADS to 2 or more makes the problem disappear entirely.
numexpr.evaluate() does not reproduce it; only a shared compiled NumExpr object does.
Reproducer
"""A single precompiled numexpr.NumExpr object shared by several Python threads."""
from multiprocessing.pool import ThreadPool
import numexpr, numpy
print(f"numexpr {numexpr.__version__}, nthreads={numexpr.nthreads}", flush=True)
EXPR = numexpr.NumExpr("where(norm == 0.0, dummy, signal / norm)")
size = 200_000
signal = numpy.random.random(size)
norm = numpy.random.random(size)
dummy = numpy.float64(0.0)
def work(n):
for _ in range(300):
EXPR(norm, dummy, signal) # same compiled object in every thread
return n
with ThreadPool(4) as pool:
pool.map(work, range(4))
print("OK")
$ NUMEXPR_NUM_THREADS=1 python reproducer.py
numexpr 2.14.2, nthreads=1
Segmentation fault (core dumped)
$ NUMEXPR_NUM_THREADS=2 python reproducer.py
numexpr 2.14.2, nthreads=2
OK
Observed vs expected
- Observed: the interpreter dies with a corrupted heap, 100 % of the runs, usually within a couple of seconds.
- Expected: either the calls are serialised internally and the script prints OK, or the documentation states that a compiled
NumExpr object must not be shared between threads.
Tested on python 3.13, 3.14 and 3.15, all with GIL (not tested in GIL-free):
- 2 Python threads are enough; 2, 3 and 4 threads all fail 3/3. There is no gradual degradation, which points at a race
rather than at resource exhaustion. - Reproduced on two Python versions and two numexpr versions, so this is not a regression of a particular release.
- Both interpreters are regular GIL builds (Py_GIL_DISABLED = 0, sys._is_gil_enabled() is True); free-threading is not
involved. - The machine has 503 GB of RAM, 428 GB free: this is not memory exhaustion.
Where the crash surfaces
A gdb backtrace taken at the moment of the abort shows that the faulting thread is doing a perfectly ordinary
numpy.arange():
Thread 68 "Thread-2 (worke" received signal SIGABRT, Aborted.
#6 malloc () from /lib/x86_64-linux-gnu/libc.so.6
#7 default_malloc () from numpy/_core/_multiarray_umath...so
#8 PyDataMem_UserNEW () from numpy/_core/_multiarray_umath...so
#9 PyArray_NewFromDescr_int () from numpy/_core/_multiarray_umath...so
#10 PyArray_ArangeObj () from numpy/_core/_multiarray_umath...so
while another thread is inside NumExpr_run:
Thread 67 "Thread-1 (worke":
#9 NumExpr_run(NumExprObject*, _object*, _object*) () from numexpr/interpreter...so
The heap is therefore already corrupted by the time numpy allocates; the faulty write happens earlier, which is why the
reported symptom varies.
Guess at the cause
With nthreads == 1, NumExpr_run presumably evaluates inline in the calling thread and reuses the per-object working buffers,
whereas with nthreads >= 2 the work goes through the thread pool and its per-thread buffers. Two Python threads entering
the same compiled object then write into the same scratch space. This is consistent with the threshold being exactly 2
threads, and with evaluate() being unaffected.
Workaround
Never let numexpr run with a single thread when compiled expressions are shared:
os.environ.setdefault("NUMEXPR_NUM_THREADS", "2")
Beware that numexpr falls back on OMP_NUM_THREADS when NUMEXPR_NUM_THREADS is unset, so OMP_NUM_THREADS=1 alone — a common setting in batch pipelines and in parallel test runners, which serialise each worker to avoid oversubscription — is enough to trigger this.
- Lingua principale
- Python
- Stelle
- 2.5k
- Fork
- 235
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di pydata/numexpr
-
Error in numexpr.validate documentationForse già presa @syahra712 l’ha presa 40 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
-
Python 3.15 wheelsAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 50/100
-
Rare dead-lock observed when using `set_num_threads()` in parallel environmentForse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
Tutte le issue di pydata/numexpr
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
abetlen/llama-cpp-python#2424 ·
I maintainer di solito rispondono entro 1 giorno
-
Bug IronPython Python 3
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
pyrevitlabs/pyRevit#4184 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 74/100
jqueryscript/awesome-claude-code#1009 ·
-
tests
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno