Heap corruption when a compiled NumExpr object is called from several threads with NUMEXPR_NUM_THREADS=1

Aperta
#569 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

@xuu33030 ci sta già lavorando.

Dal 19/9/2026.

  • #570 di @xuu33030 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
50/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
numpy, python
Ambito
data

Direzione di ricerca

Iniziare eseguendo il reproducer fornito con NUMEXPR_NUM_THREADS=1 e ispezionare il percorso single-thread di NumExpr_run, confrontandolo con numexpr.evaluate(). Determinare se le chiamate concorrenti riutilizzano i buffer di lavoro per oggetto. Il lavoro è completato quando l’oggetto compilato condiviso non corrompe più l’heap con il reproducer, con una copertura di regressione o una documentazione chiara della condivisione tra thread supportata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi,
I noticed random crashes of python when parallelizing test execution in pyFAI, noticeable when tuning the number of worker and the number of threads per worker:
https://github.com/silx-kit/pyFAI/issues/2948
I had an AI-agent investigating the issue and it arrived to numexpr with a fairly comprehensive bug-report with a reproducer without pyFAI or anything else than numpy. So I believe this is a valid bug:

Summary

Calling a single pre-compiled numexpr.NumExpr object concurrently from several Python threads corrupts the heap when numexpr
runs single-threaded (NUMEXPR_NUM_THREADS=1). The process dies with Segmentation fault, double free or corruption (out),
corrupted size vs. prev_size or malloc(): invalid size (unsorted), the exact symptom varying from run to run.

Setting NUMEXPR_NUM_THREADS to 2 or more makes the problem disappear entirely.

numexpr.evaluate() does not reproduce it; only a shared compiled NumExpr object does.

Reproducer

"""A single precompiled numexpr.NumExpr object shared by several Python threads."""
from multiprocessing.pool import ThreadPool
import numexpr, numpy
 
print(f"numexpr {numexpr.__version__}, nthreads={numexpr.nthreads}", flush=True)
EXPR = numexpr.NumExpr("where(norm == 0.0, dummy, signal / norm)")
 
size = 200_000
signal = numpy.random.random(size)
norm = numpy.random.random(size)
dummy = numpy.float64(0.0)
 
def work(n):
      for _ in range(300):
          EXPR(norm, dummy, signal)       # same compiled object in every thread
      return n
 
with ThreadPool(4) as pool:
      pool.map(work, range(4))
print("OK")

$ NUMEXPR_NUM_THREADS=1 python reproducer.py
numexpr 2.14.2, nthreads=1
Segmentation fault (core dumped)

$ NUMEXPR_NUM_THREADS=2 python reproducer.py
numexpr 2.14.2, nthreads=2
OK
Observed vs expected

  • Observed: the interpreter dies with a corrupted heap, 100 % of the runs, usually within a couple of seconds.
  • Expected: either the calls are serialised internally and the script prints OK, or the documentation states that a compiled
    NumExpr object must not be shared between threads.

Tested on python 3.13, 3.14 and 3.15, all with GIL (not tested in GIL-free):

  • 2 Python threads are enough; 2, 3 and 4 threads all fail 3/3. There is no gradual degradation, which points at a race
    rather than at resource exhaustion.
  • Reproduced on two Python versions and two numexpr versions, so this is not a regression of a particular release.
  • Both interpreters are regular GIL builds (Py_GIL_DISABLED = 0, sys._is_gil_enabled() is True); free-threading is not
    involved.
  • The machine has 503 GB of RAM, 428 GB free: this is not memory exhaustion.

Where the crash surfaces

A gdb backtrace taken at the moment of the abort shows that the faulting thread is doing a perfectly ordinary
numpy.arange():

  Thread 68 "Thread-2 (worke" received signal SIGABRT, Aborted.
  #6  malloc () from /lib/x86_64-linux-gnu/libc.so.6
  #7  default_malloc () from numpy/_core/_multiarray_umath...so
  #8  PyDataMem_UserNEW () from numpy/_core/_multiarray_umath...so
  #9  PyArray_NewFromDescr_int () from numpy/_core/_multiarray_umath...so
  #10 PyArray_ArangeObj () from numpy/_core/_multiarray_umath...so

while another thread is inside NumExpr_run:

  Thread 67 "Thread-1 (worke":
  #9  NumExpr_run(NumExprObject*, _object*, _object*) () from numexpr/interpreter...so

The heap is therefore already corrupted by the time numpy allocates; the faulty write happens earlier, which is why the
reported symptom varies.

Guess at the cause

With nthreads == 1, NumExpr_run presumably evaluates inline in the calling thread and reuses the per-object working buffers,
whereas with nthreads >= 2 the work goes through the thread pool and its per-thread buffers. Two Python threads entering
the same compiled object then write into the same scratch space. This is consistent with the threshold being exactly 2
threads, and with evaluate() being unaffected.

Workaround

Never let numexpr run with a single thread when compiled expressions are shared:

os.environ.setdefault("NUMEXPR_NUM_THREADS", "2")

Beware that numexpr falls back on OMP_NUM_THREADS when NUMEXPR_NUM_THREADS is unset, so OMP_NUM_THREADS=1 alone — a common setting in batch pipelines and in parallel test runners, which serialise each worker to avoid oversubscription — is enough to trigger this.

Lingua principale
Python
Stelle
2.5k
Fork
235
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pydata/numexpr

Tutte le issue di pydata/numexpr

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.