Heap corruption when a compiled NumExpr object is called from several threads with NUMEXPR_NUM_THREADS=1

オープン
#569 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る

@xuu33030 がすでに取り組んでいます。

2026年9月19日 から。

  • #570 @xuu33030 による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
50/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
numpy, python
領域
data

調査の方向性

まず、提供された再現コードを NUMEXPR_NUM_THREADS=1 で実行し、NumExpr_run のシングルスレッドパスを調査して、numexpr.evaluate() と比較します。並行呼び出しがオブジェクトごとのワーキングバッファを再利用するかどうかを確認します。再現コードの実行時に共有コンパイル済みオブジェクトがヒープを破壊しなくなり、リグレッションテストまたはサポートされるスレッド共有についての明確なドキュメントが用意されれば完了です。

索引モデルが issue の本文から書いたものです。

説明

Hi,
I noticed random crashes of python when parallelizing test execution in pyFAI, noticeable when tuning the number of worker and the number of threads per worker:
https://github.com/silx-kit/pyFAI/issues/2948
I had an AI-agent investigating the issue and it arrived to numexpr with a fairly comprehensive bug-report with a reproducer without pyFAI or anything else than numpy. So I believe this is a valid bug:

Summary

Calling a single pre-compiled numexpr.NumExpr object concurrently from several Python threads corrupts the heap when numexpr
runs single-threaded (NUMEXPR_NUM_THREADS=1). The process dies with Segmentation fault, double free or corruption (out),
corrupted size vs. prev_size or malloc(): invalid size (unsorted), the exact symptom varying from run to run.

Setting NUMEXPR_NUM_THREADS to 2 or more makes the problem disappear entirely.

numexpr.evaluate() does not reproduce it; only a shared compiled NumExpr object does.

Reproducer

"""A single precompiled numexpr.NumExpr object shared by several Python threads."""
from multiprocessing.pool import ThreadPool
import numexpr, numpy
 
print(f"numexpr {numexpr.__version__}, nthreads={numexpr.nthreads}", flush=True)
EXPR = numexpr.NumExpr("where(norm == 0.0, dummy, signal / norm)")
 
size = 200_000
signal = numpy.random.random(size)
norm = numpy.random.random(size)
dummy = numpy.float64(0.0)
 
def work(n):
      for _ in range(300):
          EXPR(norm, dummy, signal)       # same compiled object in every thread
      return n
 
with ThreadPool(4) as pool:
      pool.map(work, range(4))
print("OK")

$ NUMEXPR_NUM_THREADS=1 python reproducer.py
numexpr 2.14.2, nthreads=1
Segmentation fault (core dumped)

$ NUMEXPR_NUM_THREADS=2 python reproducer.py
numexpr 2.14.2, nthreads=2
OK
Observed vs expected

  • Observed: the interpreter dies with a corrupted heap, 100 % of the runs, usually within a couple of seconds.
  • Expected: either the calls are serialised internally and the script prints OK, or the documentation states that a compiled
    NumExpr object must not be shared between threads.

Tested on python 3.13, 3.14 and 3.15, all with GIL (not tested in GIL-free):

  • 2 Python threads are enough; 2, 3 and 4 threads all fail 3/3. There is no gradual degradation, which points at a race
    rather than at resource exhaustion.
  • Reproduced on two Python versions and two numexpr versions, so this is not a regression of a particular release.
  • Both interpreters are regular GIL builds (Py_GIL_DISABLED = 0, sys._is_gil_enabled() is True); free-threading is not
    involved.
  • The machine has 503 GB of RAM, 428 GB free: this is not memory exhaustion.

Where the crash surfaces

A gdb backtrace taken at the moment of the abort shows that the faulting thread is doing a perfectly ordinary
numpy.arange():

  Thread 68 "Thread-2 (worke" received signal SIGABRT, Aborted.
  #6  malloc () from /lib/x86_64-linux-gnu/libc.so.6
  #7  default_malloc () from numpy/_core/_multiarray_umath...so
  #8  PyDataMem_UserNEW () from numpy/_core/_multiarray_umath...so
  #9  PyArray_NewFromDescr_int () from numpy/_core/_multiarray_umath...so
  #10 PyArray_ArangeObj () from numpy/_core/_multiarray_umath...so

while another thread is inside NumExpr_run:

  Thread 67 "Thread-1 (worke":
  #9  NumExpr_run(NumExprObject*, _object*, _object*) () from numexpr/interpreter...so

The heap is therefore already corrupted by the time numpy allocates; the faulty write happens earlier, which is why the
reported symptom varies.

Guess at the cause

With nthreads == 1, NumExpr_run presumably evaluates inline in the calling thread and reuses the per-object working buffers,
whereas with nthreads >= 2 the work goes through the thread pool and its per-thread buffers. Two Python threads entering
the same compiled object then write into the same scratch space. This is consistent with the threshold being exactly 2
threads, and with evaluate() being unaffected.

Workaround

Never let numexpr run with a single thread when compiled expressions are shared:

os.environ.setdefault("NUMEXPR_NUM_THREADS", "2")

Beware that numexpr falls back on OMP_NUM_THREADS when NUMEXPR_NUM_THREADS is unset, so OMP_NUM_THREADS=1 alone — a common setting in batch pipelines and in parallel test runners, which serialise each worker to avoid oversubscription — is enough to trigger this.

主要言語
Python
スター
2.5k
フォーク
235
PR マージ指標
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pydata/numexpr のほかの issue

pydata/numexpr の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。