Heap corruption when a compiled NumExpr object is called from several threads with NUMEXPR_NUM_THREADS=1

未关闭
#569 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

@xuu33030 已经在做这个了。

开始于 2026年9月19日。

  • #570 来自 @xuu33030 —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
50/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
numpy, python
领域
data

调研方向

首先使用 NUMEXPR_NUM_THREADS=1 运行提供的复现程序,并检查 NumExpr_run 的单线程路径,将其与 numexpr.evaluate() 进行比较。确定并发调用是否会重用每个对象的工作缓冲区。在复现程序下,共享的已编译对象不再破坏堆,并且有回归测试覆盖或关于所支持线程共享的清晰文档,即视为完成。

由索引模型根据 Issue 内容生成。

描述

Hi,
I noticed random crashes of python when parallelizing test execution in pyFAI, noticeable when tuning the number of worker and the number of threads per worker:
https://github.com/silx-kit/pyFAI/issues/2948
I had an AI-agent investigating the issue and it arrived to numexpr with a fairly comprehensive bug-report with a reproducer without pyFAI or anything else than numpy. So I believe this is a valid bug:

Summary

Calling a single pre-compiled numexpr.NumExpr object concurrently from several Python threads corrupts the heap when numexpr
runs single-threaded (NUMEXPR_NUM_THREADS=1). The process dies with Segmentation fault, double free or corruption (out),
corrupted size vs. prev_size or malloc(): invalid size (unsorted), the exact symptom varying from run to run.

Setting NUMEXPR_NUM_THREADS to 2 or more makes the problem disappear entirely.

numexpr.evaluate() does not reproduce it; only a shared compiled NumExpr object does.

Reproducer

"""A single precompiled numexpr.NumExpr object shared by several Python threads."""
from multiprocessing.pool import ThreadPool
import numexpr, numpy
 
print(f"numexpr {numexpr.__version__}, nthreads={numexpr.nthreads}", flush=True)
EXPR = numexpr.NumExpr("where(norm == 0.0, dummy, signal / norm)")
 
size = 200_000
signal = numpy.random.random(size)
norm = numpy.random.random(size)
dummy = numpy.float64(0.0)
 
def work(n):
      for _ in range(300):
          EXPR(norm, dummy, signal)       # same compiled object in every thread
      return n
 
with ThreadPool(4) as pool:
      pool.map(work, range(4))
print("OK")

$ NUMEXPR_NUM_THREADS=1 python reproducer.py
numexpr 2.14.2, nthreads=1
Segmentation fault (core dumped)

$ NUMEXPR_NUM_THREADS=2 python reproducer.py
numexpr 2.14.2, nthreads=2
OK
Observed vs expected

  • Observed: the interpreter dies with a corrupted heap, 100 % of the runs, usually within a couple of seconds.
  • Expected: either the calls are serialised internally and the script prints OK, or the documentation states that a compiled
    NumExpr object must not be shared between threads.

Tested on python 3.13, 3.14 and 3.15, all with GIL (not tested in GIL-free):

  • 2 Python threads are enough; 2, 3 and 4 threads all fail 3/3. There is no gradual degradation, which points at a race
    rather than at resource exhaustion.
  • Reproduced on two Python versions and two numexpr versions, so this is not a regression of a particular release.
  • Both interpreters are regular GIL builds (Py_GIL_DISABLED = 0, sys._is_gil_enabled() is True); free-threading is not
    involved.
  • The machine has 503 GB of RAM, 428 GB free: this is not memory exhaustion.

Where the crash surfaces

A gdb backtrace taken at the moment of the abort shows that the faulting thread is doing a perfectly ordinary
numpy.arange():

  Thread 68 "Thread-2 (worke" received signal SIGABRT, Aborted.
  #6  malloc () from /lib/x86_64-linux-gnu/libc.so.6
  #7  default_malloc () from numpy/_core/_multiarray_umath...so
  #8  PyDataMem_UserNEW () from numpy/_core/_multiarray_umath...so
  #9  PyArray_NewFromDescr_int () from numpy/_core/_multiarray_umath...so
  #10 PyArray_ArangeObj () from numpy/_core/_multiarray_umath...so

while another thread is inside NumExpr_run:

  Thread 67 "Thread-1 (worke":
  #9  NumExpr_run(NumExprObject*, _object*, _object*) () from numexpr/interpreter...so

The heap is therefore already corrupted by the time numpy allocates; the faulty write happens earlier, which is why the
reported symptom varies.

Guess at the cause

With nthreads == 1, NumExpr_run presumably evaluates inline in the calling thread and reuses the per-object working buffers,
whereas with nthreads >= 2 the work goes through the thread pool and its per-thread buffers. Two Python threads entering
the same compiled object then write into the same scratch space. This is consistent with the threshold being exactly 2
threads, and with evaluate() being unaffected.

Workaround

Never let numexpr run with a single thread when compiled expressions are shared:

os.environ.setdefault("NUMEXPR_NUM_THREADS", "2")

Beware that numexpr falls back on OMP_NUM_THREADS when NUMEXPR_NUM_THREADS is unset, so OMP_NUM_THREADS=1 alone — a common setting in batch pipelines and in parallel test runners, which serialise each worker to avoid oversubscription — is enough to trigger this.

主要语言
Python
星标
2.5k
派生
235
PR 合并指标
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

pydata/numexpr 的其他 Issue

查看 pydata/numexpr 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。