pybind/pybind11

[BUG] pybind11::cast is not thread safe

Open

#2765 opened on Dec 31, 2020

View on GitHub
 (21 comments) (0 reactions) (0 assignees)C++ (14,677 stars) (2,005 forks)batch import
bugcastershelp wanted

Description

in particular use of loader_life_support seems problematic. This function keeps state in the global loader_patient_stack: https://github.com/pybind/pybind11/blob/b7dfe5cc84ca1891b50e728e83c9b5b393cf5272/include/pybind11/detail/internals.h#L105 In the case where a cast can release the GIL, two thread's push/pop of loader_patient_stack can become interleaved.

While releasing the GIL in a conversion seems like a bad idea, it can be surprisingly hard to avoid. Any cast that results in the bytecode interpreter being run (either directly or through something like a __del__ method) can cause python itself to release GIL (sys.setswitchinterval).

Contributor guide

Tech stack
cpppython
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
C++Pythonthreadingpybind11 internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
5
Research direction
The issue is in the `loader patient stack` global state in `include/pybind11/detail/internals.h`. Research possible solutions such as using thread local storage or adding a mutex lock. Consider the implications for GIL release during casting. Look at existing discussion in comments for proposed approaches.