Threads.@spawn can fatally crash PyCall @async task

Tái hiện script được cung cấp và so sánh các trường hợp Threads.@spawn và Threads.@threads. Bắt đầu bằng cách lần theo các đường dẫn PyCall được nêu trong src/pyinit.jl, src/pybuffer.jl, src/conversions.jl, src/callback.jl và src/pyfncall.jl, cùng với các issue #882 và #883. Công việc được xem là hoàn tất khi ranh giới an toàn luồng được ghi lại hoặc được bao phủ bởi một regression test tập trung.

I'm trying to use PyCall in a Julia solution for various IO tasks (such as handling Parquet files and interacting with various Azure resources). I would like to have these PyCall tasks run in the background on one thread, doing a bunch of IO work using a fancy Python SDK while I do multithreaded number-crunching in Julia. I'm running into a weird issue though. With this script I wrote, I'm getting a fatal error if I run the @sync python task while running a Threads.@spawn task, but everything is fine if I use Threads.@threads. Is there any reason why @Threads works while @Spawn doesn't? Is there a way to make this safe with Threads.@spawn because we can't know, unless we look at the code, if an external library uses Threads.@spawn under the hood.

I'm going through all the docs of PyCall and looking at issues with multi-threading (and there are a few of them that discuss this, like #882 and #883) but we need a better understanding of what we can and can't do in Julia while PyCall is doing something. I have a script that puts locks around the python process, and only executes Python as an @async, from the main thread so it should always happen on the main thread as suggested in #882 (I think?). Anyway, I'm getting fatal errors in Windows (and segfaults in Linux) when I try to run the script where a Threads.@spawn happens while PyCall is running, but it's fine if I use a Threads.@threads

Anyway, here's the script:

using PyCall
const PY_LOCK = ReentrantLock()
const PY_JSON = pyimport("json")


const PYLOCK = Ref{ReentrantLock}()
PYLOCK[] = ReentrantLock()

# acquire the lock before any code calls Python
pylock(f::Function) = Base.lock(f, PYLOCK[])


function write_json_file(fileName::String, outputData::Dict)
    pylock() do
        open(fileName, "w") do outputFile
            PY_JSON.dump(deepcopy(outputData), outputFile)
        end
    end
    return nothing
end

function file_operation(fileName::String)
    outputData = Dict("a"=>randn(), "b"=>randn())
    write_json_file(fileName, outputData)
    return outputData
end

function multithread_calc(x::Vector)
    y = zeros(Float64, length(x))
    Threads.@threads for ii in eachindex(y)
        y[ii] = log(exp(x[ii]))
    end
    return y
end

function calc_task(x::Vector)
    t = Threads.@spawn log.(exp.(x))
    return fetch(t)
end

calcInput = randn(10000000)
for ii in 1:100
    display(ii)
    file_operation("testfile.json")
    fileTasks   = [@async file_operation("testfile$(ii).json") for ii in 1:30]
    #calcResults =  multithread_calc(calcInput)
    calcResults = calc_task(calcInput)
    wait.(fileTasks)
end

Now, if I modify this script so that it executes a multithreaded calculation using Threads.@threads in multithread_calc() I get no issue

calcInput = randn(10000000)
for ii in 1:100
    display(ii)
    file_operation("testfile.json")
    fileTasks   = [@async file_operation("testfile$(ii).json") for ii in 1:30]
    calcResults =  multithread_calc(calcInput)
    #calcResults = calc_task(calcInput)
    wait.(fileTasks)
end

Otherwise, if I use the calc_task() function that has Threads.@spawn while running the Python tasks, I get the following error.

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffa4e385adb -- PyUnicode_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
in expression starting at g:\My Drive\tests\julia_pylock_json.jl:42
PyUnicode_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicode_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyLong_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicode_DecodeUTF8Stateful at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicode_FromId at C:\Users\user\Miniconda3\python39.dll (unknown line)
Py_FinalizeEx at C:\Users\user\Miniconda3\python39.dll (unknown line)
Py_FinalizeEx at C:\Users\user\Miniconda3\python39.dll (unknown line)
Py_Finalize at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyinit.jl:125
unknown function (ip: 00000000618a7a23)
_atexit at .\initdefs.jl:372
unknown function (ip: 00000000618a7013)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
ijl_atexit_hook at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:219
ijl_exit at /cygdrive/c/buildbot/worker/package_win64/build/src\jl_uv.c:640
jl_exception_handler at /cygdrive/c/buildbot/worker/package_win64/build/src\signals-win.c:322
__julia_personality at /cygdrive/c/buildbot/worker/package_win64/build/src\win32_ucontext.c:28
_chkstk at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlRaiseException at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
KiUserExceptionDispatcher at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
PyUnicode_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicode_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyLong_New at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicodeWriter_WriteASCIIString at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyUnicode_FromFormatV at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyErr_Format at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyObject_GetBuffer at C:\Users\user\Miniconda3\python39.dll (unknown line)
isbuftype! at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pybuffer.jl:134 [inlined]
isbuftype at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pybuffer.jl:148 [inlined]
pysequence_query at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:759
pytype_query at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:773
#36 at .\none:0 [inlined]
iterate at .\generator.jl:47
unknown function (ip: 000000006189dc50)
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:703
typetuple at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:745
unknown function (ip: 000000006189cd9a)
pysequence_query at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:754
pytype_query at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:773
pytype_query at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:806 [inlined]
convert at C:\Users\user\.julia\packages\PyCall\ygXW2\src\conversions.jl:831
julia_args at C:\Users\user\.julia\packages\PyCall\ygXW2\src\callback.jl:18 [inlined]
_pyjlwrap_call at C:\Users\user\.julia\packages\PyCall\ygXW2\src\callback.jl:24
unknown function (ip: 000000006189cc0a)
pyjlwrap_call at C:\Users\user\.julia\packages\PyCall\ygXW2\src\callback.jl:44
unknown function (ip: 000000006187c398)
PyObject_Call at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyFunction_Vectorcall at C:\Users\user\Miniconda3\python39.dll (unknown line)
Py_NewReference at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyFunction_Vectorcall at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyFunction_Vectorcall at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyVectorcall_Call at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyObject_Call at C:\Users\user\Miniconda3\python39.dll (unknown line)
macro expansion at C:\Users\user\.julia\packages\PyCall\ygXW2\src\exception.jl:95 [inlined]
#107 at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:43 [inlined]
disable_sigint at .\c.jl:473 [inlined]
__pycall! at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:42 [inlined]
_pycall! at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:29
_pycall! at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:11
unknown function (ip: 000000006188c5f5)
#_#114 at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:730
PyObject at C:\Users\user\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86
#2 at g:\My Drive\tests\julia_pylock_json.jl:16484
#open#378 at .\io.jl:384
open at .\io.jl:381 [inlined]
#1 at g:\My Drive\tests\julia_pylock_json.jl:15 [inlined]
lock at .\lock.jl:185
pylock at g:\My Drive\tests\julia_pylock_json.jl:10 [inlined]
write_json_file at g:\My Drive\tests\julia_pylock_json.jl:14 [inlined]
file_operation at g:\My Drive\tests\julia_pylock_json.jl:24
#11 at .\task.jl:484
unknown function (ip: 00000000618a5de3)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:931
Allocations: 10783290 (Pool: 10776630; Big: 6660); GC: 14