Recursive @threadpool_limits.wrap() calls do not behave as expected when recursing

已关闭
  • 哪里不对？
#293 3 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

已经有一个关联 PR 被合并了。

  • #294 来自 @itamarst —— 已合并

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
performance

调研方向

The failing case is the test test_recursive_wrap_releases_the_shared_pool_claim, which needs a non-OpenMP OpenBLAS build with at least two threads. Start by running it with recurse(1) and reading how threadpool_limits.wrap restores limits and releases the controlling thread on return. Done when the test passes with recurse(1) and the existing recursion behaviour still holds.

由索引模型根据 Issue 内容生成。

描述

Returning from a wrapped function should return to the previous level of thread limits at the time they were called, even in the face of recursion. That's not what happens in practice, though.

The following test fails on both master and 3.7 (i.e. it's not due to the controlling thread changes), but passes if recurse(1) is switched to recurse(0).

from threading import Thread

from threadpoolctl import threadpool_limits, ThreadpoolController

import numpy
import pytest

def run_in_thread(f, *args, **kwargs):
    thread = Thread(target=lambda: f(*args, **kwargs))
    thread.start()
    thread.join()
    return thread


def _non_openmp_openblas():
    openblas = ThreadpoolController().select(internal_api="openblas")
    if not openblas:
        pytest.skip("requires OpenBLAS.")
    libs = []
    seen = set()
    for lib in openblas.lib_controllers:
        if lib.threading_layer == "openmp" or lib.filepath in seen:
            continue
        seen.add(lib.filepath)
        libs.append(lib)
    if not libs:
        pytest.skip("require non-OpenMP OpenBLAS.")
    return libs


def _require_resizable_non_openmp_openblas():
    libs = _non_openmp_openblas()
    lib = libs[0]
    orig = lib.get_num_threads()
    if orig is None or orig < 2:
        pytest.skip("Test requires a non-OpenMP OpenBLAS with at least 2 threads")
    return lib, orig


def test_recursive_wrap_releases_the_shared_pool_claim():
    # One wrapped function calling itself must keep the limit between the inner
    # return and the outer return, then restore the original count and drop the
    # controlling thread. Another thread can then set the count.
    lib, orig = _require_resizable_non_openmp_openblas()

    @threadpool_limits.wrap(limits=1, user_api="blas")
    def recurse(depth):
        assert lib.get_num_threads() == 1
        if depth:
            recurse(depth - 1)
            assert lib.get_num_threads() == 1

    recurse(1)
    assert lib.get_num_threads() == orig

    observed = {}

    def other_thread():
        lib.set_num_threads(1)
        observed["threads"] = lib.get_num_threads()

    worker = run_in_thread(other_thread)
    assert observed["threads"] == 1
主要语言
Python
星标
434
派生
38
平均合并
1 天 4 小时
30 天内合并 PR
18

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

joblib/threadpoolctl 的其他 Issue

查看 joblib/threadpoolctl 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。