`importlib.util.LazyLoader` deadlocks when two threads first-touch modules in an import cycle

Start with the lazy-loading implementation in Lib/importlib/util.py and compare its locking behavior with the import system's module-lock handling described in the issue. Use the supplied two-thread repro to confirm the deadlock and guide a regression test with a timeout. Done means lazy circular imports no longer hang, including the mixed lazy/eager case, while eager-import behavior remains intact.

Bug report

_LazyModule.__getattribute__ serialises the lazy load with a per-module threading.RLock (loader_state['lock'], added in gh-114763). Unlike the import system's own module locks (_bootstrap._ModuleLock), a plain RLock has no deadlock detection.

So if two modules lazily loaded by LazyLoader use each other at import time, and two threads each trigger the load of a different one, the program deadlocks:

1. Thread 1 touches mod_a, takes mod_a's lock, and starts running mod_a.
2. Thread 2 touches mod_b, takes mod_b's lock, and starts running mod_b.
3. mod_a touches mod_b and blocks on mod_b's lock, while mod_b touches mod_a and blocks on mod_a's lock.

The is_loading check only handles re-entry from the same thread, so it doesn't help here.

The same modules imported eagerly don't deadlock. _ModuleLock.acquire() raises _DeadlockError, and _lock_unlock_module() responds by using the partially initialised module. That is the documented behaviour for circular imports.

"""
Two threads each first-touch one of two lazily loaded modules that use each other.

Usage: python repro.py [lazy|eager]
"""

import importlib
import importlib.util
import os
import pathlib
import sys
import tempfile
import threading

SOURCES = {
    # Shared by both modules, and imported normally.
    "sync": "import threading\nbarrier = threading.Barrier(2)\n",
    "mod_a": (
        "import sys, sync\n"
        "A = 1\n"
        "sync.barrier.wait()  # both threads are now loading a module\n"
        "FROM_B = sys.modules['mod_b'].B\n"
    ),
    "mod_b": (
        "import sys, sync\n"
        "B = 2\n"
        "sync.barrier.wait()\n"
        "FROM_A = sys.modules['mod_a'].A\n"
    ),
}


def lazy_import(name):
    # The recipe from the importlib docs.
    spec = importlib.util.find_spec(name)
    loader = importlib.util.LazyLoader(spec.loader)
    spec.loader = loader
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    loader.exec_module(module)
    return module


def main(mode):
    directory = tempfile.mkdtemp()
    for name, source in SOURCES.items():
        pathlib.Path(directory, f"{name}.py").write_text(source)
    sys.path.insert(0, directory)

    if mode == "lazy":
        mod_a, mod_b = lazy_import("mod_a"), lazy_import("mod_b")
        first_touch = [lambda: mod_a.FROM_B, lambda: mod_b.FROM_A]
    else:
        first_touch = [
            lambda: importlib.import_module("mod_a").FROM_B,
            lambda: importlib.import_module("mod_b").FROM_A,
        ]

    results = {}
    threads = [
        threading.Thread(target=lambda i=i, f=f: results.__setitem__(i, f()), daemon=True)
        for i, f in enumerate(first_touch)
    ]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join(timeout=5)

    hung = any(thread.is_alive() for thread in threads)
    print(f"{sys.version.split()[0]} {mode}:", "DEADLOCKED" if hung else f"ok {results}", flush=True)
    # The deadlocked threads would block interpreter shutdown.
    os._exit(1 if hung else 0)


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "lazy")

Output (lazy mode deadlocks every time; eager mode never does):

3.12.13 lazy: DEADLOCKED
3.12.13 eager: ok {1: 1, 0: 2}
3.13.11 lazy: DEADLOCKED
3.13.11 eager: ok {1: 1, 0: 2}
3.14.7 lazy: DEADLOCKED
3.14.7 eager: ok {1: 1, 0: 2}
3.15.0a6 lazy: DEADLOCKED
3.15.0a6 eager: ok {1: 1, 0: 2}

The barrier only makes the timing deterministic. Without it, the hang is intermittent.

The code on main is unchanged.

Suggested fix

Use the import system's module lock for the lazy load too. On a _DeadlockError, proceed without the lock, as _lock_unlock_module() does. The thread that holds the lock is then already loading the module, so the existing is_loading branch returns the partially initialised module, the same result as an eager circular import.

This also removes the deferred import threading from exec_module().

--- a/Lib/importlib/util.py
+++ b/Lib/importlib/util.py
@@ -1,5 +1,6 @@
 """Utility code for constructing importers, etc."""
 
+from . import _bootstrap
 from ._bootstrap import _resolve_name
 from ._bootstrap import _find_spec
 
@@ -168,6 +169,33 @@
         return -1 if self.disable_check else 1
 
 
+class _LazyModuleLock:
+
+    """Hold the import system's lock for a module while it loads lazily.
+
+    Unlike a plain lock, the module lock detects a deadlock between threads
+    that are loading each other's modules. On a deadlock, proceed without the
+    lock as _bootstrap._lock_unlock_module() does for eager imports, so that
+    the partially initialized module is returned.
+    """
+
+    def __init__(self, name):
+        self._lock = _bootstrap._get_module_lock(name)
+        self._acquired = False
+
+    def __enter__(self):
+        try:
+            self._lock.acquire()
+        except _bootstrap._DeadlockError:
+            pass
+        else:
+            self._acquired = True
+
+    def __exit__(self, *args):
+        if self._acquired:
+            self._lock.release()
+
+
 class _LazyModule(types.ModuleType):
 
     """A subclass of the module type which triggers loading upon attribute access."""
@@ -176,7 +204,7 @@
         """Trigger the load of the module and return the attribute."""
         __spec__ = object.__getattribute__(self, '__spec__')
         loader_state = __spec__.loader_state
-        with loader_state['lock']:
+        with _LazyModuleLock(__spec__.name):
             # Only the first thread to get the lock should trigger the load
             # and reset the module's class. The rest can now getattr().
             if object.__getattribute__(self, '__class__') is _LazyModule:
@@ -259,9 +287,6 @@
 
     def exec_module(self, module):
         """Make the module load lazily."""
-        # Threading is only needed for lazy loading, and importlib.util can
-        # be pulled in at interpreter startup, so defer until needed.
-        import threading
         module.__spec__.loader = self.loader
         module.__loader__ = self.loader
         # Don't need to worry about deep-copying as trying to set an attribute
@@ -271,7 +296,6 @@
         loader_state = {}
         loader_state['__dict__'] = module.__dict__.copy()
         loader_state['__class__'] = module.__class__
-        loader_state['lock'] = threading.RLock()
         loader_state['is_loading'] = False
         module.__spec__.loader_state = loader_state
         module.__class__ = _LazyModule

With main's Lib/importlib/util.py patched this way and loaded into 3.13.11, 3.14.7 and 3.15.0a6, the repro passes 15 of 15 runs (ok {0: 2, 1: 1}), compared with 15 of 15 deadlocks without the patch.

Sharing the eager module lock also covers a mixed case: one thread lazily loading mod_a while another eagerly imports mod_b.

A regression test could follow the repro: a threading.Barrier(2) in a helper module, two lazily loaded modules in a cycle, and a join with a timeout.

I'm happy to open a PR.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15 (3.15.0a6), CPython main branch (by inspection)

Operating systems tested on:

macOS