`set_trace`-releated deadlock when running inside Ansible with explicit listen-and-debug breakpoint

Reproduce the issue with the Ansible lookup plugin containing the debugpy listen-and-wait stanza, then compare it with the provided hello-world script. Start at debugpy/_vendored/pydevd/pydevd.py around wait_for_ready_to_run() and use the attached launch.json for the client configuration. Done means attaching during the Ansible run no longer leaves Variables spinning or debugger commands timing out.

Environment data

• debugpy version: 1.8.1 (run import debugpy; print(debugpy.__version__) if uncertain)
• OS and version: Mac OS X Sonoma 14.2.1 (23C71)
• Python version: Python 3.11.7 that comes with Mac OS X
• Using VS Code version 1.86.1 (likely irrelevant, as I observe the same behavior with Emacs and dape.el)

Actual behavior

Upon attempting to attach the client,

• Variables → Locals keeps spinning inside VS Code
• Debugger is catatonic; every command times out (same when tried out of Emacs)
• Pressing C-c yields the following stack trace (even after waiting for several minutes):

^CTraceback (most recent call last):
  File "/Users/quatrava/Dev/ops/xaasible-ops/ansible-deps-cache/python-libs/lib/python/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 3489, in <module>
 [ERROR]: User interrupted execution
    main()
  File "/Users/quatrava/Dev/ops/xaasible-ops/ansible-deps-cache/python-libs/lib/python/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 3482, in main
    globals = debugger.run(setup['file'], None, None, is_module)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/quatrava/Dev/ops/xaasible-ops/ansible-deps-cache/python-libs/lib/python/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 2476, in run
    self.wait_for_ready_to_run()
  File "/Users/quatrava/Dev/ops/xaasible-ops/ansible-deps-cache/python-libs/lib/python/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 838, in wait_for_ready_to_run
    self._py_db_command_thread_event.wait(0.1)
  File "/usr/local/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 629, in wait
    signaled = self._cond.wait(timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331, in wait
    gotit = waiter.acquire(True, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Expected behavior

Debugger session should proceed as usual, even though some slow JSON-RPCs are expected (i.e. because my test case has large visible variables). Note that that is exactly what happens if instead of running Ansible, I run a hello-world Python script that features the same debugpy stanza to await debugger connect, i.e.

import debugpy; debugpy.listen(5678); debugpy.wait_for_client(); debugpy.breakpoint()

print("hello world")

Steps to reproduce:

1. Configure VS Code as per the documentation. My .vscode/launch.json is attached below
2. Install Ansible
3. Create lookup_plugins/my_python_lookup.py and add the following code inside:

   from ansible.plugins.lookup import LookupBase
   
   class LookupModule(LookupBase):
       def run(self, terms, variables=None, **kwargs):
           import debugpy; debugpy.listen(5678); debugpy.wait_for_client(); debugpy.breakpoint()
   
           return ["toto"]
   

4. Prepare a trivial Ansible playbook that calls '{{ lookup("my_python_lookup") }}' somewhere
5. Run the Ansible playbook

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Débogueur Python : Attachement à distance",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "."
                }
            ]
        }
    ]
}