Various tests fail due to starting kernel on wrong Python version

ipykernel/tests/test_async.py で失敗を再現し、特に start_new_kernel に注目して ipykernel/tests/utils.py を調査します。呼び出しを jupyter_client の manager.start_new_kernel まで追い、pytest を実行している Python 実行ファイルと比較します。影響を受けるテストが意図したインタープリターでカーネルを起動し、バージョンの不一致が原因で失敗しなくなれば完了です。

With ipykernel-6.0.1:

$ PYTHONPATH=. python3.8 -m pytest -x ipykernel/tests/test_async.py 
========================================================= test session starts =========================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /tmp/ipykernel-6.0.1
plugins: timeout-1.4.2, case-1.5.3, xprocess-0.17.1, mock-3.6.1, localserver-0.5.0, expect-1.1.0, httpx-0.12.0, asyncio-0.15.1, snapshottest-0.5.1, forked-1.3.0, freezegun-0.4.2, datadir-1.3.1, metadata-1.8.0, requests-mock-1.9.3, subtests-0.5.0, betamax-0.8.1, regressions-2.2.0, pyfakefs-4.5.0, httpbin-1.0.0, django-4.4.0, shutil-1.7.0, virtualenv-1.7.0, subtesthack-0.1.2, pkgcore-0.12.1, xdist-2.3.0, pylama-7.7.1, cov-2.12.1, xdoctest-0.15.4, anyio-3.2.1, hypothesis-6.14.1, rerunfailures-10.1, flaky-3.7.0, services-2.2.1, aiohttp-0.3.0, Faker-8.10.0, lazy-fixture-0.6.3, xvfb-2.0.0, trio-0.7.0
collected 4 items                                                                                                                     

ipykernel/tests/test_async.py E

=============================================================== ERRORS ================================================================
_________________________________________________ ERROR at setup of test_async_await __________________________________________________

    def setup_function():
        """start the global kernel (if it isn't running) and return its client"""
        global KM, KC
>       KM, KC = start_new_kernel()

ipykernel/tests/test_async.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ipykernel/tests/utils.py:37: in start_new_kernel
    return manager.start_new_kernel(startup_timeout=STARTUP_TIMEOUT, **kwargs)
/usr/lib/python3.8/site-packages/jupyter_client/manager.py:919: in start_new_kernel
    kc.wait_for_ready(timeout=startup_timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <jupyter_client.blocking.client.BlockingKernelClient object at 0x7fc81942b430>, timeout = 60

    def wait_for_ready(self, timeout=None):
        """Waits for a response when a client is blocked
    
        - Sets future time for timeout
        - Blocks on shell channel until a message is received
        - Exit if the kernel has died
        - If client times out before receiving a message from the kernel, send RuntimeError
        - Flush the IOPub channel
        """
        if timeout is None:
            abs_timeout = float('inf')
        else:
            abs_timeout = time.time() + timeout
    
        from ..manager import KernelManager
        if not isinstance(self.parent, KernelManager):
            # This Client was not created by a KernelManager,
            # so wait for kernel to become responsive to heartbeats
            # before checking for kernel_info reply
            while not self.is_alive():
                if time.time() > abs_timeout:
                    raise RuntimeError("Kernel didn't respond to heartbeats in %d seconds and timed out" % timeout)
                time.sleep(0.2)
    
        # Wait for kernel info reply on shell channel
        while True:
            self.kernel_info()
            try:
                msg = self.shell_channel.get_msg(block=True, timeout=1)
            except Empty:
                pass
            else:
                if msg['msg_type'] == 'kernel_info_reply':
                    # Checking that IOPub is connected. If it is not connected, start over.
                    try:
                        self.iopub_channel.get_msg(block=True, timeout=0.2)
                    except Empty:
                        pass
                    else:
                        self._handle_kernel_info_reply(msg)
                        break
    
            if not self.is_alive():
>               raise RuntimeError('Kernel died before replying to kernel_info')
E               RuntimeError: Kernel died before replying to kernel_info

/usr/lib/python3.8/site-packages/jupyter_client/blocking/client.py:111: RuntimeError
-------------------------------------------------------- Captured stdout setup --------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ipykernel-6.0.1/ipykernel_launcher.py", line 15, in <module>
    from ipykernel import kernelapp as app
  File "/tmp/ipykernel-6.0.1/ipykernel/kernelapp.py", line 42, in <module>
    from .ipkernel import IPythonKernel
  File "/tmp/ipykernel-6.0.1/ipykernel/ipkernel.py", line 20, in <module>
    from .debugger import Debugger
  File "/tmp/ipykernel-6.0.1/ipykernel/debugger.py", line 14, in <module>
    import debugpy
ModuleNotFoundError: No module named 'debugpy'
======================================================= short test summary info =======================================================
ERROR ipykernel/tests/test_async.py::test_async_await - RuntimeError: Kernel died before replying to kernel_info
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================== 1 error in 2.83s ===========================================================

If add some debug, it turns out that the underlying cause is that the kernel is started using python3.10 rather than python3.8 (and debugpy doesn't support py3.10). This might be a problem in jupyter_client (version 6.1.12 here) but figuring out where it forces python3.10 is above my pay grade.