Can't load source files when .pyc files have wrong filepaths inside.

No source files, tests, or entry points are named. Start by reproducing the shared-.pyc case with the provided localRoot and remoteRoot pathMappings, then trace how debugpy resolves source paths from .pyc files; done means source loads correctly or the failure explains the invalid path and fallback behavior.

Summary

debugpy + VSCode Python Debugger extension is unable to properly load source files when .pyc files are shared across multiple machines, on which the python source is stored at different filepaths.

Extended Details

I run my Python code sometimes locally (on my Macbook) and sometimes inside a docker container (in which I have my source mounted). I run VSCode on my Macbook, and I would like to be able to use VSCode to attach to and debug python programs (with debugpy) running on either my Macbook, or on the container.

Locally, the root of my project is at /Users/username/myproject. On the container, the root of the project is in /root.

My __pycache__ directories and .pyc files are also mounted, so the same cache files are shared between my local machine and the docker container.

.pyc files contain the absolute path of the source file they were generated from.

It appears debugpy (or VSCode Python Debugger extension?) is attempting to use filepaths stored in .pyc files, to look up source files. The observable behavior is if the .pyc file corresponding to the file I was debugging has the wrong filepath stored inside it, vs the filepath of the source file on that machine, then loading source in the debugger usually fails.

Specifically, if the .pyc file is generated on my Docker container and contains the source filepath /root/file.py, then when debugging on my local machine I get the error message:

Could not load source '/root/file.py': Unable to retrieve source for /root/file.py.

If the .pyc file is generated on my local machine and contains the source filepath /Users/username/myproject/file.py, then when debugging my code on my Docker container, I get the error message:

Could not load source '/Users/username/myproject/file.py': Unable to retrieve source for /Users/username/myproject/file.py.

In either case, if I delete the .pyc file, then rerun the program (which recreates the .pyc file with the right filepath), then debugging and loading the proper source files works properly.

Note this behavior may be somewhat tricky to reproduce, I see it pretty consistently attempting to debug the test suite of my large project, but wasn't able to get it working with a toy example. Sometimes instead of showing the error message, it instead shows bytecode.

Environment data

• debugpy version: 1.6.7.post1
• OS and version:
  • Macbook: macOS Sonoma 14.3 (Apple M2 Pro)
  • Container: Ubuntu 22.04.3 LTS x86_64
• Python version (& distribution if applicable, e.g. Anaconda): 3.11.6 (Anaconda)
• Using VS Code or Visual Studio: VS Code (Python Debugger v2024.0.0)

Debug launch.json

    "configurations": [
        {
            // Attach to any local process with debugpy listening on port 5678
            "name": "Attach to Local Process",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "127.0.0.1",
                "port": 5678
            },
            "justMyCode": true
        },
        {
            // Attach to a Container with this port forwarded 
            "name": "Attach to Container",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "127.0.0.1",
                "port": "5679"
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/root"
                }
            ],
            "justMyCode": true
        },

Workarounds

• Use python -B or PYTHONDONTWRITEBYTECODE to prevent writing .pyc files at all
• Delete pycache files when you encounter this issue.
• (Haven't done this yet but should work) don't share pycache files across different machines. Instead of pycache files being stored alongside source, extract them and store them in a separate directory using PYTHONPYCACHEPREFIX. Either: each machine has a separate set of python cache files (and they aren't shared), or set up two separate directories (ok to have them in the Mount), one directory for use by local and one for use by the docker container.
• I don't think there is any way to solve this issue using "pathMappings" config in launch.json.

Ask

debugpy & VSCode Python Debugger Extension should be resilient to .pyc files with incorrect filepaths. Perhaps one of these approaches would work?

• Don't use .pyc files to find source code
• If source not found at the location specified by .pyc file, fall back to the default behavior (when no .pyc file)
• Or even just a better error message! Figuring out what was happening here was tricky!