Python: allow building the sdist without the upb C extension (pure-Python only)

Open Beginner friendly
#31,654 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp, python
Domain
build-system

Research direction

Start by reading python/dist/setup.py, which is copied into the source distribution and declares the google._upb._message extension. Add the proposed environment variable opt-out there, then build the sdist with PROTOBUF_PYTHON_NO_EXTENSIONS=1 and confirm it succeeds without a compiler and produces a py3-none-any wheel containing the same google/protobuf modules as the published pure-Python wheel.

Written by the indexing model from the issue text.

Description

feature request packaging & distribution python

Since 4.21, the sdist's setup.py (copied from python/dist/setup.py) always declares the google._upb._message extension, so installing protobuf from source requires a C compiler. There is no way to opt out at build time. Protobuf 3.x built pure-Python by default and only compiled the C++ backend with --cpp_implementation.

At runtime the pure-Python backend is still fully supported: api_implementation falls back to it automatically when google._upb._message is missing. Only the build is missing an opt-out.

Use case: reproducible builds that install all dependencies from sdists (pip install --no-binary :all:) where no C compiler is available. Today the only option there is the py3-none-any wheel, which forces such projects to special-case protobuf in their source-only pipeline.

Proposal: an explicit environment variable that skips the extension:

ext_modules = [
    Extension(
        'google._upb._message',
        ...
    )
]
if os.environ.get('PROTOBUF_PYTHON_NO_EXTENSIONS'):
  ext_modules = []

setup(
    ...
    ext_modules=ext_modules,
    ...
)

With this change, PROTOBUF_PYTHON_NO_EXTENSIONS=1 pip wheel --no-binary :all: protobuf==7.36.2 builds without a compiler and produces a py3-none-any wheel with the same google/protobuf modules as the published pure-Python wheel.

An explicit opt-out is preferable to Extension(optional=True): a silent fallback would make the build output depend on whether a compiler happens to be present, which breaks reproducible builds.

Dominant language
C++
Stars
72.2k
Forks
16.4k
Avg merge
1d 8h
Merged PRs (30d)
342

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from protocolbuffers/protobuf

All issues in protocolbuffers/protobuf

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.