Linux ARM64 packaging omits worker modules and generates protos incompatible with bundled runtimes

Aperta
  • Qualcosa non va?
#1,924 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 4 giorni

@hallvictoria ci sta già lavorando.

Dal 29/10/2026.

  • #1925 di @hallvictoria — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
grpc, python
Ambito
backend, build-system, testing

Direzione di ricerca

Start with eng/pack/scripts/nix_arm64_deps.sh and eng/pack/scripts/rc_nix_arm64_deps.sh to trace how worker files and generated protos enter the ARM64 payload. Then read eng/pack/templates/nix_arm64_env_gen.yml and inspect the packaged Python worker and runtime dependency versions. Done means both packaging paths include complete worker modules, compatible generated code, and validation that catches missing files or import failures.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The Linux ARM64 packaging path produces incomplete Python worker payloads. Core Tools 4.14.0–4.16.0 ship worker directories containing generated protos but missing the executable worker modules.

Downstream report: https://github.com/Azure/azure-functions-core-tools/issues/5772

Affected versions

Core Tools release Python worker dependency Worker modules
4.13.0 4.44.0 Present
4.14.0 4.45.1 Missing
4.15.0–4.16.0 4.46.0 Missing

Inspection of 28 published Core Tools ZIP archives (seven OS/architecture combinations across these four releases) confirms the missing modules across every bundled Python version, 3.9–3.14, on Linux ARM64. The equivalent Linux x64, macOS x64/ARM64, and Windows x64/x86 archives retain the worker entry modules and dispatcher. Windows ARM64 does not bundle the Python worker and is a separate, pre-existing limitation.

These findings are based on published archive inspection and source tracing; local execution on ARM64 was reported in the downstream issue, not independently reproduced during this investigation.

Root cause

The ARM64 packaging rewrite in https://github.com/Azure/azure-functions-python-worker/pull/1845 (commit b991f234f0decd481544c52ac06e3b82b6ec3b4b) replaced installation of the local worker project with dependency downloads in eng/pack/scripts/nix_arm64_deps.sh and eng/pack/scripts/rc_nix_arm64_deps.sh:

pip download \
    --platform manylinux_2_17_aarch64 \
    --only-binary=:all: \
    --dest /tmp/arm64_wheels \
    .

The subsequent loop extracts only downloaded wheels. pip download . resolves/downloads dependencies but does not build/save the local source directory itself as a wheel. Pip's RequirementPreparer.save_linked_requirement explicitly skips copying a non-editable local directory. Therefore the worker package itself never reaches deps.

Later steps copy back only generated protos/ and, for the older worker, _vendored/. This matches the published incomplete payloads:

  • Python 3.9–3.12 lack azure_functions_worker/main.py, dispatcher.py, and other worker modules.
  • Python 3.13–3.14 lack proxy_worker/start_worker.py, dispatcher.py, and other worker modules.
  • No .pyc files remain in these worker packages as an executable substitute in the affected Linux ARM64 releases.

Relevant source: https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/scripts/nix_arm64_deps.sh#L24-L84

The launcher worker.py is still present. For Python 3.13/3.14 it imports proxy_worker.start_worker, so startup fails before the worker can connect to the host:

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/workers/python/proxyV4/worker.py

Additional generated-code/runtime mismatch

The ARM64 scripts install unpinned grpcio-tools into their build environment and generate protos there, independently of the constrained runtime dependencies downloaded for distribution:

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/scripts/nix_arm64_deps.sh#L15-L58

Direct inspection of the published Core Tools 4.16.0 Linux ARM64 archive confirmed:

Python Generated protobuf version Bundled protobuf runtime Generated gRPC requirement
3.11 7.35.0 5.29.6 grpcio >=1.82.1
3.14 7.35.0 6.33.6 grpcio >=1.82.1

The downstream reporter additionally records bundled grpcio 1.75.1 for Python 3.14 and the following failures after manually restoring missing worker modules:

VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading FunctionRpc.proto: gencode 7.35.0 runtime 6.33.6.

RuntimeError: The grpc package installed is at version 1.75.1, but the generated code in FunctionRpc_pb2_grpc.py depends on grpcio>=1.82.1.

Restoring worker source files alone therefore does not fully repair the payload.

Why artifact validation misses this

eng/pack/templates/nix_arm64_env_gen.yml checks the worker directories and generated proto files, but not required executable entry modules or successful worker/proto imports. A worker directory containing only protos can pass these checks.

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/templates/nix_arm64_env_gen.yml#L109-L141

Reproduction

  1. Install Core Tools 4.16.0 on Linux ARM64, using the release ZIP or .deb.
  2. Create a Python Functions app using Python 3.13 or 3.14.
  3. Run func start --verbose.

Reported failure:

Azure.Functions.Rpc.Server: python3 exited with code 1 (0x1). ImportError: cannot import name 'start_worker' from 'proxy_worker' (unknown location).

Archive-only verification:

unzip -l Azure.Functions.Cli.linux-arm64.4.16.0.zip \
  | grep 'python/3.14/LINUX/Arm64/proxy_worker/' \
  | grep -v protos

No worker implementation files are listed.

Expected resolution

Explicitly build/include the local worker package in both ARM64 packaging paths, align code-generation tools with packaged runtime versions, and validate imports/startup using matching ARM64 Python environments. Publish a corrected worker package for Core Tools to consume.

Acceptance criteria

  • Complete worker payloads are packaged for every supported Linux ARM64 Python version.
  • Worker entry modules and generated protobuf/gRPC modules import successfully using packaged dependencies.
  • An ARM64 Python function app starts, indexes, and executes successfully.
  • Validation rejects missing worker modules and incompatible generated-code/runtime combinations.
  • Both stable and RC ARM64 packaging paths are covered.
Lingua principale
Python
Stelle
357
Fork
118
Merge medio
4g 8h
PR unite (30g)
5

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Azure/azure-functions-python-worker

Tutte le issue di Azure/azure-functions-python-worker

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.