Linux ARM64 packaging omits worker modules and generates protos incompatible with bundled runtimes

Đang mở
  • Có gì không ổn?
#1,924 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 4 ngày

@hallvictoria đang làm issue này rồi.

Từ ngày 29/10/2026.

  • #1925 của @hallvictoria — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
grpc, python
Lĩnh vực
backend, build-system, testing

Hướng nghiên cứu

Start with eng/pack/scripts/nix_arm64_deps.sh and eng/pack/scripts/rc_nix_arm64_deps.sh to trace how worker files and generated protos enter the ARM64 payload. Then read eng/pack/templates/nix_arm64_env_gen.yml and inspect the packaged Python worker and runtime dependency versions. Done means both packaging paths include complete worker modules, compatible generated code, and validation that catches missing files or import failures.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

The Linux ARM64 packaging path produces incomplete Python worker payloads. Core Tools 4.14.0–4.16.0 ship worker directories containing generated protos but missing the executable worker modules.

Downstream report: https://github.com/Azure/azure-functions-core-tools/issues/5772

Affected versions

Core Tools release Python worker dependency Worker modules
4.13.0 4.44.0 Present
4.14.0 4.45.1 Missing
4.15.0–4.16.0 4.46.0 Missing

Inspection of 28 published Core Tools ZIP archives (seven OS/architecture combinations across these four releases) confirms the missing modules across every bundled Python version, 3.9–3.14, on Linux ARM64. The equivalent Linux x64, macOS x64/ARM64, and Windows x64/x86 archives retain the worker entry modules and dispatcher. Windows ARM64 does not bundle the Python worker and is a separate, pre-existing limitation.

These findings are based on published archive inspection and source tracing; local execution on ARM64 was reported in the downstream issue, not independently reproduced during this investigation.

Root cause

The ARM64 packaging rewrite in https://github.com/Azure/azure-functions-python-worker/pull/1845 (commit b991f234f0decd481544c52ac06e3b82b6ec3b4b) replaced installation of the local worker project with dependency downloads in eng/pack/scripts/nix_arm64_deps.sh and eng/pack/scripts/rc_nix_arm64_deps.sh:

pip download \
    --platform manylinux_2_17_aarch64 \
    --only-binary=:all: \
    --dest /tmp/arm64_wheels \
    .

The subsequent loop extracts only downloaded wheels. pip download . resolves/downloads dependencies but does not build/save the local source directory itself as a wheel. Pip's RequirementPreparer.save_linked_requirement explicitly skips copying a non-editable local directory. Therefore the worker package itself never reaches deps.

Later steps copy back only generated protos/ and, for the older worker, _vendored/. This matches the published incomplete payloads:

  • Python 3.9–3.12 lack azure_functions_worker/main.py, dispatcher.py, and other worker modules.
  • Python 3.13–3.14 lack proxy_worker/start_worker.py, dispatcher.py, and other worker modules.
  • No .pyc files remain in these worker packages as an executable substitute in the affected Linux ARM64 releases.

Relevant source: https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/scripts/nix_arm64_deps.sh#L24-L84

The launcher worker.py is still present. For Python 3.13/3.14 it imports proxy_worker.start_worker, so startup fails before the worker can connect to the host:

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/workers/python/proxyV4/worker.py

Additional generated-code/runtime mismatch

The ARM64 scripts install unpinned grpcio-tools into their build environment and generate protos there, independently of the constrained runtime dependencies downloaded for distribution:

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/scripts/nix_arm64_deps.sh#L15-L58

Direct inspection of the published Core Tools 4.16.0 Linux ARM64 archive confirmed:

Python Generated protobuf version Bundled protobuf runtime Generated gRPC requirement
3.11 7.35.0 5.29.6 grpcio >=1.82.1
3.14 7.35.0 6.33.6 grpcio >=1.82.1

The downstream reporter additionally records bundled grpcio 1.75.1 for Python 3.14 and the following failures after manually restoring missing worker modules:

VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading FunctionRpc.proto: gencode 7.35.0 runtime 6.33.6.

RuntimeError: The grpc package installed is at version 1.75.1, but the generated code in FunctionRpc_pb2_grpc.py depends on grpcio>=1.82.1.

Restoring worker source files alone therefore does not fully repair the payload.

Why artifact validation misses this

eng/pack/templates/nix_arm64_env_gen.yml checks the worker directories and generated proto files, but not required executable entry modules or successful worker/proto imports. A worker directory containing only protos can pass these checks.

https://github.com/Azure/azure-functions-python-worker/blob/4.46.0/eng/pack/templates/nix_arm64_env_gen.yml#L109-L141

Reproduction

  1. Install Core Tools 4.16.0 on Linux ARM64, using the release ZIP or .deb.
  2. Create a Python Functions app using Python 3.13 or 3.14.
  3. Run func start --verbose.

Reported failure:

Azure.Functions.Rpc.Server: python3 exited with code 1 (0x1). ImportError: cannot import name 'start_worker' from 'proxy_worker' (unknown location).

Archive-only verification:

unzip -l Azure.Functions.Cli.linux-arm64.4.16.0.zip \
  | grep 'python/3.14/LINUX/Arm64/proxy_worker/' \
  | grep -v protos

No worker implementation files are listed.

Expected resolution

Explicitly build/include the local worker package in both ARM64 packaging paths, align code-generation tools with packaged runtime versions, and validate imports/startup using matching ARM64 Python environments. Publish a corrected worker package for Core Tools to consume.

Acceptance criteria

  • Complete worker payloads are packaged for every supported Linux ARM64 Python version.
  • Worker entry modules and generated protobuf/gRPC modules import successfully using packaged dependencies.
  • An ARM64 Python function app starts, indexes, and executes successfully.
  • Validation rejects missing worker modules and incompatible generated-code/runtime combinations.
  • Both stable and RC ARM64 packaging paths are covered.
Ngôn ngữ chính
Python
Star
357
Fork
118
Merge trung bình
4 ngày 8 giờ
Pull request đã merge (30 ngày)
5

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Azure/azure-functions-python-worker

Tất cả issue của Azure/azure-functions-python-worker

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.