python arm build issue still failing in some cases

Tái hiện bản build arm64 với py7zr bằng sam build -u --debug, sau đó kiểm tra luồng PythonPipBuilder trong workflows/python_pip/actions.py và packager.py, đặc biệt là việc phân giải dependency và xử lý wheel bị thiếu. Công việc được xem là hoàn tất khi cùng một tập dependency hoàn thành bản build arm64 mà không có MissingDependencyError đối với zipfile-deflate64.

Description:

python lambda function builds using the sam cli (which calls this library) are still failing for certain dependencies

Steps to reproduce the issue:

1. setup python project with sam cli
2. add py7zr to the requirements.txt of a function
3. run command sam build -u

Observed result:
build fails on resolve dependencies
Expected result:
build completes
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws codebuild
image
aws/codebuild/amazonlinux2-x86_64-standard:3.0
buildspec.yaml:
version: 0.2
env:
secrets-manager:
DOCKER_PASSWORD: *******
DOCKER_USER: ******
phases:
install:
runtime-versions:
python: 3.9
build:
commands:
- printenv
- pip install --upgrade aws-sam-cli
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- echo $DOCKER_PASSWORD | docker login --username $DOCKER_USER --password-stdin
- sam build -u --debug
- sam package --s3-bucket ******i --output-template-file template-out.yaml
artifacts:
files:
- template-out.yaml

sam build -u --debug output near failure:

2022-03-22 18:35:43,680 | Building codeuri: /codebuild/output/src531839338/src/lambda/******** runtime: python3.9 metadata: {} architecture: arm64 functions: ['*****']
2022-03-22 18:35:43,680 | Building to following folder /codebuild/output/src531839338/src/.aws-sam/build****

Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image......
2022-03-22 18:35:44,617 | Mounting /codebuild/output/src531839338/src/lambda/****** as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(sdist), texttable==1.6.4(wheel), zipfile-deflate64==0.0.0(sdist), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial compatible: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial incompatible: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
Downloading missing wheels: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch pycryptodomex==3.14.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch zipfile-deflate64==0.0.0
compatible wheels after second download pass: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
Build missing wheels from sdists (C compiling True): {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycryptodomex-3.14.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (no C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Build missing wheels from sdists (C compiling False): {zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final compatible: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final incompatible: set()
Final missing wheels: {zipfile-deflate64==0.0.0(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}
Builder workflow failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/main.py", line 118, in main
builder.build(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build
return workflow.run()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper
func(self, *args, **kwargs)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 309, in run
raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}
2022-03-22 18:43:02,875 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}"}}

Build Failed
2022-03-22 18:43:03,179 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '64785e7d-a17c-4f7f-a347-aca7b7adc804', 'installationId': '75a7108d-f9a0-4aa3-ad9a-b637b8ea0647', 'sessionId': 'c7ec421f-a00b-47f1-b1e6-b34189e99768', 'executionEnvironment': 'AWSCodeBuild', 'ci': True, 'pyversion': '3.9.5', 'samcliVersion': '1.41.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 667631, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2022-03-22 18:43:03,217 | Telemetry response: 200
Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}

[Container] 2022/03/22 18:43:03 Command did not exit successfully sam build -u --debug exit status 1
[Container] 2022/03/22 18:43:03 Phase complete: BUILD State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build -u --debug. Reason: exit status 1
[Container] 2022/03/22 18:43:03 Entering phase POST_BUILD
[Container] 2022/03/22 18:43:03 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/03/22 18:43:03 Phase context status code: Message:
[Container] 2022/03/22 18:43:03 Expanding base directory path: .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding .
[Container] 2022/03/22 18:43:03 Expanding file paths for base directory .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding template-out.yaml
[Container] 2022/03/22 18:43:03 Skipping invalid file path template-out.yaml
[Container] 2022/03/22 18:43:03 Phase complete: UPLOAD_ARTIFACTS State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: CLIENT_ERROR Message: no matching artifact paths found