Nondeterministc deploy bug with 5+ different non-descript failures on two different OS's, worked before, support stumped

Open
#1,306 12 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Assessment

This issue has not been assessed yet.

Description

area:python-functions needs-attention :wave:

Unsure how to title this. It's been a week of debugging and isolating with no results.
Is this issue in the correct repo? Also unsure. Help me out here.
I'll walk you through my journey.

I have a very large code base, it runs flawlessly locally. I've deployed it just fine for a long time until recently.
I made a change, including google-cloud-texttospeech in requirements.txt.
It stopped deploying after this (maybe relevant, maybe not).
Removing this change, to the exact same code base as before, still fails to deploy.

Some errors I get at random, and I've tried incredibly hard to isolate.
Both development environments can deploy other things successfully, and push to a new app fine as well.
I have not made any local environment changes at all when it began to fail.

Details:
Azure Functions runtime version 4.24.4.4, Linux premium plan with Elastic Premium EP3.
I deploy directly from visual studio code azure function extension.
My python code follows the folder structure of the v1 coding model (no decorators, lots of folders), however I am on v2 (host.json etc.), this has always worked, and runs locally of course.
Deploying to python 3.9.7.

First development environment:
Manjaro linux, vscode 1.81.1, azure extension 1.12.3

Errors I've encountered seemingly at random, when trying to deploy:

2:50:41 PM debugApp123: **Deployment successful**. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
2:51:07 PM debugApp123: Syncing triggers...
2:51:12 PM debugApp123: Querying triggers...
2:51:18 PM debugApp123: **No HTTP triggers found.**
3:34:11 PM debugApp123: Deployment Failed. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
3:34:25 PM debugApp123: Deployment failed.
4:00:49 PM: Error: The operation was aborted.
10:52:10 AM ae-API-compute: Deployment successful. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
10:52:42 AM ae-API-compute: Syncing triggers...
10:52:49 AM ae-API-compute: Querying triggers...
10:52:52 AM ae-API-compute: WARNING: Some http trigger urls cannot be displayed in the output window because they require an authentication token. Instead, you may copy them from the Azure Functions explorer.

Except it was not successful and functions are empty / don't run (the zip file in webjobstorage contains them thought).

11:55:04 AM debugApp123: Syncing triggers...
11:55:45 AM debugApp123: Syncing triggers (Attempt 2/6)...
11:55:56 AM debugApp123: Syncing triggers (Attempt 3/6)...
11:56:17 AM debugApp123: Syncing triggers (Attempt 4/6)...
11:56:59 AM debugApp123: Syncing triggers (Attempt 5/6)...
11:58:20 AM debugApp123: Syncing triggers (Attempt 6/6)...
11:59:12 AM: Error: Encountered an error (ServiceUnavailable) from host runtime.

And a very exciting shiny rare one:

Offset to Central Directory cannot be held in an Int64.

Second development environment:
Windows, visual studio code, azure extension v1.12.4.
I've also used this environment before, and it has worked, same as the above one.

Errors:

One successful deploy, and randomly most of the above (with no changes), as well as a new one:

10:35:34 AM ae-API-compute: Starting deployment...
10:35:34 AM ae-API-compute: Creating zip package...
10:45:11 AM: Error: socket hang up

Some other random things I've tried:

  • different development environments with different OS's, cloning the repo fresh
  • rolling back azure function extension in visual studio code to various versions
  • protobuf==3.20.*
  • setting AzureWebJobsFeatureFlags:EnableWorkerIndexing
  • setting SCM_DO_BUILD_DURING_DEPLOYMENT:1
  • removing type hints
  • restarting

What now?
The errors are not helpful. Logs are missing, I can drill down into events, insights, many different logs, they are all over the place and they are all useless or empty. I've mentioned that I've had multiple 2hr+ calls with the technical support team of various ever-increasing escalations, and they are just as stumped as me (and I am grateful for their efforts).

Any thoughts?

What do I try next? Any information I can provide?

See also (maybe relevant, I don't know at this point):
https://github.com/microsoft/vscode-azurefunctions/issues/3805
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774
https://stackoverflow.com/questions/76478668/adding-python-module-google-cloud-storage-is-causing-a-working-azure-function-ap
https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly
https://github.com/projectkudu/kudu/issues/3348
https://github.com/microsoft/azure-pipelines-tasks/issues/14201
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774

Dominant language
Python
Stars
357
Forks
116
Avg merge
32m
Merged PRs (30d)
1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/azure-functions-python-worker

All issues in Azure/azure-functions-python-worker

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.