Nondeterministc deploy bug with 5+ different non-descript failures on two different OS's, worked before, support stumped
Nobody has claimed this yet.
Assessment
This issue has not been assessed yet.
Description
Unsure how to title this. It's been a week of debugging and isolating with no results.
Is this issue in the correct repo? Also unsure. Help me out here.
I'll walk you through my journey.
I have a very large code base, it runs flawlessly locally. I've deployed it just fine for a long time until recently.
I made a change, including google-cloud-texttospeech in requirements.txt.
It stopped deploying after this (maybe relevant, maybe not).
Removing this change, to the exact same code base as before, still fails to deploy.
Some errors I get at random, and I've tried incredibly hard to isolate.
Both development environments can deploy other things successfully, and push to a new app fine as well.
I have not made any local environment changes at all when it began to fail.
Details:
Azure Functions runtime version 4.24.4.4, Linux premium plan with Elastic Premium EP3.
I deploy directly from visual studio code azure function extension.
My python code follows the folder structure of the v1 coding model (no decorators, lots of folders), however I am on v2 (host.json etc.), this has always worked, and runs locally of course.
Deploying to python 3.9.7.
First development environment:
Manjaro linux, vscode 1.81.1, azure extension 1.12.3
Errors I've encountered seemingly at random, when trying to deploy:
2:50:41 PM debugApp123: **Deployment successful**. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
2:51:07 PM debugApp123: Syncing triggers...
2:51:12 PM debugApp123: Querying triggers...
2:51:18 PM debugApp123: **No HTTP triggers found.**
3:34:11 PM debugApp123: Deployment Failed. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
3:34:25 PM debugApp123: Deployment failed.
4:00:49 PM: Error: The operation was aborted.
10:52:10 AM ae-API-compute: Deployment successful. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
10:52:42 AM ae-API-compute: Syncing triggers...
10:52:49 AM ae-API-compute: Querying triggers...
10:52:52 AM ae-API-compute: WARNING: Some http trigger urls cannot be displayed in the output window because they require an authentication token. Instead, you may copy them from the Azure Functions explorer.
Except it was not successful and functions are empty / don't run (the zip file in webjobstorage contains them thought).
11:55:04 AM debugApp123: Syncing triggers...
11:55:45 AM debugApp123: Syncing triggers (Attempt 2/6)...
11:55:56 AM debugApp123: Syncing triggers (Attempt 3/6)...
11:56:17 AM debugApp123: Syncing triggers (Attempt 4/6)...
11:56:59 AM debugApp123: Syncing triggers (Attempt 5/6)...
11:58:20 AM debugApp123: Syncing triggers (Attempt 6/6)...
11:59:12 AM: Error: Encountered an error (ServiceUnavailable) from host runtime.
And a very exciting shiny rare one:
Offset to Central Directory cannot be held in an Int64.
Second development environment:
Windows, visual studio code, azure extension v1.12.4.
I've also used this environment before, and it has worked, same as the above one.
Errors:
One successful deploy, and randomly most of the above (with no changes), as well as a new one:
10:35:34 AM ae-API-compute: Starting deployment...
10:35:34 AM ae-API-compute: Creating zip package...
10:45:11 AM: Error: socket hang up
Some other random things I've tried:
- different development environments with different OS's, cloning the repo fresh
- rolling back azure function extension in visual studio code to various versions
- protobuf==3.20.*
- setting AzureWebJobsFeatureFlags:EnableWorkerIndexing
- setting SCM_DO_BUILD_DURING_DEPLOYMENT:1
- removing type hints
- restarting
What now?
The errors are not helpful. Logs are missing, I can drill down into events, insights, many different logs, they are all over the place and they are all useless or empty. I've mentioned that I've had multiple 2hr+ calls with the technical support team of various ever-increasing escalations, and they are just as stumped as me (and I am grateful for their efforts).
Any thoughts?
What do I try next? Any information I can provide?
See also (maybe relevant, I don't know at this point):
https://github.com/microsoft/vscode-azurefunctions/issues/3805
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774
https://stackoverflow.com/questions/76478668/adding-python-module-google-cloud-storage-is-causing-a-working-azure-function-ap
https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly
https://github.com/projectkudu/kudu/issues/3348
https://github.com/microsoft/azure-pipelines-tasks/issues/14201
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774
- Dominant language
- Python
- Stars
- 357
- Forks
- 116
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/azure-functions-python-worker
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
bug python
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Azure/azure-functions-python-worker#1881 · 2 comments · 2 reactions ·
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
Azure/azure-functions-python-worker#1908 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Azure/azure-functions-python-worker#1906 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
All issues in Azure/azure-functions-python-worker
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100