serverless/serverless

Silent failure of Serverless deploy

Open

#8839 opened on Jan 26, 2021

View on GitHub
 (11 comments) (6 reactions) (0 assignees)JavaScript (46,915 stars) (5,734 forks)batch import
bugcat/deploymenthelp wanted

Description

With large files and slow/unstable network, Serverless fails to deploy stack without causing error status code or error message. I have only seen it when deploying to AWS CN.

It is partially dependent on environment variable AWS_CLIENT_TIMEOUT, large values makes it less frequent. For the example below I have set AWS_CLIENT_TIMEOUT=10 to make it easier to reproduce, but it still happens with default (or larger than default) values. It does not happen if uploaded file is too small, so to test you can add any large file to the project folder (to test I used a movie file I had, about 48 MB). Smaller files (5-10 MB) also trigger this problem.

The critical part is not that the upload times out or fails, it is that there is no error code or error message. When put into a deployment pipeline it then looks like deployment worked without any problems.

service: myproject
frameworkVersion: "2"

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
  region: cn-north-1

functions:
  hello:
    handler: handler.hello

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 1.258s 0 retries] describeStacks({ StackName: 'myproject-dev' })
Serverless: [AWS cloudformation 200 1.639s 0 retries] describeStackResource({
  StackName: 'myproject-dev',
  LogicalResourceId: 'ServerlessDeploymentBucket'
})
Serverless: [AWS s3 200 1.518s 0 retries] listObjectsV2({
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Prefix: 'serverless/myproject/dev'
})
Serverless: [AWS s3 200 1.223s 0 retries] headObject({
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Key: 'serverless/myproject/dev/1611684988009-2021-01-26T18:16:28.009Z/compiled-cloudformation-template.json'
})
Serverless: [AWS lambda 404 1.371s 0 retries] getFunction({ FunctionName: 'myproject-dev-hello' })
Serverless: [AWS sts 200 1.387s 0 retries] getCallerIdentity({})
Serverless: Uploading CloudFormation file to S3...
Serverless: [AWS s3 200 1.279s 0 retries] putObject({
  Body: <Buffer 7b 22 41 57 53 54 65 6d 70 6c 61 74 65 46 6f 72 6d 61 74 56 65 72 73 69 6f 6e 22 3a 22 32 30 31 30 2d 30 39 2d 30 39 22 2c 22 44 65 73 63 72 69 70 74 ... 2611 more bytes>,
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Key: 'serverless/myproject/dev/1611685045928-2021-01-26T18:17:25.928Z/compiled-cloudformation-template.json',
  ContentType: 'application/json',
  Metadata: { filesha256: '4QjwYux9cZLlJCWZp1iRQK/Gfu4YbEtHOv8nSifHZ0k=' }
})
Serverless: Uploading artifacts...
Serverless: Uploading service myproject.zip file to S3 (45.72 MB)...
Serverless: [AWS s3 200 1.293s 0 retries] createMultipartUpload({
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Key: 'serverless/myproject/dev/1611685045928-2021-01-26T18:17:25.928Z/myproject.zip',
  ContentType: 'application/zip',
  Metadata: { filesha256: 'RO0zY/mKJWYrYklD1X8wUIIZb8gVQBngmKxkZvgXLCQ=' }
})
Serverless: [AWS s3 undefined 13.585s 3 retries] uploadPart({
  Body: <Buffer 16 cc 31 bc 27 30 70 99 b6 ee 08 09 a6 bd 18 cb aa b3 06 21 6d c8 fc 1b 2f 57 47 af e8 a4 81 7c c2 5f c9 e6 16 d2 fe 18 be 50 37 a6 b8 bb 31 35 31 59 ... 5242830 more bytes>,
  ContentLength: 5242880,
  PartNumber: 3,
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Key: 'serverless/myproject/dev/1611685045928-2021-01-26T18:17:25.928Z/myproject.zip',
  UploadId: 'KfGGAI5wuSkg4PiNsCJOyTwB2JmOJZ3JWV87cefOpamFCvScSQfcq_whIkmhluH7zByUQofVli8MGcRNRqhEkdr6vKc7s7jkXpSSeteE7S7PrA7STI6E60W3msZnnzScGzE4ePpQIV7AGemE334Fet1OUjWDJVS9MnxhAAIi3To-'
})
Serverless: Recoverable error occurred (Connection timed out after 1000ms), sleeping for ~7 seconds. Try 1 of 4
Serverless: [AWS s3 204 1.562s 0 retries] abortMultipartUpload({
  Bucket: 'myproject-dev-serverlessdeploymentbucket-ayafrv4s2ty9',
  Key: 'serverless/myproject/dev/1611685045928-2021-01-26T18:17:25.928Z/myproject.zip',
  UploadId: 'KfGGAI5wuSkg4PiNsCJOyTwB2JmOJZ3JWV87cefOpamFCvScSQfcq_whIkmhluH7zByUQofVli8MGcRNRqhEkdr6vKc7s7jkXpSSeteE7S7PrA7STI6E60W3msZnnzScGzE4ePpQIV7AGemE334Fet1OUjWDJVS9MnxhAAIi3To-'
})

Installed version

Framework Core: 2.21.0 (local)
Plugin: 4.4.2
SDK: 2.3.2
Components: 3.5.1

Contributor guide

Tech stack
javascriptnodejsaws
Domain
backendclouddevops
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Node.jsAWS SDK basicsknowledge of multipart upload
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate how the Serverless Framework handles S3 multipart upload errors. Look at the deploy command's upload logic, specifically error handling around 'uploadPart' calls. Check if there's a missing error callback or unhandled promise rejection. Also review the AWS client timeout configuration and see if the 'abortMultipartUpload' is being called correctly. The issue mentions a recoverable error but no final error status.