[Lambda][CloudWatch] Throttled Executions Counted as Success, Error, or Ignored
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- aws
- Domain
- cloud, observability-sre
Research direction
No repository files, tests, or implementation entry points are identified in the issue. Start by reviewing how AWS Lambda throttling is represented in CloudWatch metrics, then define how configurable success, error, and ignored outcomes should behave and how each option would be verified.
Written by the indexing model from the issue text.
Description
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request
When using AWS Lambda with reserved concurrency, Lambda executions can be throttled. Currently, in CloudWatch, these throttled executions are recorded as successes. We request the ability to configure the default behavior of throttled executions, allowing users to choose whether they are logged as successes (default), errors, or ignored entirely.
Which service(s) is this request for?
AWS Lambda / CloudWatch
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
When using AWS Lambda with reserved concurrency, throttled executions can occur. In CloudWatch, these throttled executions are currently logged as successes, which can lead to misleading metrics and unreliable alarms.
For example:
- Input: A Lambda function with reserved concurrency set to 1, a timeout of 15 minutes, and a rule to invoke the function every 5 minutes.
- Behavior: If an issue occurs during execution (e.g., Lambda timeout), the following output is observed:
- 1 execution logged as an error (timeout)
- 2 throttled executions logged as successes
This behavior makes it difficult to build reliable CloudWatch alarms. For instance, if we want to allow occasional failures but not consecutive ones (e.g., intermittent issues), the current logging behavior of throttled executions as successes complicates alarm configuration.
We propose adding the ability to configure how throttled executions are logged in CloudWatch:
- Success (default)
- Error
- Ignored (no logs)
Are you currently working around this issue?
To work around this issue, I configure CloudWatch alarms with an evaluation period equal to the Lambda timeout plus the invocation delay (15 + 5 minutes). I then set the alarm to trigger if there are 2 errors out of 3 evaluation periods. However, this workaround is complex and not ideal for all use cases.
Additional context
Here is an example of the issue: In the "Error count and success rate" graph, throttled executions are logged as successes, even though all real executions are errors. This behavior leads to misleading metrics and unreliable alarms.
Attachments
Attached is an example graph showing the issue, where throttled executions are logged as successes despite all actual executions being errors.
Overall view:
Error count and success rate: some executions count as "not in error".
- Dominant language
- No language data
- Stars
- 196
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/aws-lambda-roadmap
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
aws/aws-lambda-roadmap#101 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
aws/aws-lambda-roadmap#100 ·
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
aws/aws-lambda-roadmap#97 · 1 comment · 1 reaction ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
aws/aws-lambda-roadmap#96 · 1 reaction ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
aws/aws-lambda-roadmap#95 · 1 comment · 1 reaction ·
All issues in aws/aws-lambda-roadmap
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
level/task module/gcp type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
bug needs-triage service/elbv2
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
hashicorp/terraform-provider-aws#50100 · 1 comment ·