Add CloudWatch alarm for ApprovalMetricsPublisher EMF rate ceiling
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 64/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- aws, typescript
- Domain
- infrastructure, observability
Research direction
Start by reading cdk/src/handlers/approval-metrics-publisher.ts and verify the exact skipped-count metric name, then inspect cdk/src/constructs/approval-metrics-publisher-consumer.ts for existing CloudWatch alarms and construct tests. Add the alarm with a documented threshold and verify its presence in the construct test; link a runbook only if one exists.
Written by the indexing model from the issue text.
Description
Follow-up from PR #88 — observability blind spot in the metrics publisher path.
Functional description
The ApprovalMetricsPublisher Lambda emits CloudWatch metrics via the EMF (Embedded Metric Format) pattern: instead of calling PutMetricData directly, it logs structured JSON that CloudWatch Logs auto-extracts into metrics. EMF has its own per-account throttle ceiling — 100 EMF metric writes per second per account. Cross that ceiling and CloudWatch silently drops metrics with no error visible to the Lambda.
The publisher already self-rate-limits and emits an internal MetricEmitSkipped count when it self-limits, but there's no alarm on MetricEmitSkipped > 0. So if approvals scale to where the publisher hits the ceiling, the dashboard quietly underreports without anyone noticing.
This pairs with issue #4 (DLQ alarms): both are about "the metrics path is broken but the dashboard still shows old data." Filing as separate issues since the threshold-tuning conversation will be different (DLQ count = 1 is the right alarm; EMF skipped count needs more thought — burst spikes can be normal).
User-visible impact:
- Approval-volume metrics under-report at high load. Operators see lower
ApprovalRequestCountthan reality. - No signal to "scale up the Lambda's batch size" or "adjust EMF emission cadence."
- Capacity-planning conversations rely on CloudWatch numbers that are silently lower than actual.
Technical context
Where the metric is already emitted:
cdk/src/handlers/approval-metrics-publisher.ts— search forMetricEmitSkipped(or similar; the exact name needs verification). The publisher's self-rate-limit logic increments this when it skips an EMF write to stay under the ceiling.
What's missing:
cloudwatch.AlarmonMetricEmitSkipped > <threshold>over a sensible window.- Threshold tuning: a low threshold (1 over 5 min) will fire on legitimate burst spikes; a high threshold (>100 over 1 hour) misses sustained problems. Recommend starting at "10 over a 15-minute window" as a conservative baseline; tune after observing real traffic.
Why EMF instead of PutMetricData:
- EMF batches 100x cheaper than PutMetricData for the same observability.
- The trade-off is the global per-account ceiling. PutMetricData has its own throttle but it's per-region per-account at a much higher number.
- The publisher's choice of EMF is the right one; this issue isn't asking to change that.
Proposed fix
Add the alarm in cdk/src/constructs/approval-metrics-publisher-consumer.ts:
new cloudwatch.Alarm(this, 'MetricEmitSkippedAlarm', {
metric: new cloudwatch.Metric({
namespace: 'ABCA/Cedar-HITL',
metricName: 'MetricEmitSkipped',
period: Duration.minutes(15),
statistic: 'Sum',
}),
threshold: 10,
evaluationPeriods: 1,
comparisonOperator: ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
alarmDescription: 'ApprovalMetricsPublisher hit the EMF rate ceiling and dropped metrics; consider increasing batch size or reducing per-event metric volume',
treatMissingData: TreatMissingData.NOT_BREACHING,
});
Acceptance criteria
- Alarm exists on
MetricEmitSkipped(or whatever the existing skipped-count metric is named — verify the name inapproval-metrics-publisher.tsfirst) - Threshold is documented in the alarm description with rationale
- Construct test verifies alarm presence
- If a runbook exists for "publisher hit EMF ceiling,"
alarmDescriptionlinks to it
Out of scope
- Refactoring the publisher to use PutMetricData (different cost/perf profile, separate decision).
- Adaptive batch-size tuning (publisher could observe its own skip rate and adjust; that's a separate enhancement).
- Multi-region failover for metric publishing (out of scope for ABCA today).
References
cdk/src/handlers/approval-metrics-publisher.tscdk/src/constructs/approval-metrics-publisher-consumer.ts- AWS EMF reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html
- Per-account EMF ceiling: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html
- Companion issue: #4 (DLQ alarms, same observability gap shape)
- Dominant language
- TypeScript
- Stars
- 146
- Forks
- 46
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 27
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws-samples/sample-autonomous-cloud-coding-agents
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug v1
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
bug v1
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
documentation P2 security
Difficulty 2/5 1-2 days Newbie friendliness 74/100
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
aws-samples/sample-autonomous-cloud-coding-agents#767 · 2 comments ·
All issues in aws-samples/sample-autonomous-cloud-coding-agents
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100