Add CloudWatch alarm for ApprovalMetricsPublisher EMF rate ceiling

Open Beginner friendly
#114 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
64/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
aws, typescript

Research direction

Start by reading cdk/src/handlers/approval-metrics-publisher.ts and verify the exact skipped-count metric name, then inspect cdk/src/constructs/approval-metrics-publisher-consumer.ts for existing CloudWatch alarms and construct tests. Add the alarm with a documented threshold and verify its presence in the construct test; link a runbook only if one exists.

Written by the indexing model from the issue text.

Description

observability

Follow-up from PR #88 — observability blind spot in the metrics publisher path.

Functional description

The ApprovalMetricsPublisher Lambda emits CloudWatch metrics via the EMF (Embedded Metric Format) pattern: instead of calling PutMetricData directly, it logs structured JSON that CloudWatch Logs auto-extracts into metrics. EMF has its own per-account throttle ceiling — 100 EMF metric writes per second per account. Cross that ceiling and CloudWatch silently drops metrics with no error visible to the Lambda.

The publisher already self-rate-limits and emits an internal MetricEmitSkipped count when it self-limits, but there's no alarm on MetricEmitSkipped > 0. So if approvals scale to where the publisher hits the ceiling, the dashboard quietly underreports without anyone noticing.

This pairs with issue #4 (DLQ alarms): both are about "the metrics path is broken but the dashboard still shows old data." Filing as separate issues since the threshold-tuning conversation will be different (DLQ count = 1 is the right alarm; EMF skipped count needs more thought — burst spikes can be normal).

User-visible impact:

  • Approval-volume metrics under-report at high load. Operators see lower ApprovalRequestCount than reality.
  • No signal to "scale up the Lambda's batch size" or "adjust EMF emission cadence."
  • Capacity-planning conversations rely on CloudWatch numbers that are silently lower than actual.

Technical context

Where the metric is already emitted:

  • cdk/src/handlers/approval-metrics-publisher.ts — search for MetricEmitSkipped (or similar; the exact name needs verification). The publisher's self-rate-limit logic increments this when it skips an EMF write to stay under the ceiling.

What's missing:

  • cloudwatch.Alarm on MetricEmitSkipped > <threshold> over a sensible window.
  • Threshold tuning: a low threshold (1 over 5 min) will fire on legitimate burst spikes; a high threshold (>100 over 1 hour) misses sustained problems. Recommend starting at "10 over a 15-minute window" as a conservative baseline; tune after observing real traffic.

Why EMF instead of PutMetricData:

  • EMF batches 100x cheaper than PutMetricData for the same observability.
  • The trade-off is the global per-account ceiling. PutMetricData has its own throttle but it's per-region per-account at a much higher number.
  • The publisher's choice of EMF is the right one; this issue isn't asking to change that.

Proposed fix

Add the alarm in cdk/src/constructs/approval-metrics-publisher-consumer.ts:

new cloudwatch.Alarm(this, 'MetricEmitSkippedAlarm', {
  metric: new cloudwatch.Metric({
    namespace: 'ABCA/Cedar-HITL',
    metricName: 'MetricEmitSkipped',
    period: Duration.minutes(15),
    statistic: 'Sum',
  }),
  threshold: 10,
  evaluationPeriods: 1,
  comparisonOperator: ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
  alarmDescription: 'ApprovalMetricsPublisher hit the EMF rate ceiling and dropped metrics; consider increasing batch size or reducing per-event metric volume',
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

Acceptance criteria

  • Alarm exists on MetricEmitSkipped (or whatever the existing skipped-count metric is named — verify the name in approval-metrics-publisher.ts first)
  • Threshold is documented in the alarm description with rationale
  • Construct test verifies alarm presence
  • If a runbook exists for "publisher hit EMF ceiling," alarmDescription links to it

Out of scope

  • Refactoring the publisher to use PutMetricData (different cost/perf profile, separate decision).
  • Adaptive batch-size tuning (publisher could observe its own skip rate and adjust; that's a separate enhancement).
  • Multi-region failover for metric publishing (out of scope for ABCA today).

References

Dominant language
TypeScript
Stars
146
Forks
46
Avg merge
3d 2h
Merged PRs (30d)
27

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws-samples/sample-autonomous-cloud-coding-agents

All issues in aws-samples/sample-autonomous-cloud-coding-agents

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.