Plugin conformance tests are flaky when CloudWatch logs are not yet visible

Open
#708 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
aws, python

Research direction

Start with .github/workflows/conformance-tests.yml and the “Run conformance suite” step, then rerun the plugin suite repeatedly against the same commit. Investigate CloudWatch propagation or log-stream lookup timing and capture the suggested log-group, execution-filter, event-count, and retrieved-record diagnostics; done means the conformance tests reliably find expected records.

Written by the indexing model from the issue text.

Description

bug pkg:testing

Summary

The conformance (plugin) job can fail because expected plugin log records are not returned by CloudWatch, even though the durable execution and history validation complete successfully.

Evidence

  • Previous successful run: 33907972521 at commit 4ce54bcf.
  • Failed run: 33922277509 at commit 676c75a.
  • Attempt 1, job 101185043031: test 10-11 (plugin_nested_parent_linkage) failed because both expected operation-end records matched 0 logs: the inner step record and the child-context record.
  • Retry attempt 2, job 101188790974: test 10-11 passed, but unrelated test 10-7 (plugin_terminal_failure) failed because its invocation-start and invocation-end records matched 0 logs.

The failure moving between tests on the same commit strongly suggests CloudWatch ingestion/filter timing or log-stream lookup flakiness rather than a deterministic SDK regression. The diff from the previous successful commit only changes OTel release metadata and does not affect the regular core plugin suite.

Suggested investigation

  • Increase or make configurable the CloudWatch log propagation/polling window in the conformance runner.
  • Log the resolved log group, execution ARN filter, event count, and final retrieved records when ExpectedLogs fails.
  • Pin the conformance runner to a known commit in CI instead of installing @main.
  • Re-run the plugin suite repeatedly against the same commit to confirm the failure rate.

Relevant workflow: .github/workflows/conformance-tests.yml, step Run conformance suite.

Dominant language
Python
Stars
59
Forks
32
Avg merge
1d 5h
Merged PRs (30d)
76

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/aws-durable-execution-sdk-python

All issues in aws/aws-durable-execution-sdk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.