feat(testing): emulator-backed integration tier (ADR-013 Tier 2a), with fixes for the defects found
メンテナーはふだん 1 日以内に返信
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 15/100
調査の方向性
The issue describes a tier spanning CDK, CLI and agent suites, a new emulator workflow, documentation, and numerous behavioral fixes; the linked pull request #1048 is already open. Start with ADR-013 and the issue’s acceptance criteria, then review the pull request and its emulator research report at docs/research/aws-emulator-local-tests.md. Done means the listed integration, guard, workflow and build criteria pass, with unit coverage for behavior changing fixes.
索引モデルが issue の本文から書いたものです。
説明
Component
Other: test tooling and CI. The fixes that come with it touch CDK handlers and constructs, the agent runtime and the CLI.
Describe the feature
An integration test tier that runs ABCA's real handler, CLI and agent code against the AWS Emulator. The code runs in jest and pytest as usual; only its AWS SDK calls leave the process, to a pinned emulator on loopback. It needs Docker and no AWS account.
This is Tier 2a of ADR-013 (container integration tests, between the local unit build and remote CI), which the ADR marks "does not exist". It starts as an advisory check: a workflow runs it on every pull request and in the merge queue, and it is not a required status check. It does not replace the real-AWS integ smoke.
Use case
Between the mocked unit suites and the real-AWS integ smoke, nothing runs ABCA's code against an AWS API. The unit suites mock the SDK at module level, so they cannot see what DynamoDB does with a condition, an update expression, an index key or Limit together with a filter. The integ smoke sits behind an approval gate and evaluates one conditional write.
An investigation of the emulator (the report is docs/research/aws-emulator-local-tests.md in the pull request) built a proof of concept of the tier. It reproduced #51 and #1003 and found further defects on main that no issue tracked.
Proposed solution
One pull request, from one branch:
mise run test:integrationstarts a pinned, signature-verified emulator in a container, runs thecdk,cliand agent emulator suites, and stops it. The suites refuse to run against anything but the emulator, and the unit suites cannot pick them up.- Tests of those refusals in the unit suites, so that a guard that stops refusing fails the required build.
- An
emulatorworkflow that runs the tier on every pull request and in the merge queue, not required. - The rules for tests in the tier and its known differences from AWS, documented beside the tooling; ADR-013 and the
AGENTS.mdguides point to it. - Fixes for defects the investigation found. Not fixed here: those an open pull request addresses, and a few that stay open; the Outcome section of the report lists both. Each fix that changes behaviour has a unit test that fails without it; the comment, type and documentation corrections have none.
- #51: the task list is not newest first, and a filtered page can come back empty while matching tasks exist. The fix adds an index,
UserCreatedAtIndex, to the task table. On an existing stack the handler reads the old index, in its old order, until DynamoDB reports the new oneACTIVE; the deployment guide gets an entry. Known limit: a filtered list reads at most ten queries per request (about 210 rows at the default page size) andbgagent listdoes not follownext_token, so a match further back still shows as an empty list with "(More results available)". GET /v1/webhooksapplies its filter afterLimit, as #51 did.- The stranded-approval sweep measures from the start of
RUNNING, because the agent's approval transitions do not maintainstatus_created_at. - A session role that cannot be assumed does not stop the task.
- The trajectory writer never retries the creation of its log stream.
- Four writers of a terminal status leave out an attribute a terminal row needs: the Slack cancel button, the presign-failure path of
create-task, the stranded-task sweep (nottl) and confirm-uploads when screening blocks an attachment (nocompleted_at). - The link handlers for Slack, Linear and Jira check a code's
ttlthemselves. - The confirm-uploads Lambda never receives the per-user concurrency cap.
- CLI: a GitHub token secret that was never configured is reported as configured;
ops stuck-tasksages tasks by creation time;ops concurrencyshows a default limit of 3 where the stack deploys 10. - The
integjob's command has been unrunnable since #567: it passescdk deployarguments tointeg-runner. - Comments and types that blame the SDK for numbers the agent stores as strings, and an order dependence in the agent unit suite.
- Documentation that gives a default limit of 3, or describes behaviour from before #441 (the event
admission_rejected, a task failed at the concurrency limit), and comments that give the Linear webhook dedup window as 60 seconds.
- #51: the task list is not newest first, and a filtered page can come back empty while matching tasks exist. The fix adds an index,
Other information
Acceptance criteria:
-
mise run test:integrationpasses with no AWS account or credential, andmise run builddoes not run it. - The unit suites test the guard's refusals.
- The
emulatorworkflow runs on pull requests and in the merge queue and is not a required check. - Each fix that changes behaviour has a unit test that fails without the fix.
-
mise run buildpasses.
Behaviour that changes besides the fixes themselves. None of it changes the shape of an API response:
ListTasksFngets 15 s and 256 MB (it had Lambda's 3 s and 128 MB). It asksDescribeTablefor the status of the new index on each request that starts a list until the answer isACTIVE, then not again in that execution environment; a failed check is a500.age_secondsinbgagent ops stuck-tasks --output jsonbecomes the time in the current status, not the time since creation.bgagent github set-webhook-secretno longer reads the secret first and always prints its notice.- An STS failure at task start that outlasts the SDK's retries fails the task; before, the task ran on and lost one best-effort write.
bgagent platform doctorfails the token check for a stored value that is not in a GitHub token format.- A link code past its
ttlis refused from the deploy on.
Not verified: the emulator workflow has not run on GitHub, and nothing was compared with real AWS, including the deploy that adds the index.
Not in scope, left to open pull requests: #904 (heartbeat during approval waits), #1009 (seat release and the statuses the reconciler counts; #1003), #1013 (its new approval writer needs status_created_at too).
Related: the pull request fixes #51. Issue #61 asks for a trace round-trip test on moto; the tier has that test on the emulator, and its README covers what #61 asks of a README, at another path.
Acknowledgements
- I may be able to implement this feature
- This might be a breaking change
- 主要言語
- TypeScript
- スター
- 173
- フォーク
- 53
- 平均マージ
- 2日 9時間
- マージ済み PR(30日)
- 14
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
aws-samples/sample-autonomous-cloud-coding-agents のほかの issue
-
bug infra-cdk
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
aws-samples/sample-autonomous-cloud-coding-agents#1044 ·
メンテナーはふだん 1 日以内に返信
-
cdk: support Docker-compatible engines such as Finch (clean:disk hard-codes docker)対応中かも @alvarohare が 20 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
aws-samples/sample-autonomous-cloud-coding-agents#969 · コメント 1 件 · リアクション 1 件 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
aws-samples/sample-autonomous-cloud-coding-agents#968 ·
メンテナーはふだん 1 日以内に返信
-
documentation P2 security
難易度 2/5 1〜2日 初心者へのやさしさ 74/100
aws-samples/sample-autonomous-cloud-coding-agents#793 ·
メンテナーはふだん 1 日以内に返信
-
Docs: How to clear previously set attributes for a blueprint再び着手できるかも @rkumarus-aws が 77 日前に担当しましたが、オープン中のプルリクエストはありません。 オープンdocumentation
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
aws-samples/sample-autonomous-cloud-coding-agents#767 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
aws-samples/sample-autonomous-cloud-coding-agents の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
vercel-labs/ui#809 ·
メンテナーはふだん 1 日以内に返信
-
bot:ai-assisted component:platform-js status:untriaged
難易度 1/5 1時間未満 初心者へのやさしさ 74/100
midnightntwrk/platform-js#38 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
ranxianglei/billion-context#7068 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
mybigday/whisper.rn#348 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
six-group/six-webcomponents#600 ·