feat(testing): emulator-backed integration tier (ADR-013 Tier 2a), with fixes for the defects found

オープン
#1,041 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@krokoko がすでに取り組んでいます。

2026年11月16日 から。

  • #1048 @krokoko による — オープン

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
15/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
停滞
技術スタック
aws, docker, python, typescript

調査の方向性

The issue describes a tier spanning CDK, CLI and agent suites, a new emulator workflow, documentation, and numerous behavioral fixes; the linked pull request #1048 is already open. Start with ADR-013 and the issue’s acceptance criteria, then review the pull request and its emulator research report at docs/research/aws-emulator-local-tests.md. Done means the listed integration, guard, workflow and build criteria pass, with unit coverage for behavior changing fixes.

索引モデルが issue の本文から書いたものです。

説明

approved tooling validation-loop
Component

Other: test tooling and CI. The fixes that come with it touch CDK handlers and constructs, the agent runtime and the CLI.

Describe the feature

An integration test tier that runs ABCA's real handler, CLI and agent code against the AWS Emulator. The code runs in jest and pytest as usual; only its AWS SDK calls leave the process, to a pinned emulator on loopback. It needs Docker and no AWS account.

This is Tier 2a of ADR-013 (container integration tests, between the local unit build and remote CI), which the ADR marks "does not exist". It starts as an advisory check: a workflow runs it on every pull request and in the merge queue, and it is not a required status check. It does not replace the real-AWS integ smoke.

Use case

Between the mocked unit suites and the real-AWS integ smoke, nothing runs ABCA's code against an AWS API. The unit suites mock the SDK at module level, so they cannot see what DynamoDB does with a condition, an update expression, an index key or Limit together with a filter. The integ smoke sits behind an approval gate and evaluates one conditional write.

An investigation of the emulator (the report is docs/research/aws-emulator-local-tests.md in the pull request) built a proof of concept of the tier. It reproduced #51 and #1003 and found further defects on main that no issue tracked.

Proposed solution

One pull request, from one branch:

  • mise run test:integration starts a pinned, signature-verified emulator in a container, runs the cdk, cli and agent emulator suites, and stops it. The suites refuse to run against anything but the emulator, and the unit suites cannot pick them up.
  • Tests of those refusals in the unit suites, so that a guard that stops refusing fails the required build.
  • An emulator workflow that runs the tier on every pull request and in the merge queue, not required.
  • The rules for tests in the tier and its known differences from AWS, documented beside the tooling; ADR-013 and the AGENTS.md guides point to it.
  • Fixes for defects the investigation found. Not fixed here: those an open pull request addresses, and a few that stay open; the Outcome section of the report lists both. Each fix that changes behaviour has a unit test that fails without it; the comment, type and documentation corrections have none.
    • #51: the task list is not newest first, and a filtered page can come back empty while matching tasks exist. The fix adds an index, UserCreatedAtIndex, to the task table. On an existing stack the handler reads the old index, in its old order, until DynamoDB reports the new one ACTIVE; the deployment guide gets an entry. Known limit: a filtered list reads at most ten queries per request (about 210 rows at the default page size) and bgagent list does not follow next_token, so a match further back still shows as an empty list with "(More results available)".
    • GET /v1/webhooks applies its filter after Limit, as #51 did.
    • The stranded-approval sweep measures from the start of RUNNING, because the agent's approval transitions do not maintain status_created_at.
    • A session role that cannot be assumed does not stop the task.
    • The trajectory writer never retries the creation of its log stream.
    • Four writers of a terminal status leave out an attribute a terminal row needs: the Slack cancel button, the presign-failure path of create-task, the stranded-task sweep (no ttl) and confirm-uploads when screening blocks an attachment (no completed_at).
    • The link handlers for Slack, Linear and Jira check a code's ttl themselves.
    • The confirm-uploads Lambda never receives the per-user concurrency cap.
    • CLI: a GitHub token secret that was never configured is reported as configured; ops stuck-tasks ages tasks by creation time; ops concurrency shows a default limit of 3 where the stack deploys 10.
    • The integ job's command has been unrunnable since #567: it passes cdk deploy arguments to integ-runner.
    • Comments and types that blame the SDK for numbers the agent stores as strings, and an order dependence in the agent unit suite.
    • Documentation that gives a default limit of 3, or describes behaviour from before #441 (the event admission_rejected, a task failed at the concurrency limit), and comments that give the Linear webhook dedup window as 60 seconds.
Other information

Acceptance criteria:

  • mise run test:integration passes with no AWS account or credential, and mise run build does not run it.
  • The unit suites test the guard's refusals.
  • The emulator workflow runs on pull requests and in the merge queue and is not a required check.
  • Each fix that changes behaviour has a unit test that fails without the fix.
  • mise run build passes.

Behaviour that changes besides the fixes themselves. None of it changes the shape of an API response:

  • ListTasksFn gets 15 s and 256 MB (it had Lambda's 3 s and 128 MB). It asks DescribeTable for the status of the new index on each request that starts a list until the answer is ACTIVE, then not again in that execution environment; a failed check is a 500.
  • age_seconds in bgagent ops stuck-tasks --output json becomes the time in the current status, not the time since creation.
  • bgagent github set-webhook-secret no longer reads the secret first and always prints its notice.
  • An STS failure at task start that outlasts the SDK's retries fails the task; before, the task ran on and lost one best-effort write.
  • bgagent platform doctor fails the token check for a stored value that is not in a GitHub token format.
  • A link code past its ttl is refused from the deploy on.

Not verified: the emulator workflow has not run on GitHub, and nothing was compared with real AWS, including the deploy that adds the index.

Not in scope, left to open pull requests: #904 (heartbeat during approval waits), #1009 (seat release and the statuses the reconciler counts; #1003), #1013 (its new approval writer needs status_created_at too).

Related: the pull request fixes #51. Issue #61 asks for a trace round-trip test on moto; the tier has that test on the emulator, and its README covers what #61 asks of a README, at another path.

Acknowledgements
  • I may be able to implement this feature
  • This might be a breaking change
主要言語
TypeScript
スター
173
フォーク
53
平均マージ
2日 9時間
マージ済み PR(30日)
14

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

aws-samples/sample-autonomous-cloud-coding-agents のほかの issue

aws-samples/sample-autonomous-cloud-coding-agents の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。