feat(testing): emulator-backed integration tier (ADR-013 Tier 2a), with fixes for the defects found

Abierto
#1,041 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@krokoko ya está trabajando en esto.

Desde el 16/11/2026.

  • #1048 de @krokoko — abierto

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
aws, docker, python, typescript

Línea de trabajo

The issue describes a tier spanning CDK, CLI and agent suites, a new emulator workflow, documentation, and numerous behavioral fixes; the linked pull request #1048 is already open. Start with ADR-013 and the issue’s acceptance criteria, then review the pull request and its emulator research report at docs/research/aws-emulator-local-tests.md. Done means the listed integration, guard, workflow and build criteria pass, with unit coverage for behavior changing fixes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

approved tooling validation-loop
Component

Other: test tooling and CI. The fixes that come with it touch CDK handlers and constructs, the agent runtime and the CLI.

Describe the feature

An integration test tier that runs ABCA's real handler, CLI and agent code against the AWS Emulator. The code runs in jest and pytest as usual; only its AWS SDK calls leave the process, to a pinned emulator on loopback. It needs Docker and no AWS account.

This is Tier 2a of ADR-013 (container integration tests, between the local unit build and remote CI), which the ADR marks "does not exist". It starts as an advisory check: a workflow runs it on every pull request and in the merge queue, and it is not a required status check. It does not replace the real-AWS integ smoke.

Use case

Between the mocked unit suites and the real-AWS integ smoke, nothing runs ABCA's code against an AWS API. The unit suites mock the SDK at module level, so they cannot see what DynamoDB does with a condition, an update expression, an index key or Limit together with a filter. The integ smoke sits behind an approval gate and evaluates one conditional write.

An investigation of the emulator (the report is docs/research/aws-emulator-local-tests.md in the pull request) built a proof of concept of the tier. It reproduced #51 and #1003 and found further defects on main that no issue tracked.

Proposed solution

One pull request, from one branch:

  • mise run test:integration starts a pinned, signature-verified emulator in a container, runs the cdk, cli and agent emulator suites, and stops it. The suites refuse to run against anything but the emulator, and the unit suites cannot pick them up.
  • Tests of those refusals in the unit suites, so that a guard that stops refusing fails the required build.
  • An emulator workflow that runs the tier on every pull request and in the merge queue, not required.
  • The rules for tests in the tier and its known differences from AWS, documented beside the tooling; ADR-013 and the AGENTS.md guides point to it.
  • Fixes for defects the investigation found. Not fixed here: those an open pull request addresses, and a few that stay open; the Outcome section of the report lists both. Each fix that changes behaviour has a unit test that fails without it; the comment, type and documentation corrections have none.
    • #51: the task list is not newest first, and a filtered page can come back empty while matching tasks exist. The fix adds an index, UserCreatedAtIndex, to the task table. On an existing stack the handler reads the old index, in its old order, until DynamoDB reports the new one ACTIVE; the deployment guide gets an entry. Known limit: a filtered list reads at most ten queries per request (about 210 rows at the default page size) and bgagent list does not follow next_token, so a match further back still shows as an empty list with "(More results available)".
    • GET /v1/webhooks applies its filter after Limit, as #51 did.
    • The stranded-approval sweep measures from the start of RUNNING, because the agent's approval transitions do not maintain status_created_at.
    • A session role that cannot be assumed does not stop the task.
    • The trajectory writer never retries the creation of its log stream.
    • Four writers of a terminal status leave out an attribute a terminal row needs: the Slack cancel button, the presign-failure path of create-task, the stranded-task sweep (no ttl) and confirm-uploads when screening blocks an attachment (no completed_at).
    • The link handlers for Slack, Linear and Jira check a code's ttl themselves.
    • The confirm-uploads Lambda never receives the per-user concurrency cap.
    • CLI: a GitHub token secret that was never configured is reported as configured; ops stuck-tasks ages tasks by creation time; ops concurrency shows a default limit of 3 where the stack deploys 10.
    • The integ job's command has been unrunnable since #567: it passes cdk deploy arguments to integ-runner.
    • Comments and types that blame the SDK for numbers the agent stores as strings, and an order dependence in the agent unit suite.
    • Documentation that gives a default limit of 3, or describes behaviour from before #441 (the event admission_rejected, a task failed at the concurrency limit), and comments that give the Linear webhook dedup window as 60 seconds.
Other information

Acceptance criteria:

  • mise run test:integration passes with no AWS account or credential, and mise run build does not run it.
  • The unit suites test the guard's refusals.
  • The emulator workflow runs on pull requests and in the merge queue and is not a required check.
  • Each fix that changes behaviour has a unit test that fails without the fix.
  • mise run build passes.

Behaviour that changes besides the fixes themselves. None of it changes the shape of an API response:

  • ListTasksFn gets 15 s and 256 MB (it had Lambda's 3 s and 128 MB). It asks DescribeTable for the status of the new index on each request that starts a list until the answer is ACTIVE, then not again in that execution environment; a failed check is a 500.
  • age_seconds in bgagent ops stuck-tasks --output json becomes the time in the current status, not the time since creation.
  • bgagent github set-webhook-secret no longer reads the secret first and always prints its notice.
  • An STS failure at task start that outlasts the SDK's retries fails the task; before, the task ran on and lost one best-effort write.
  • bgagent platform doctor fails the token check for a stored value that is not in a GitHub token format.
  • A link code past its ttl is refused from the deploy on.

Not verified: the emulator workflow has not run on GitHub, and nothing was compared with real AWS, including the deploy that adds the index.

Not in scope, left to open pull requests: #904 (heartbeat during approval waits), #1009 (seat release and the statuses the reconciler counts; #1003), #1013 (its new approval writer needs status_created_at too).

Related: the pull request fixes #51. Issue #61 asks for a trace round-trip test on moto; the tier has that test on the emulator, and its README covers what #61 asks of a README, at another path.

Acknowledgements
  • I may be able to implement this feature
  • This might be a breaking change
Lenguaje dominante
TypeScript
Estrellas
173
Forks
53
Merge medio
2 d 9 h
PR fusionados (30 d)
14

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de aws-samples/sample-autonomous-cloud-coding-agents

Todos los issues de aws-samples/sample-autonomous-cloud-coding-agents

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.