[FEATURE] Waiter semantics for `callback`: stop on terminal failure, time-based bounds, backoff

Start with run_callback_poll in src/core/utils.rs, then trace run_callback and run_troubleshoot in src/commands/base.rs and the callback paths in src/commands/build.rs and src/commands/teardown.rs. Review the documented callback example in website/docs/resource-query-files.md. Done means terminal failures stop polling with diagnostics and troubleshooting, time-based bounds and backoff work, and existing callback behavior remains compatible.

Problem

Asynchronous mutations return a handle that has to be polled until the operation finishes. awscc (Cloud Control) is the main case: every create, update and delete returns a RequestToken. callback is the hook for polling it, but a callback result has only two states: done, or keep polling. There is no way to say "this operation reached a terminal failure, stop". The docs already list this under the callback's known limitations.

Here is a real case. An awscc.s3.buckets create for a bucket name deleted about 40 minutes earlier in another region. Cloud Control accepted the request, reported IN_PROGRESS (with an interim ErrorCode: ServiceInternalError) for about 3 minutes while it retried the S3 call internally, and then finished:

OperationStatus: FAILED
ErrorCode:       ResourceConflict
StatusMessage:   A conflicting conditional operation is currently in progress against this resource. Please try again. (Service: S3, Status Code: 409, ...)

None of the ways to handle this today reports that message:

1. Callback that succeeds on OperationStatus = 'SUCCESS' (the documented pattern). It keeps polling a request that has already FAILED until retries runs out, then exits with callback timeout for [x] create operation after N retries. run_callback exits through catch_error_and_exit without running troubleshoot, so the reason is never shown.
2. No callback, relying on the post-create exists check plus troubleshoot:create. The exists check gives up after the statecheck's retries x retry_delay (50 s here). troubleshoot then runs once, while the request is still IN_PROGRESS. The log showed IN_PROGRESS / ServiceInternalError / StatusMessage: NULL, which isn't the reason.
3. Workaround: a callback that treats any terminal state as done, OperationStatus IN ('SUCCESS', 'FAILED', 'CANCEL_COMPLETE') AS success. troubleshoot does then report the real message. But the post-create exists check still polls for a resource that will never exist (another 50 s), and the final error is not found after create post-deploy check, create operation may have failed, although the failure was known and explained a minute earlier.

Two more problems with the poll budget:

• It is a count, not a time. The budget is retries x a fixed retry_delay. Authors have to guess a count that covers the slowest normal case, which makes failures slow to report and still doesn't guarantee coverage.
• Provider hints are ignored. Cloud Control returns RetryAfter in the progress event, and nothing can use it.

Other providers have the same shape: Azure long-running operations (provisioningState = 'Failed'), Google operations resources (done = true with an error), and Databricks resources with a FAILED / ERROR lifecycle state.

Proposal

Give callback waiter semantics, modelled on AWS SDK waiters, where each poll result is success, failure or retry. Extending the existing hook keeps one polling construct instead of two, and it stays backward compatible.

1. Three-state result. The callback query can return an optional failed column alongside success:
   • success truthy -> done (as today)
   • failed truthy -> terminal failure
   • neither -> still pending, poll again
2. Stop at once on failure. Stop polling and log the whole row as diagnostics, so extra columns such as message or error_code appear in the output. Then run troubleshoot:<op> if one is defined, and fail the resource under the existing --on-failure handling. Skip the post-deploy exists and statecheck for that resource, since the outcome is already known.
3. Run troubleshoot on callback timeout, before exiting.
4. Time-based bounds.
   • timeout=<seconds>: a wall-clock budget, alongside or instead of retries.
   • backoff=<multiplier> and max_delay=<seconds>: grow the delay between polls.
5. Honour provider hints (optional). If the query returns a retry_after column (seconds, or an epoch timestamp, which is what Cloud Control's RetryAfter is), use it for the next delay, capped by max_delay and timeout.

For awscc the whole contract becomes:

/*+ create */
INSERT INTO awscc.s3.buckets (BucketName, region)
SELECT '{{ bucket_name }}', '{{ region }}'
RETURNING *

/*+ callback:create, timeout=600, retry_delay=5, backoff=1.5, max_delay=30 */
SELECT
  OperationStatus = 'SUCCESS' AS success,
  OperationStatus IN ('FAILED', 'CANCEL_COMPLETE') AS failed,
  ErrorCode AS error_code,
  StatusMessage AS message
FROM awscc.cloud_control.resource_request
WHERE RequestToken = '{{ callback.RequestToken }}'
AND region = '{{ region }}'

A normal create finishes on the first or second poll. A failed one stops as soon as Cloud Control reports FAILED and prints ResourceConflict plus the S3 message. The run fails on that error, not on a later "not found".

Behaviour to pin down

• success and failed both truthy. Suggest treating it as a failure (the conservative choice) and logging a warning, since the query is ambiguous.
• Query errors and empty results while polling. Treat both as pending, as today: some providers briefly return 404 for a handle just after dispatch. Log the error at debug level on each attempt, and include the last one in the timeout message.
• retries and timeout both set. Stop at whichever limit comes first. With neither set, keep the current defaults (retries=3, retry_delay=5).
• Short circuit. short_circuit_field / short_circuit_value keep working. The same three-state check could run on the RETURNING * row, so a request that fails synchronously never needs a poll.
• Teardown. callback:delete gets the same semantics. A terminal failure means the resource is reported as not confirmed deleted under --on-failure ignore and aborts under --on-failure error.
• Dry run. Unchanged: callbacks are rendered and logged, not executed.
• Out of scope. A failed column on statecheck / exists (for example Azure provisioningState = 'Failed') could reuse the same evaluation later, but it's a separate change.

Related

• website/docs/resource-query-files.md, callback section: the "no mechanism to short-circuit retries on a terminal failure" known limitation goes away.
• The same page's awscc callback example is out of date. It polls awscc.cloudcontrol.resource_request_statuses with {{ callback.ProgressEvent.RequestToken }}. The current provider resource is awscc.cloud_control.resource_request, and RETURNING * returns the progress event fields flat, so it's {{ callback.RequestToken }} (the examples/aws/sqlserver stack uses the flat RequestToken). Worth fixing when this lands, or on its own.
• Code: run_callback_poll in src/core/utils.rs, run_callback / run_troubleshoot in src/commands/base.rs, the create and update callback blocks and the post-deploy exists -> troubleshoot path in src/commands/build.rs, and callback:delete in src/commands/teardown.rs.