Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Terraform S3 resolver: unbounded, un-deduplicated concurrent state fetches cause intermittent `Error: aborted` under serverless-compose

Đã đóng
  • Có gì không ổn?
#13,844 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@czubocha đang làm issue này rồi.

Từ ngày 8/9/2026.

  • #13852 của @czubocha — đã merge

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Issue description

Using the Terraform S3 state resolver (type: terraform, backend: s3) across a serverless-compose project of 87 services, deploys intermittently fail during the variable-resolution phase with:

✖ <service>
    Failed to resolve variable '<output_name>' with resolver 'outputs' and provider '<resolver_name>':
    Error: Error fetching Terraform outputs from S3: Error: aborted

A single aborted fetch fails the entire compose deploy. Re-running the identical commit succeeds (87 services succeeded, 0 failed).

The failure is non-deterministic — the service and the output name differ on every occurrence:

Run Output that failed Service
Run 1 output A service A
Run 2 output B service B
Run 3 output C service A
Run 4 output B service A

What we ruled out

  • Object size / transfer time — the state objects are ~448 KB and have been flat for months. This is not a large-payload timeout.
  • A version regression — identical Serverless version (4.18.0) across the entire ~2-month period.
  • Credentials / network / permissions — other S3 operations in the same job (deployment-bucket uploads) succeed, and the same commit deploys cleanly on retry.
  • Config error — a wrong bucket/key/output would fail deterministically on the same variable every time; this fails on a different variable each run.

What the evidence points to: resolution-phase fetch concurrency

Compose resolves the variable graph for all services up front, so every terraform-type resolver fires its S3 GetObject in one burst. In our repo, 48 services declare such a resolver:

State object Services resolving it
State object A 46
State object B 28
State object C 2
State object D 1
State object E 1

That is ~78 concurrent S3 GETs issued from a single Node process, and 46 of them request the byte-identical object.

Error: aborted on a ~448 KB object under that load reads as a client-side abort (socket-pool exhaustion / request queued past a timeout) rather than transfer time. With no retry, one abort takes down the whole deploy.

This also explains why it has been getting worse over time: every new service that adds a terraform resolver increases the concurrent fetch count. The growth vector is the number of resolvers, not the size of the state.

Expected behaviour

Variable resolution should be resilient to a single transient fetch failure:

  1. De-duplicate identical fetches. A given (bucket, key, region) should be fetched once per invocation and shared across every service that references it. Today 46 services each issue a separate GET for the same object — wasteful independent of the aborts, and the main driver of the concurrency.
  2. Retry transient fetch failures with a small backoff (the AWS SDK's standard retry mode would cover aborted / reset sockets).
  3. Bound concurrency of resolver fetches instead of firing all of them simultaneously.

Actual behaviour

Each service resolves independently, with no caching, no retry, and no concurrency cap — so a single aborted GET fails the entire multi-service deploy.

Current workaround

Retrying the deploy step in CI. Compose is idempotent and skips unchanged services, so the retry only re-resolves and redeploys what aborted. This masks the symptom but leaves the underlying contention in place.

(1) alone would cut our resolution-phase fetches from ~78 to ~5 and would likely resolve this for us entirely. Happy to test a patch or provide more detail.

Context

Serverless Framework version: 4.18.0
Node: 24.x · Runner: GitHub Actions ubuntu-latest · Region: us-east-1

serverless.yml (redacted). Each affected service declares its own resolver of this shape:

frameworkVersion: '4'

stages:
  default:
    params:
      statebucket: <redacted-state-bucket>
    resolvers:
      tfstate:
        type: terraform
        backend: s3
        bucket: ${param:statebucket}
        key: <redacted>.tfstate
        region: us-east-1

...and references outputs from it, e.g.:

Endpoint: ${tfstate:outputs:<output_b>}

48 of the 87 services in serverless-compose.yml declare a resolver like the above, across 5 distinct state objects (see the table above).

Command:

serverless deploy --stage <stage>

(run from the compose root, i.e. all 87 services in one invocation)

Full error output:

✖ <service>
    Failed to resolve variable '<output_name>' with resolver 'outputs' and provider '<resolver_name>':
    Error: Error fetching Terraform outputs from S3: Error: aborted

Results: 36 services succeeded, 1 failed, 50 skipped, 87 total

On --debug: we don't have a stack trace to attach — the failure is intermittent in CI (roughly weekly) and does not reproduce on demand, so we haven't been able to catch a --debug run at the moment it fires. If you can point us at what you'd want captured, we're happy to add --debug to the pipeline and report back with the stack trace next time it occurs.

Ngôn ngữ chính
JavaScript
Star
46.9k
Fork
5.7k
Merge trung bình
22 giờ
Pull request đã merge (30 ngày)
50

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của serverless/serverless

Tất cả issue của serverless/serverless

Issue tương tự

Thêm issue về JavaScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.