[coverage] Conformance findings: CLOUDFETCH-018

Bắt đầu với hai bài kiểm thử xfail được nêu tên trong coverage PR dưới tests/ và lần theo các đường dẫn drain của CloudFetch cho các backend Thrift và SEA, bao gồm hành vi retry và link-refresh được mô tả ở đây. Được coi là hoàn tất khi một lượt tải xuống bị đình trệ thực hiện ít nhất một yêu cầu cloud, trả về trong vòng 150 giây và đưa ra lỗi chứa timeout hoặc timed out thay vì bị chặn.

Summary

Surfaced by the multi-language coverage fan-out while conformance-testing these SPEC-IDs against databricks/databricks-sql-nodejs. Each finding is committed as an expected-failure (xfail) test in the coverage PR — the test asserts the CORRECT (post-fix) behavior and stays red until THIS driver (databricks/databricks-sql-nodejs) is fixed, then flips green as a tripwire.

Findings

• CLOUDFETCH-018 [thrift]: Thrift CloudFetch downloader enforces no absolute end-to-end per-chunk deadline: with every cloud GET stalled 180s the drain does not return within the 150s budget and no timeout-classified error surfaces (audit finding H05).
  • failing test: CloudFetch — Stalled Download Deadline > stalled chunk download — drain raises a timeout-classified error within an absolute per-chunk deadline [thrift] (see the coverage PR diff under tests/)
• CLOUDFETCH-018 [sea]: SEA kernel bounds a stalled CloudFetch chunk only by a static per-request timeout inside a 5-attempt retry loop, so the two multiply instead of bounding the chunk: with every GET stalled 180s the drain does not return within the 150s budget and no timeout-classified error surfaces (audit finding H05).
  • failing test: CloudFetch — Stalled Download Deadline > stalled chunk download — drain raises a timeout-classified error within an absolute per-chunk deadline [sea] (see the coverage PR diff under tests/)
• CLOUDFETCH-018: A stalled CloudFetch chunk download is not bounded by any absolute end-to-end per-chunk deadline on either backend: with every cloud GET held 180s the drain does not return within 150s and no error surfaces. The SEA kernel bounds each attempt only by a static per-request timeout under a 5-attempt loop ("Chunk N download failed (attempt 1/5): NetworkError … after 1 attempts … retrying"), so the two multiply instead of bounding the chunk; Thrift shows the same non-termination. Fix: enforce a wall-clock deadline over the whole chunk (connect + response headers + body + retry backoff + link refresh) and surface it as a timeout-classified terminal error (audit finding H05).

Reproduce & Expected

CLOUDFETCH-018 — A CloudFetch chunk download that STALLS -- the cloud-storage GET is accepted but response headers/body never arrive -- must be abandoned under an ABSOLUTE end-to-end wall-clock budget for that chunk,…

Reproduce:

• Stall EVERY CloudFetch download: the proxy accepts each GET and holds it for
  180s, so no attempt ever completes and the chunk can only finish by the driver
  giving up.
• A result large enough to be delivered via CloudFetch external links; drain it
  and expect a terminal timeout rather than a multi-minute block.

Expected (per the shared spec):

• full assertion contract:

result:
- label: stalled_drain
  exception_thrown: true
- label: stalled_drain
  elapsed_seconds_range:
    max: 150
- label: stalled_drain
  error:
    contains:
    - timeout
    - timed out
    - deadline
protocol:
  thrift:
  - label: stalled_drain
    cloud_downloads_min: 1
  sea:
  - label: stalled_drain
    cloud_downloads_min: 1

Context

• The behavior was first fixed in a DIFFERENT driver — reference PR: https://github.com/databricks/databricks-sql-kernel/pull/323 — which seeded the shared language-neutral spec. This issue tracks the same conformance gap in databricks/databricks-sql-nodejs; the reference PR is for cross-referencing the intended behavior, NOT a change to this repo.
• Coverage PR carrying the reproducing xfail test(s): https://github.com/databricks/databricks-driver-test/pull/1577