Track host telemetry bugs: Azure metadata trickles bypass timeout

Start with scripts/nvx_tools/host_telemetry.py, especially read_azure_compute_metadata(), and follow its caller in _azure_vm_provenance(); review scripts/test_host_telemetry.py for existing timeout coverage. Add a local HTTP server test that trickles a valid response and a fast-response control, then run the focused host telemetry tests. Done means a slow response is bounded by the configured total timeout and recorded as a non-fatal Azure metadata error.

Bug hunt: 2026-10-12

Revision: latest fetched origin/dev, 9a2721a713a1bb78f0c94c189900cf89a728a26e.

Scope: benchmark host-telemetry provenance and sampling/error handling, then time-ABI status and warp-probe validation. These focused paths were selected after reviewing the open bug-hunt trackers; this is not a duplicate of the CPU qualification, TCP console, benchmark result collection, or performance-history findings.

Stopping rule: investigation stopped at the first confirmed bug. No repository files were modified and no commit was created.

1. Azure metadata trickle responses bypass the configured timeout

Severity: MEDIUM. Optional diagnostic provenance can hold up benchmark setup for far longer than the intended two-second metadata timeout. This is a host-side HTTP error-handling bug; it does not require a hypervisor or a guest failure.

Code and caller trace

• AZURE_IMDS_TIMEOUT_SECONDS = 2.0.
• read_azure_compute_metadata() passes the timeout to opener.open() and then calls response.read(MIB). The byte limit is not an elapsed-time limit.
• _azure_vm_provenance() synchronously calls the reader and records an error only if an exception is raised. Its documentation says the metadata query on another Hyper-V guest fails after its timeout.
• host_provenance() synchronously includes Azure metadata.
• benchmark_host_provenance() is the benchmark caller, used when constructing workload metadata and lifecycle results.
• The existing request test mocks Response.read() and verifies timeout forwarding, not a wall-clock bound on a real response.

Trigger

On a host for which may_be_azure_vm() is true, the metadata endpoint accepts the connection and sends response headers/body slowly, with each piece arriving before the socket's inactivity timeout. The reader has no end-to-end deadline. The benchmark coordinator waits synchronously for it.

Expected versus actual

Expected: a metadata query configured with timeout T should return or abort within approximately T, allowing the existing non-fatal azure.error path to continue benchmark setup when metadata is unavailable.

Actual: Python's HTTP socket timeout limits inactivity in individual blocking operations; it does not bound the complete response. A peer that keeps making progress can keep response.read(MIB) occupied well beyond T. The cached provenance wrapper cannot report the timeout while that read is still in progress.

The reproduction below returned valid JSON after 0.775 s with T = 0.100 s, without raising an error. The default two-second value has the same inactivity-only behavior. A larger slow response can hold up setup for minutes or longer; the one-MiB size cap does not enforce the intended two-second duration.

Smallest reproduction

Run the following Python script from a repository checkout whose Git objects contain the recorded revision. It can be saved outside the checkout or piped into python -B -; it creates no repository files. It loads the exact pinned source directly from Git, uses only a loopback HTTP server, and changes only the metadata URL in that in-memory module. No real Azure request, machine identity, credential, or hypervisor is involved.

import json
import subprocess
import threading
import time
import types
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from unittest.mock import patch

SHA = "9a2721a713a1bb78f0c94c189900cf89a728a26e"
source = subprocess.check_output(
    ["git", "show", f"{SHA}:scripts/nvx_tools/host_telemetry.py"], text=True
)
telemetry = types.ModuleType("pinned_host_telemetry")
exec(compile(source, "pinned_host_telemetry.py", "exec"), telemetry.__dict__)
TIMEOUT = 0.1
BODY = b'{"vmSize":"Standard_D16ds_v5"}'


class MetadataHandler(BaseHTTPRequestHandler):
    delay = 0.0

    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(BODY)))
        self.end_headers()
        for byte in BODY:
            self.wfile.write(bytes([byte]))
            self.wfile.flush()
            time.sleep(self.delay)

    def log_message(self, *args):
        pass


server = ThreadingHTTPServer(("127.0.0.1", 0), MetadataHandler)
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
try:
    url = f"http://127.0.0.1:{server.server_port}/metadata/instance/compute"
    with patch.object(telemetry, "AZURE_IMDS_URL", url):
        for delay in (0.0, 0.025):
            MetadataHandler.delay = delay
            start = time.monotonic()
            result = telemetry.read_azure_compute_metadata(timeout=TIMEOUT)
            elapsed = time.monotonic() - start
            print(
                f"timeout={TIMEOUT:.3f}s byte_delay={delay:.3f}s "
                f"elapsed={elapsed:.3f}s result={json.dumps(result)}"
            )
            if delay:
                assert elapsed > 5 * TIMEOUT, (elapsed, TIMEOUT)
    print("CONFIRMED: a valid trickle response exceeds the configured "
          "metadata timeout without raising.")
finally:
    server.shutdown()
    server.server_close()
    worker.join(timeout=2)

Observed output on Windows, Python 3.14.7:

timeout=0.100s byte_delay=0.000s elapsed=0.014s result={"vmSize": "Standard_D16ds_v5"}
timeout=0.100s byte_delay=0.025s elapsed=0.775s result={"vmSize": "Standard_D16ds_v5"}
CONFIRMED: a valid trickle response exceeds the configured metadata timeout without raising.

Suggested fix

Enforce a monotonic end-to-end deadline across connection, response headers, and body reads, rather than relying only on the socket inactivity timeout. Bound each read by the remaining budget, keep the response byte cap and proxy bypass, and preserve the cached, non-fatal azure.error behavior. Avoid a timeout wrapper whose shutdown still waits for the blocked read.

Add a regression test using a real local HTTP server that trickles a valid response at intervals below the socket timeout. Assert that the reader aborts near the total deadline and that the provenance wrapper records the error instead of holding up the benchmark. Include a fast-response control.

Explored and validation

• Read the repository guidance and the authoritative check-quality and validate-nvx action definitions before investigating.
• Fetched origin/dev and pinned the full SHA above. The reproduction executed the Git blob at that SHA. Relevant telemetry, time-ABI, caller, guest, and specification files used for tracing were also checked for equality between the existing worktree and that revision.
• python -B -m unittest scripts.test_host_telemetry -q: 27 tests, OK, one Linux-only test skipped. The Windows live-reader test ran.
• Independently loaded the pinned telemetry source and read live Windows scratch-volume, CPU, memory/writeback, and Defender counters: no reader errors. Machine-specific values were not recorded in this issue.
• Focused time-ABI field parsing, monitor, and warp-probe tests: 19 tests, OK.
• python -B -m unittest scripts.test_time_abi.RunnerWiringTests -q: 21 tests, OK.
• The real loopback HTTP reproduction above confirmed the finding. Investigation stopped immediately afterward.

Checked, not filed

• Telemetry metadata filtering, proxy bypass, mount selection, counter parsing, sampler cadence/truncation, final-sample retention, and phase-window summaries: existing focused tests passed; no separate confirmed bug was established.
• Device-I/O resume excluding diagnostic host provenance: explicitly documented behavior, not filed as a bug.
• Time-ABI status markers, chunked console monitoring, warp rounds, and runner wiring: traced through their callers and existing tests; no additional confirmed finding. Reviewed permissive parsing/cached-marker edge cases without establishing a production trigger or material impact, so these are not reported as bugs.

Validation limits

• python -B scripts\nvx.py verify could not pass because this worktree's OpenVMM submodule is uninitialized:

  error: initialized OpenVMM submodule not found: <worktree>\openvmm\Cargo.toml
  

  The submodule was not initialized or changed, in accordance with the read-only task.

• No NVX/OpenVMM build, guest boot, hardware benchmark, remote-host operation, or full cross-platform gate was run. Guest C status/probe paths were source-traced, not compiled or executed. The reported bug itself is confirmed using the exact host-side implementation and a real HTTP exchange, not a simulated HTTP reader.

• Final verification after filing: the tracking issue is open with both bug and bug-hunt labels, and all nine source permalinks use the recorded SHA. The worktree and index remain clean; HEAD is unchanged at c29f3d1b032938503369ba9c14a0063241317627, and the OpenVMM submodule remains uninitialized. No repository edits or commits were made.