Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Stabilize tls_stress_test metrics queries under exhausted session caps

Closed
  • Is something wrong?
#8,167 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
networking, testing-qa

Research direction

Start in tests/connections.py at get_session_metrics(), run_connection_caps_tests(), and the nested create_connections_until_exhaustion(), then inspect tests/infra/clients.py for httpx.ReadError conversion. Run tls_stress_test repeatedly through tests.sh on Azure Linux 4 and determine the intended existing-session behavior at the caps. Done means transient metrics failures no longer make the test flaky while persistent failures and active/peak/soft-cap/hard-cap assertions remain actionable.

Written by the indexing model from the issue text.

Description

ci failed-test testing

Summary

tls_stress_test is intermittently failing in AL4 VMSS Virtual C while the
connection-cap scenario deliberately exhausts sessions and file descriptors.
After expected connection failures, the test performs an unguarded
GET /node/metrics through clients[0]. That request may be reset or receive
503 SessionCapExhausted, failing the whole CI job.

This has occurred on unrelated PRs and predates the PR where it was most
recently observed.

Most recent occurrence

tests/connections.py:231 run_connection_caps_tests
tests/connections.py:176 create_connections_until_exhaustion
  r = clients[0].get("/node/metrics")
tests/infra/clients.py:696
  raise CCFIOException from exc
infra.clients.CCFIOException

The underlying exception was:

ConnectionResetError: [Errno 104] Connection reset by peer
httpx.ReadError: [Errno 104] Connection reset by peer

The normal VMSS Virtual C job passed on the same PR commit; only the AL4 job
failed:

Evidence that this is pre-existing and intermittent

The same request at tests/connections.py:176 failed with
infra.clients.CCFIOException on July 26:

That recovery PR did not modify tests/connections.py or the TLS/session-cap
implementation. Its preceding AL4 run passed, this run failed, and its next 11
AL4 runs passed without a change to tests/connections.py.

Related failures in the same connection-cap scenario have also returned
503 SessionCapExhausted from metrics queries:

The metrics request at tests/connections.py:176 dates from 2021. The loop
immediately before it already treats CCFConnectionException,
CCFIOException, and RuntimeError as expected while searching for the
session/file-descriptor limit, but the subsequent metrics request has no
equivalent handling.

Relevant code

  • tests/connections.py
    • get_session_metrics()
    • run_connection_caps_tests()
    • nested create_connections_until_exhaustion()
    • particularly the unguarded clients[0].get("/node/metrics")
  • tests/infra/clients.py
    • conversion of httpx.ReadError to CCFIOException

Investigation notes

Determine which invariant is intended:

  1. An already admitted session must remain usable while new sessions are
    rejected. If so, the reset/503 may expose a CCF session-cap bug and the test
    should retain a strict assertion.
  2. A reset or temporary 503 is valid while the test is deliberately exhausting
    file descriptors and session caps. If so, the test should explicitly wait
    for cap state to settle and retry the metrics query using an appropriate
    existing session, rather than failing on a timing-dependent request.

Avoid simply swallowing the exception. Preserve the assertions on active/peak
session metrics and ensure a persistent inability to query metrics still fails
with useful diagnostics.

Also check whether the test should use the existing get_session_metrics()
helper consistently rather than making a separate direct request.

Suggested validation

  1. Build Debug on Azure Linux 4, matching the failing workflow.
  2. Run tls_stress_test repeatedly through the repository tests.sh wrapper.
  3. Confirm the connection-cap assertions still detect real cap regressions.
  4. Confirm transient reset/503 behavior no longer fails the test if it is
    considered valid.
  5. Run the complete AL4 bucket C job.

Acceptance criteria

  • The intended behavior of existing sessions at the soft/hard cap is explicit
    in the test.
  • tls_stress_test is stable under repeated AL4 runs.
  • Persistent metrics-query failure remains actionable and fails the test.
  • Session active/peak/soft-cap/hard-cap assertions are preserved.
Dominant language
C++
Stars
874
Forks
263
Avg merge
1d 14h
Merged PRs (30d)
157

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/CCF

All issues in microsoft/CCF

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.