Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

iOS runner smoke: alert-observation XCTests are red or flaky on main

Closed
  • Is something wrong?
#2,546 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
ios, swift
Domain
mobile-dev, testing

Research direction

Start with apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift and run the two named alert-observation XCTests on a hosted macOS runner. Read the related alert-deadline tests and docs/agents/device-verification.md, then determine whether probe timing, deadline sizing, or runner ordering explains the empty action list. Done means a pinned fix or diagnosis, five consecutive green iOS Smoke Tests runs without reruns, and observeFixtureHome fixed or split into its own issue.

Written by the indexing model from the issue text.

Description

bug needs-triage

Purpose

Smoke Tests in the iOS workflow is not deterministic on main. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.

Observed on main

  • main@973b74cc14 (push run 34741577232, job 103682270067, step Run targeted iOS runner XCTest regressions), RunnerTests+AlertObservationTests.swift:72:

    XCTAssertEqual failed: ("false") is not equal to ("true")
- Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")

    Failing test: testAlertDismissDoesNotActivateAReplacementWithTheSameTitle.

  • PR #2533 head a16821b661 (run 34742468875 attempt 1, docs-only diff), same file at :75 and :80:

    XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")

    Failing test: testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.

Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through ALERT_DEADLINE_EXCEEDED.

Separate symptom on the same step family, worth tracking apart from the two above: main@8e8eeb2ced (run 34712516058) failed Run fixture-backed iOS simulator E2E smoke with an assertion raised at test/integration/ios-simulator-e2e/live-automation-scenario.ts:33 (observeFixtureHome).

Required behavior

  • The alert-observation assertions in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift pass or fail on their own terms, not on simulator alert presentation timing.
  • A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
  • Whatever the resolution, an unchanged main must not require a manual re-run to go green.

Observable completion conditions

  • A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
  • If it is timing: the probe or deadline change lands and the iOS workflow's Smoke Tests job is green across at least five consecutive main pushes, with no manual re-runs.
  • If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
  • The observeFixtureHome case is either fixed or split into its own issue with the above evidence linked.

Dependencies

  • Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or check:affected runs, so this is device-facing and docs/agents/device-verification.md applies.
  • Related history for the alert horizon: ADR-tracked alert deadline work behind testAlertResolutionCannotBypassRequestedDeadline and testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched, which pass in the same runs where the two above fail.

Discovered while babysitting #2533, whose diff touched no Apple source.

Dominant language
TypeScript
Stars
4.7k
Forks
304
Avg merge
11h 19m
Merged PRs (30d)
539

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from callstack/agent-device

All issues in callstack/agent-device

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.