Ship prebuilt iOS XCTest agent artifacts and re-sign them locally instead of building per device

Start with the existing xctest build flow, external .xctestrun handling through HARNESS_IOS_XCTESTRUN_FILE, simulator cache manifest, and release CI; the issue identifies these as the integration points. First verify cross-Xcode artifact reuse and re-signed installation on a physical iPhone, then work through the listed command, publishing, resolver, and fallback tasks. Done means a fresh checkout can launch a verified re-signed artifact without building, with fallback on mismatch or verification failure.

Summary

Harness builds the iOS XCTest agent from source on every run. On physical devices this costs a full
xcodebuild build-for-testing per machine, per run, and it requires a working signing setup on the
runner (including live Developer Portal access via -allowProvisioningUpdates).

Research suggests both costs are avoidable: publish prebuilt, unsigned agent artifacts and let
consumers re-sign them locally with their own certificate and provisioning profile. This is the
distribution model Appium already ships for WebDriverAgent, and it matches the intent already
documented for xctest build:

Unsigned device builds are allowed intentionally for workflows that upload the iphoneos artifacts
and re-sign them elsewhere. — website/src/docs/api/cli.mdx

This issue is the umbrella for that work. Findings below are measured on this repo (Xcode 27.0,
build 27A266a); the one material gap is listed under Unverified.

Observed findings

The artifact is fully relocatable

build-for-testing against generic/platform=iOS emits a .xctestrun that uses only the
__TESTROOT__ / __TESTHOST__ relocation macros. Copying Build/Products and the .xctestrun to an
unrelated path and running test-without-building works end to end:

AGENT HEALTH OK after ~16s
{"permissions":{"autoAcceptPermissions":false},"status":"ok"}
[HarnessXCTestAgent] HTTP server started on port 7099

The only absolute paths in the file are SourceFilesCommonPathPrefix and
ClangProfileDataDirectoryPath — coverage/instrumentation metadata, not read by coverage-ios.
getExternalXCTestConfiguration() already consumes an external .xctestrun via
HARNESS_IOS_XCTESTRUN_FILE, so the reuse path is largely wired.

No device coupling is baked in

• No UDID appears anywhere in the artifacts (grepped).
• No embedded Frameworks/. Binaries resolve @rpath/XCTest.framework/XCTest from device-local
  rpaths (/System/Developer/Library/Frameworks, /Developer/usr/lib). This is what lets one
  artifact serve many devices and OS versions — the device supplies its own XCTest.
• XCTest is linked with compatibility version 0.0.0, i.e. satisfied by any version.

Re-signing works and reproduces Xcode's own output

An unsigned device build leaves the .xctest plugin completely unsigned and the runner host app
carrying Apple's own com.apple.XCTRunner signature (TeamID 59GAB85EFG, copied from Apple's
XCTRunner.app template). Re-signing inside-out (plugin → runner + embedded profile → app) with plain
codesign produces:

valid on disk
satisfies its Designated Requirement        # --verify --deep --strict

with entitlements identical to a fresh Xcode-signed build (application-identifier,
com.apple.developer.team-identifier, get-task-allow=true, plus keychain-access-groups on the
runner; the .xctest plugin legitimately carries none).

Bundle-ID remapping also works without a rebuild — rewrite CFBundleIdentifier in all three
Info.plists plus TestHostBundleIdentifier and the crash-report IDs in the .xctestrun, re-sign,
launch. This is the escape hatch for accounts whose profile cannot cover com.callstackincubator.*.

Note: the re-signed binary is not byte-identical to Xcode's signed one (differs at byte 3889;
Xcode links its signature differently). Equivalence is established by launch + healthy agent, not by hash.

Cost comparison

Path	Wall time	Needs network	Needs Xcode
build-for-testing (clean)	10–22 s	yes, with automatic signing	yes
Re-sign a prebuilt artifact	0.9 s	no	yes (codesign only)

Artifact size

Compressed, after stripping *.dSYM and *.swiftmodule (~1.9 MB of dead weight):

• device (iphoneos, arm64): 582 KB
• simulator (iphonesimulator, arm64 + x86_64 universal): 712 KB

Current-state gaps

• Device artifacts land in .harness/xctest-agent/device, i.e. outside .harness/cache/, so
  action.yml (which globs xctest-agent-simulator-*) never caches them. Every physical-device run
  pays the build.
• No re-sign command exists.
• The strict cache.json compatibility manifest from #108 is applied to simulator only.
• SimulatorXCTestAgentCacheManifest keys on hostArchitecture, but simulator binaries are built
  universal (x86_64 arm64), so that axis is a no-op and can be dropped.

How many artifacts do we need?

iOS version is not a multiplier. Binaries do not link against a specific iOS; minos comes from
our IPHONEOS_DEPLOYMENT_TARGET and is a floor, not a version key; the .xctestrun filename encodes
the SDK, not the OS. Device-vs-simulator and Xcode version are the only real multipliers.

Assuming we support Xcode 16.4 through current (see version note below):

Axis	Multiplies?	Notes
Device vs simulator	yes, ×2	the only structural multiplier
Xcode version	yes	one set per supported Xcode minor
iOS version (15/16/17…27)	no	device-local XCTest + 0.0.0 compat version
Device model / UDID	no	only the profile must cover the device
iPhone vs iPad	no	UIDeviceFamily = [1, 2] (TARGETED_DEVICE_FAMILY = "1,2")
Host Mac arch	no	simulator build is universal; device build ignores host
User team / profile	no	resolved at re-sign time (0.9 s)

Version-range note (this is the trap)

Xcode 17 through 25 do not exist. Apple jumped straight from Xcode 16.4 to Xcode 26, then to 27,
to align with the year-based OS naming. So "one version before Xcode 26" cannot mean a 25.x line —
both plausible readings land on the 16.x line:

Reading	Xcodes in scope	Artifact sets (device + sim)
Release immediately preceding 26 → 16.4	16.4, 26.6, 27.1 (latest per line)	3 → 6
Any 16.x → Xcode 16.0–16.4	16.0, 16.1, 16.2, 16.3, 16.4, 26.x, 27.x	up to 7 → 14

Recommendation: key artifacts on the Xcode minor, publish only the minors CI actually pins plus
one current per line, and fall back to build-for-testing on a miss. That is 3 sets / 6 assets
(~3.9 MB total) as of today, retiring superseded pairs.

Highest-leverage open question: if an artifact built with the oldest supported Xcode also runs
under newer xcodebuild test-without-building sessions, we ship 2 artifacts total instead of 6.
The .xctestrun is FormatVersion 2 on both, which is encouraging but not proof. See task 0.

Two consequences of the 16.4 floor, from Apple's system requirements

Xcode	macOS required	On-device debugging	Simulator
16.4	Sequoia 15.3 – Tahoe 26.1.x	iOS 15+	iOS 15+
26.x	Sequoia 15.6 – Tahoe 26.x	iOS 15+	iOS 15+
27	Tahoe 26.6+	iOS 17+	iOS 17+

1. The minimum testable iOS version is set by the user's Xcode, not by us. Xcode 27 dropped
   iOS 15/16 device support entirely. This matches what we measured independently: the Xcode 27
   XCTRunner.app template carries MinimumOSVersion = 17.0, overriding our 15.0 deployment
   target. Harness should emit a clear diagnostic in this case instead of an opaque XCTest failure.
2. The publish matrix needs two macOS runner images. Xcode 16.4 and 26.x both run on macOS 15
   (16.4 needs ≥ 15.3, so macos-14 is too old); Xcode 27 needs macOS 26.6+. Note our e2e workflow
   currently pins 26.4.1.

Suggested behaviour

1. Publish artifacts as GitHub Release assets, not inside the npm tarball. Keep shipping the
   .xcodeproj source as the correctness floor. Key each asset on
   {agentVersion, platform, arch, xcodeVersion, sdkVersion} with a checksum.
2. Move device artifacts under .harness/cache/ and give them the same manifest-keyed validation
   the simulator path already has.
3. Add xctest resign with --teamId / --signingIdentity / --provisioningProfile / optional
   --bundleIdPrefix. Sign inside-out, embed the profile, apply the entitlements validated above,
   and gate on codesign --verify --deep --strict.
4. Reuse the existing launch path untouched: fetch → resign → set
   HARNESS_IOS_XCTESTRUN_FILE / HARNESS_IOS_XCTEST_DERIVED_DATA_PATH → the external-xctestrun
   branch handles the rest. No changes needed in ensureStarted.
5. Fall back to build-for-testing on artifact miss, Xcode/SDK mismatch, or verification failure.
6. Document that a dev profile with get-task-allow=true only runs on provisioned devices, and
   that test bundles must be reinstalled every session regardless — re-signing saves compilation,
   not installation.

Recommended implementation order

• 0. Prove or disprove cross-Xcode reuse. Build with the oldest supported Xcode, launch under
  the newest test-without-building. If it holds, the artifact count drops from 6 to 2 and most
  of the matrix work disappears. Everything downstream is sized by this answer.
• 1. Physical-device verification (see Unverified) — install and launch a re-signed device
  bundle on a real iPhone. This is the one blocking unknown.
• 2. xctest resign command + unit tests, with a --dryRun/inspect mode.
• 3. Publish pipeline in release CI for the supported Xcode matrix (two macOS images).
• 4. Artifact resolver — fetch, checksum, manifest validation, cache placement, device
  artifacts under the cache root.
• 5. Wire into prepare() ahead of buildXCTestAgent, with fallback.
• 6. Diagnostics for unsupported Xcode/iOS combinations and profile gaps.
• 7. Docs: artifact model, re-sign flow, iOS 17 floor under Xcode 27, free-account and
  bundle-ID-remap path.

Success criteria

• A fresh checkout with a valid profile reaches a running agent without build-for-testing.
• Re-signing completes in ~1 s offline and passes strict code-sign verification.
• Xcode/SDK mismatch rejects the artifact and rebuilds rather than launching a bad binary.
• Physical-device runs no longer depend on Developer Portal reachability at run time.

Non-goals

• Do not embed artifacts in the npm package.
• Do not attempt to support iOS below the floor imposed by the user's Xcode.
• Do not distribute signing identities, private keys, or provisioning profiles.
• Do not remove build-for-testing or the checked-in project.

Unverified (blocking)

• No physical-device test yet. Both available devices reported unavailable to devicectl and no
  re-sign tooling (resigner, pymobiledevice3, go-ios, ios-deploy) was installed. Device-side
  install and launch of a re-signed bundle is currently inferred from Appium's production practice,
  not observed. Simulator reuse and re-signing are verified; the device path is not.
• Cross-Xcode artifact reuse (task 0) is untested — only one Xcode was installed.
• Forward compatibility of an older-SDK artifact running on the newest iOS is unproven; low risk given
  weak-linked Swift runtime libraries and 0.0.0 XCTest compatibility, but worth a smoke test.

Prior art

Appium ships CODE_SIGNING_ALLOWED=NO "generic" WebDriverAgent builds on GitHub Releases, recommends
signing them with a wildcard profile, and provides download-wda / sign-wda. Free accounts, which
cannot mint wildcard profiles, are directed to bundle-ID remapping via resigner — the same escape
hatch measured above. Maestro, by contrast, still builds on the user's machine.

Related

• #108 closed with explicit non-goals "Do not cache device XCTest artifacts yet" and "Do not
  prebundle simulator binaries into the npm package" — this umbrella takes on both, on top of the
  cache layout and strict-manifest machinery that issue landed.