iOS simulator input through the HID injector

Start by reading src/drivers/ios/screen-capture.ts and the Driver.screenCapture hook in src/core/driver.ts, then implement src/drivers/ios/input.ts against #1017's input port. Add writeStdin to src/ports/byte-pipe.ts and its fake, and extend vendor/screen-capture/Sources/simlock-screen-capture/main.swift with --input plus the re-vendored HIDInjector.swift. Verify with src/ports/byte-pipe.test.ts, the 0x82 handshake and keyboardUnavailable behaviour, and enabling the iOS tap case in e2e/slow-agent-device.test.ts.

Part of #1013.

Scope

After this PR, a person holding an iOS simulator lease taps, swipes, scrolls, presses Home, and types hardware keys on an iOS 26 or older runtime, through WS /v1/leases/{id}/input. A physical iPhone lease answers inputUnavailable: true. The app switcher, Lock, rotation and keys on iOS 27 runtimes are dropped; #1024 holds them.

In short

An iOS simulator's device input is the vendored capture program run in a new input mode. It loads serve-sim's HID injector, which Simlock now vendors, and reads input messages on stdin. Home brings SpringBoard forward with simctl, the way serve-sim does it on Xcode 26 and later. On an iOS 27 runtime the guest ignores injected keys, so Simlock drops keys there and says so in 0x82.

flowchart LR
  S[input module, #1017] --> P[device input port, #1017]
  P --> I[iOS device input NEW]
  I -- spawn --> H[simlock-screen-capture --input NEW]
  I -- stdin: len, tag, JSON --> H
  H -- HIDInjector, re-vendored --> SIM[(simulator)]
  I -- simctl --set launch springboard --> SIM
  P --> X[physical iPhone: unavailable]

• input mode: simlock-screen-capture <udid> --input, with SIMLOCK_IOS_DEVICE_SET set as for a capture. It writes no picture.
• On stdin, each message is the stream's envelope: 4-byte big-endian length, one tag byte, then the serve-sim JSON, for example tag 0x03 with {"type":"begin","x":0.5,"y":0.25}. End of stdin stops the helper.
• On stdout, the helper writes one envelope with tag 0x82 and {"width":<px>,"height":<px>} once its HID client is set up. The size is the device type's native screen size (F19).
• 0x82 on iOS always says "orientation":"portrait". It adds "keyboardUnavailable":true when the simulator's runtime is iOS 27 or later, or cannot be read (F10).
• Failed, not unavailable: a missing binary, no 0x82 within 10 s, or a helper that exits early makes the open reject, and #1017 closes with 1013. A helper that exits later calls onFailed.

If this goes wrong, a person sees taps do nothing, or a helper left running after the socket closed.

Technical spec

Facts

Commit e1f2f43e6.

#	Fact	Source
F1	Our serve-sim copy is from commit 6d009686ed08f7d8bd19bca9f9f67a9237931f44.	vendor/screen-capture/NOTICE.md:18
F2	HIDInjector.swift, DeviceHubKeyboardBridge.swift, DuoHIDBridge.swift and sim-module.swift were not taken.	vendor/screen-capture/NOTICE.md:93-103
F3	DevicePose.swift and DeviceHubKeyboardRouting.swift were taken and later removed.	vendor/screen-capture/NOTICE.md:87-91
F4	Our FrameCapture.findSimDevice looks the UDID up in the set named by SIMLOCK_IOS_DEVICE_SET.	vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift:425-436
F5	The capture program reads stdin only as a liveness handle; EOF means stop.	vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift:280-285
F6	The driver spawns the capture with the UDID and flags, and sets SIMLOCK_IOS_DEVICE_SET for a simulator only.	src/drivers/ios/screen-capture.ts:90-102, src/drivers/ios/screen-capture.ts:120-124
F7	A stop closes stdin, then kills after a 2 s grace.	src/drivers/ios/screen-capture.ts:29
F8	The envelope: 4-byte big-endian length (payload + 1), tag byte, payload.	src/core/screen.ts:71-77
F9	scripts/build-capture.mjs builds each product with swift build --product and copies it to dist/bin/.	scripts/build-capture.mjs:46-61, scripts/build-capture.mjs:70-73
F10	iosRuntimeVersionFromId reads the iOS major and minor from a runtime id, or undefined.	src/drivers/ios/index.ts:2157-2167
F11	BytePipeProcess offers stdout, exited, closeStdin() and kill(); nothing writes to stdin.	src/ports/byte-pipe.ts:15-22
F12	NodeBytePipeLauncher drops every stdin error, because EPIPE on a stop is expected.	src/ports/byte-pipe.ts:33-35
F13	FakeBytePipeProcess records only stdinClosed and stdinCloses for stdin.	src/ports/byte-pipe.ts:70-75, src/ports/byte-pipe.ts:113-117
F14	A driver offers its screen capture through an optional screenCapture member; #1017 adds input beside it.	src/core/driver.ts:374
F15	HIDInjector.setup loads the frameworks and finds the device through FrameCapture.findSimDevice.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:83-100
F16	HIDInjector imports SimNativeSupport and uses DuoHIDBridge for Duo devices, DeviceHubKeyboardBridge for keys, and DevicePose for poses.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:5, v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:44, v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:75, v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:550-556
F17	sendTouch takes begin, move or end and a fraction position; sendScroll emulates a scroll as a touch drag.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:209-215, v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:370
F18	home relaunches SpringBoard with xcrun simctl launch <udid> com.apple.springboard, with no --set, because "Xcode 26+ silently drops the Indigo HID home-button press"; app_switcher and lock are Indigo button presses.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:464-500, v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:621-626
F19	FrameCapture.nativeScreenSizes(for:) (private) gives the device type's native pixel sizes, falling back to mainScreenSize; getScreenSize answers only after frames arrive.	vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift:305-308, vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift:358-366
F20	sendKey tries the Device Hub bridge first, then the Indigo keyboard message.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/HIDInjector.swift:277-300
F21	The iOS 27 guest no longer consumes Indigo keyboard messages.	EvanBacon/serve-sim v6d009686:packages/serve-sim/Sources/SimNative/DeviceHubKeyboardBridge.swift:7-10
F22	Expo serve-sim reports keyboardUnavailable in its screen config when keys cannot be injected.	expo/expo-device-hub v1357114:packages/serve-sim/packages/serve-sim/src/device-session.ts:2809, v1357114:packages/serve-sim/packages/serve-sim/src/device-session.ts:2826
F23	A physical iPhone is captured by a second program; nothing in it injects input.	src/drivers/ios/screen-capture.ts:12-16, vendor/screen-capture/Package.swift:61-64
F24	The slow lane's agent-device test opens a lease per platform, installs the Appduct fixture, and its snapshot shows tab-status.	e2e/slow-agent-device.test.ts:118-123, e2e/slow-agent-device.test.ts:176-187

Modules touched

Searches: screenCapture, createIosScreenCapture, findSimDevice, nativeScreenSizes, SIMLOCK_IOS_DEVICE_SET, HIDInjector, build-capture, BytePipeProcess, closeStdin in src/, vendor/, scripts/; snapshot in e2e/.

• vendor/screen-capture/Sources/simlock-screen-capture/: HIDInjector.swift (re-vendored, patched), an input mode in main.swift, FrameCapture.nativeScreenSizes made internal.
• vendor/screen-capture/NOTICE.md, docs/internal/VENDORED-CODE.md, vendor/README.md: the file taken and two new deviations.
• src/ports/byte-pipe.ts, src/ports/byte-pipe.test.ts, src/ports/AGENTS.md: BytePipeProcess.writeStdin(bytes); the fake records stdinWrites.
• src/drivers/ios/input.ts (new): the iOS device input.
• src/drivers/ios/index.ts: sets the driver's input (#1017's Driver.input) next to screenCapture.
• src/drivers/ios/AGENTS.md, website/src/content/docs/docs/using-devices/watching-the-screen.mdx.
• e2e/slow-agent-device.test.ts: turns on, for iOS, the tap case #1018 adds (#1018 skips it on iOS).

flowchart LR
  core[core driver catalog, #1017] --> drv[ios/index.ts *]
  drv --> inp[ios/input.ts NEW]
  inp --> bp[ports/byte-pipe * writeStdin]
  bp --> bin[simlock-screen-capture --input *]
  bin --> hid[HIDInjector.swift re-vendored]
  bin --> fc[FrameCapture *]

sequenceDiagram
  participant M as input module
  participant I as iOS device input
  participant H as helper --input
  M->>I: openInput(simulator)
  I->>H: spawn udid --input, SIMLOCK_IOS_DEVICE_SET
  H-->>I: 0x82 {width, height}
  I-->>M: device input; onConfig (+ keyboardUnavailable on iOS 27)
  M->>I: 0x03 touch begin
  I->>H: writeStdin [len][0x03]{...}
  Note over I,H: no 0x82 in 10 s
  I-->>M: open rejects (1013)
  Note over H: helper exits later
  I-->>M: onFailed (1013)

Surface and extension points

createIosDeviceInput(options) returns the driver's input, the implementation of #1017's device input port (feature Shared values, "Device input port"). Its options mirror IosScreenCaptureOptions: launcher, command, deviceSet, baseEnvironment, clock, plus a process runner for simctl. A physical iPhone gets {unavailable: "Input is not supported on physical iPhones"}. It is one of two implementations beside #1018's; the port and Driver.input are #1017's.

BytePipeProcess.writeStdin(bytes): Promise<void> (new): writes bytes to the child's stdin and rejects when the write fails. NodeBytePipeLauncher still ignores EPIPE after closeStdin() (F12), but a write before that surfaces its error.

Contract and event changes

none. Searched src/contract/ for input shapes: the messages are #1017's core/input.ts. No event.

Other code on the same state

Searches: simlock-screen-capture, SIMLOCK_IOS_DEVICE_SET, simctl launch, springboard, closeStdin in src/ and vendor/ (7 hits).

• A running capture of the same simulator: both are separate processes of the same binary. The input mode opens a HID client, the capture a framebuffer; neither reads the other.
• The capture's own use of BytePipeProcess: it never calls writeStdin; its closeStdin and EPIPE handling are unchanged.
• agent-device on the same simulator: both inject input. Whichever lands last wins; that is the take-over the feature asks for.
• home relaunches SpringBoard. An app the agent was driving goes to the background, as a Home press does.
• The health monitor rebooting a crashed simulator under the lease: the helper keeps a HID client bound to the old boot, so taps may vanish while it stays alive. A known gap, as the capture's own "a screen stream can freeze after a reboot under its lease" (src/drivers/ios/AGENTS.md, Known gaps).

Failure modes

Call	Outcome	What happens	Limit, counted from
spawn helper	ok, 0x82 on stdout	open resolves; onConfig with the size	10 s, from spawn
spawn helper	binary missing (exit with error)	open rejects; #1017 closes 1013	none
spawn helper	no 0x82 in 10 s, or exits first	helper stopped; open rejects	10 s, from spawn
writeStdin	rejects	onFailed; helper stopped	none
helper	exits mid-session	onFailed; #1017 closes sockets with 1013	none
xcrun simctl --set <set> launch <udid> com.apple.springboard	error	Home does nothing; logged	10 s, from the call
close	ok	stdin closed; killed after 2 s	2 s, from close (F7)

A worker restart mid-session leaves a helper whose stdin closed, so it exits (F5). #1017 shares one device input per lease, so two helpers never run for one lease.

Rules in play

• vendor-and-third-party-code.md rule 6, "names its origin per file, never by a count": NOTICE.md's file table gains HIDInjector.swift.
• vendor-and-third-party-code.md rule 11, "updates the notice and the inventory in the same change": NOTICE.md, the VENDORED-CODE.md row and vendor/README.md.
• VENDORED-CODE.md "How a new vendored thing gets here", step 4: src/source-hygiene.test.ts checks the shape.
• NOTICE.md "Deviations": two new ones. HIDInjector drops the Duo, Device Hub, pose, Home, app switcher and Lock paths (F16, F18): Home is the TS device input's simctl --set … launch … springboard, so it has one owner (architecture rule 10); FrameCapture.nativeScreenSizes is internal, not private (F19).
• architecture rule 2, "iOS and Android are encapsulated in their own driver modules": HID and the runtime rule stay in src/drivers/ios/.
• architecture rule 11, "Every wait that crosses a process boundary is bounded": spawn, ready, simctl and close have limits.

Tests

Seam: createIosDeviceInput with a FakeBytePipeLauncher and a scripted helper, on a FakeClock; model on src/drivers/ios/screen-capture.test.ts. The port change extends src/ports/byte-pipe.test.ts.

• Touches, buttons, scrolls and keys on an iOS simulator reach the HID helper: touch begin/move/end written to stdin as envelopes; home runs simctl --set <set> launch <udid> com.apple.springboard; lock and app_switcher dropped; scroll written to stdin; a key on an iOS 26 runtime written to stdin; a key on iOS 27 or an unknown runtime dropped, with keyboardUnavailable: true in the config; helper missing → rejected; no 0x82 from the helper in 10 s → rejected (at 9.9 s still waiting); helper exits → failed.
• A physical iPhone's device input is unavailable: physical device; no process is started.

Done when

• The two Tests lines and the byte-pipe test pass in pnpm test, and src/source-hygiene.test.ts passes.
• pnpm build:capture builds simlock-screen-capture with the input mode where Xcode is selected (toolchain: Swift is built only there).
• On a simulator, e2e/slow-agent-device.test.ts opens a lease and the Appduct fixture through agent-device, opens WS /v1/leases/{id}/input, gets 0x83 and 0x82, taps the centre of tab-status as agent-device's snapshot places it (as a fraction of the screen size in the same snapshot --json), and the next snapshot shows the Status tab's screen (slow lane, iOS simulator). This task turns on, for iOS, the tap case #1018 adds.
• watching-the-screen.mdx lists what input does on iOS: touch, scroll, Home, keys on iOS 26 runtimes; not the app switcher, Lock, rotation or keys on iOS 27 (#1024).

Out of scope

• Rotation, the app switcher, Lock, and keys on iOS 27 runtimes (#1024).
• A 0x82 that follows the simulator's real orientation. The HID takes positions as fractions of the device's native frame, and today's capture keeps that frame's size after a rotation, so picture and taps stay in the same frame (Assumption for the implementer to confirm on the slow lane).
• Duo devices and their hinge.
• Input on physical iPhones.

Depends on

• #1017
• #1018 (adds the shared tap case in e2e/slow-agent-device.test.ts this task turns on for iOS)

Written by an agent.