feat(hardware-button): press phone volume and power buttons on Android and iOS
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 25/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Ferma
- Stack tecnologico
- android, ios, typescript
Direzione di ricerca
Start with packages/contracts/src/system-button-runtime.ts and the existing home/actionButton family binder, then trace the Android ADB, Appium, and local Apple runner owners named in the support matrix. Read the command registry and provider scenario guidance before mapping the CLI, MCP, and typed client API. Done means the listed platform refusals and button dispatches are covered, the completion conditions pass, and pnpm check:affected --run succeeds; PR #3859 is already open against this work.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Purpose
Agents can't press a phone's volume or power button. Today they fall back to raw adb shell input keyevent on Android and have no route at all on iOS. PR #3817 was going to add a hardware-key command, but it is being cut down to TV-only tv-remote keys, so phones are left out. This issue adds a small, honest phone vocabulary for those buttons. It is a separate decision from the TV remote keys.
iOS can press some hardware buttons; our implementation just has no route for them yet. XCUIDevice.Button has volumeUp and volumeDown, and Appium XCUITest mobile: pressButton accepts volumeup and volumedown on real devices. iOS does not expose a public power or lock button press.
API
agent-device hardware-button volume-up
agent-device hardware-button volume-down
agent-device hardware-button power
await client.command.hardwareButton({ button: 'volume-down' });
// => { action: 'hardware-button', button: 'volume-down', message: 'Pressed hardware button volume-down' }
- One positional argument,
<volume-up|volume-down|power>. No aliases. An unknown name returnsINVALID_ARGSbefore any device call. - No hold verb or duration flag in the MVP (see "Long press" below).
- No
--settleand no post-action observation. The command must not activate or re-foreground the session app. This usesaction-button's trait set, notpress's. - The result reports that the press was dispatched. It does not report a resulting volume level or screen state.
Ownership: extend the system-button family, not a new dispatcher
packages/contracts/src/system-button-runtime.ts already owns no-argument device presses (home, appSwitcher, actionButton): one fact cell per button, a family unsupported default, and bindSystemButton. This change adds volumeUp, volumeDown and power to SYSTEM_BUTTONS and SYSTEM_BUTTON_LABELS, and adds optional Interactor members volumeUp?(), volumeDown?(), power?(). The positional picks the cell, the way keyboard <status|dismiss|enter> picks a keyboard-family cell.
- Every owner that already calls
systemButtonRuntimeOperationFactsgets the new cells refused through its existingunsupporteddenial. Only the owners in the matrix below name a cell as available. - No
SystemControlunion, no key table shared withtv-remote, noandroid-keycode:Nescape hatch, and nocase 'hardware-button'dispatcher that bypasses the family binder. home,app-switcherandaction-buttonkeep their existing commands.hardware-buttondoes not accepthome,app-switcheroraction, so each button has exactly one public route. Foldingaction-buttonintohardware-buttonwould be a separate migration decision.- On TV targets (Android TV, tvOS),
hardware-buttonrefuses with a hint that points totv-remote. After #3817,tv-remoteowns TV volume and power keys.
Support matrix (MVP)
Every known platform, device-kind, and provider refusal happens at fact admission, before any adb, runner, or provider dispatch, and uses an existing RuntimeOperationUnavailability reason. A physical device hardware-presence check may run in the runner, but it must refuse before pressing an absent button.
| Owner / backend | Device kind | volume-up / volume-down | power | Mechanism / refusal |
|---|---|---|---|---|
| Local Android (ADB) | emulator, physical phone/tablet | available | available | input keyevent 24 / 25 / 26 (KEYCODE_VOLUME_UP/DOWN, KEYCODE_POWER), same route as home (keyevent 3) |
| Local Android (ADB) | Android TV target | refused | refused | unsupported-platform-leaf, hint: use tv-remote |
| Limrun Android | cloud emulator | available | available | same input keyevent route Limrun Android home/app-switcher use |
| Appium Android (UiAutomator2) | phone/tablet session only | available when the provider declares the capability | same | mobile: pressKey { keycode }, the call keyboard Enter already makes. Undeclared capability: owner-capability-missing with the provider note |
| Local Apple runner | physical iPhone/iPad | available | refused | XCUIDevice.shared.press(.volumeUp/.volumeDown), guarded by hasHardwareButton(_:) (iOS 16+). No public XCUIDevice lock/power button: unsupported-platform-leaf |
| Local Apple runner | iOS/iPadOS Simulator | refused | refused | unsupported-device-kind. The SDK header marks XCUIDeviceButtonVolumeUp/Down XCUI_SIMULATOR_UNAVAILABLE, so the runner code must be inside #if !targetEnvironment(simulator) |
| Local Apple runner | tvOS, macOS, watchOS, visionOS | refused | refused | unsupported-platform-leaf (tvOS hint: tv-remote) |
| Appium iOS (XCUITest) | real device | available when declared | refused | mobile: pressButton { name: 'volumeup' | 'volumedown' } (real devices only, per driver docs). mobile: lock sets lock state; it is not a button press, so it is not mapped to power |
| Appium iOS (XCUITest) | simulator | refused | refused | unsupported-device-kind. If the provider does not report the kind, the cell refuses rather than guessing |
| Limrun iOS | cloud simulator | refused in MVP | refused in MVP | SDK performActions types list buttonDown/buttonUp with volumeUp, volumeDown, lock and side, but there is no live evidence. See Future |
| Web, Linux, macOS desktop, HarmonyOS, Vega | — | refused | refused | each owner's existing family denial |
Within an admitted Apple leaf, if hasHardwareButton returns false at press time, the runner returns UNSUPPORTED_OPERATION with a hardware hint. This is the same pattern as actionButton in RunnerTests+CommandExecution.swift. It is never error-text matching.
Press events, not volume settings
hardware-button volume-down sends one button event. What the OS does with it (which stream changes, whether a HUD shows, whether the foreground app intercepts it) is device behavior, and the command does not claim any of it. Setting an absolute volume (Android cmd media_session volume --set, iOS: no public API) is a different feature. If it is wanted, it belongs in the settings catalog (packages/contracts/src/settings-catalog.ts, which has no volume entry today), not here.
power toggles the screen on Android. After a press that turns the screen off, later commands may hit a locked screen. The help text says so. It does not add an automatic wake.
Long press
Long press is not in the MVP. The backends do not agree:
- Android ADB:
input keyevent --longpress <code>, already used bytv-remote longpress. Appium Android:mobile: pressKey { isLongPress: true }. Both use a fixed system long-press duration, not a caller-chosen duration. - Local Apple and Appium iOS:
XCUIDevice.press(_:)has noforDuration:overload. Appium'spressButtonduration applies only to tvOS. - Limrun iOS:
buttonDown+wait+buttonUpcould hold for a duration, but this is unverified.
If long press is added later, represent it as a press shape, without a separate runtime-operation fact cell (ADR 0019 §2). Unsupported shapes must refuse before the button event is sent. Its API and validation are a separate decision.
Out of scope / future (each its own decision)
| Candidate | Evidence | Why not MVP |
|---|---|---|
sleep, wake |
Android KEYCODE_SLEEP 223 / KEYCODE_WAKEUP 224 (API 20), idempotent |
Android-only. Cheap to add as two more family cells once someone needs a lock/wake flow |
camera |
Android KEYCODE_CAMERA 27. Apple XCUIDevice.Button.camera (iOS, Xcode 16.3+). Appium camera (Xcode 16+, iOS 16+, real devices only) |
#2699 left Camera Control out on purpose (press only, no slide/light-press). Needs that decision, plus a physical iPhone 16 for evidence |
mute |
Android KEYCODE_VOLUME_MUTE 164 |
Android-only. iPhones have no mute key |
iOS power / lock |
No public XCUIDevice button. Appium mobile: lock / unlock / isLocked is lock state. Limrun lock/side buttons (types only) |
A lock-state command would be a different contract from a press |
| Limrun iOS volume | SDK types only | Admit once a live Limrun run shows a volume change |
| Siri | XCUISiriService.activate(voiceRecognitionText:) is a text query API, not a button |
Different feature. No promise here |
| Silent / ring switch | No XCUITest, Appium or Limrun API found | Unsupported. No promise |
| Absolute volume | — | settings catalog decision, see above |
Completion conditions
agent-device hardware-button volume-downon a local Android emulator returns{ action: 'hardware-button', button: 'volume-down', message }, andadb shell cmd media_session volume --stream 3 --getreads one step lower than before.volume-upraises it back.agent-device hardware-button poweron a local Android emulator changesadb shell dumpsys powermWakefulnessfromAwaketoAsleep. A second press restoresAwake.- On a physical iPhone over the local runner,
hardware-button volume-upsucceeds. The evidence is the system volume HUD in a screenshot taken right after, or a changedAVAudioSession.outputVolumeread in a fixture app. The session app is not activated or re-foregrounded. hardware-button volume-upon an iOS Simulator, andhardware-button poweron any Apple leaf, refuse at admission with a typed reason and hint. No runner request is issued, and a test asserts this. The runner still builds for the Simulator destination.- Appium Android: a unit test asserts
mobile: pressKeywith{ keycode: 24 | 25 | 26 }. Appium iOS: a unit test assertsmobile: pressButtonwith{ name: 'volumeup' | 'volumedown' }, and thatpoweris refused. A live run on any Appium Android session, and on a real-device Appium iOS session if available, is posted on the PR. If none is possible, the PR says so. hardware-button volume-upon Android TV and tvOS refuses with a hint totv-remote. On web, Linux, macOS, HarmonyOS, Vega and Limrun iOS, it refuses at admission.hardware-button sidewaysreturnsINVALID_ARGSbefore any adb, runner or provider call.- A provider scenario under
test/integration/provider-scenarios/covers the Android ADB argv and the response shape (integration-progressstays complete). The MCP tool andclient.command.hardwareButtonexist, and the MCP output schema is derived fromCommandResultMap. - CLI help,
website/docs/docs/commands.md, and every refusal hint name the same backends the facts admit. pnpm check:affected --runpasses, including the registry, owner-files and generic-route completeness gates.
Dependencies
- Blocked by: #3817 landing in its TV-only form (it removes
hardware-keyand folds TV volume/power intotv-remote). Rebase onto it so there is exactly one route per target class. - If the command registry has no room for a descriptor, make a statement-neutral registry split under ADR 0027. Do not raise its cap to bypass the gate. The TV-only update to #3817 removes its descriptor and its proposed cap increase.
- Device verification needs a physical iPhone for condition 3 (see
docs/agents/device-verification.md). The Android conditions run on any emulator.
Remaining uncertainties
- Whether a volume
XCUIDevice.presson a physical iPhone shows the HUD or changesoutputVolumewhile an XCUITest session is attached. Not live-verified. Appium documents it as working on real devices. - Whether a cloud Appium iOS session reports real-device vs simulator in a way the fact can read before dispatch. If not, the iOS cell refuses unless the provider declares real devices.
Sources
- Apple,
XCUIDevice.Button(home, volumeUp, volumeDown, action, camera; Xcode 16.3): https://developer.apple.com/documentation/xcuiautomation/xcuidevice/button - Apple,
hasHardwareButton(_:)(iOS 16+): https://developer.apple.com/documentation/xcuiautomation/xcuidevice/hashardwarebutton(_:) - Xcode 27.1 SDK header
XCUIAutomation.framework/Headers/XCUIDeviceButton.h:XCUIDeviceButtonVolumeUp/DownareXCUI_SIMULATOR_UNAVAILABLE.XCUIDevice.hhas onlypressButton:(no duration) and no lock button. - Appium XCUITest
mobile: pressButton,performIoHidEvent,lock: https://github.com/appium/appium-xcuitest-driver/blob/master/docs/reference/execute-methods.md#mobile-pressbutton - Appium UiAutomator2
mobile: pressKey(isLongPress),lock: https://github.com/appium/appium-uiautomator2-driver#mobile-presskey - Android
KeyEventconstants: https://developer.android.com/reference/android/view/KeyEvent @limrun/api0.64.1ios-client.d.tsPerformActionbuttonDown/buttonUpbutton names.- Precedent: #2699 / #2702 (action-button), #2715 (system-button family), PR #3817 (Android emulator volume transcript, as reported).
- Lingua principale
- TypeScript
- Stelle
- 5.1k
- Fork
- 339
- Merge medio
- 12h 23m
- PR unite (30g)
- 730
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di callstack/agent-device
-
bug ready-for-agent
Difficoltà 2/5 1-3 ore Idoneità per principianti 87/100
callstack/agent-device#3819 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
callstack/agent-device#1869 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 76/100
callstack/agent-device#3851 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
callstack/agent-device#3844 ·
I maintainer di solito rispondono entro 1 giorno
-
press/click: refuse a zero-area target instead of tapping its originForse già presa @faresrafat3 l’ha presa 1 giorno fa. Apertabug ready-for-agent
Difficoltà 2/5 1-3 ore Idoneità per principianti 20/100
callstack/agent-device#3841 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di callstack/agent-device
Issue simili
-
ready-for-agent
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
xavxyz/bei-shan-lin-shui#57 ·
I maintainer di solito rispondono entro 1 giorno
-
[BUG] rag: sentence content can cross the reported sentence boundariesForse già presa @LJK-ikun l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
bug needs-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
I maintainer di solito rispondono entro 1 giorno
-
README: the Resource URI Schemas example's file://documents/* pattern only matches that literal URIAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 88/100
platformatic/mcp#361 ·
I maintainer di solito rispondono entro 1 giorno
-
最近文档这个弹窗,左右宽度没有最小宽度限制Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
siyuan-note/siyuan#20799 · 7 commenti ·
I maintainer di solito rispondono entro 1 giorno