Device view phase 2: Android H.264 video through the hub (scrcpy), plan
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 28/100
Direzione di ricerca
This issue is a multi-part implementation plan and names no source files. Start by reading the hub, shared device-view component, and CLI entry points, then verify WebCodecs support in Studio's macOS WebView as the plan requires. Work is done when the Android video path, input mapping, cleanup, and agent-frame fallback meet the listed acceptance criteria and tests pass.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Goal
A smooth, full-screen Android view in the device view (Studio, VS Code panel, probe device view) that also shows system UI and native screens, which the agent's Flutter-layer frames cannot. Android only. iOS stays on agent frames. No code in this issue; it is the plan for the work.
Why
Agent frames are Flutter-layer screenshots: no status bar, no keyboard, no native dialogs, and a rate bounded by capture cost on the device (the agent re-renders each frame). A hardware-encoded H.264 stream is cheaper on the device and shows everything. An earlier local spike exercised the pieces below.
What the spike found (facts to build on)
- scrcpy is Apache-2.0. Its server is one small jar (about 734 KB for the version tried) pushed to the device and started with
app_process; video and control travel over local sockets reached withadb forward. - The control channel takes touch events as fixed-size binary messages (32 bytes for a touch) in the video coordinate space, not the device's logical pixels. Using the wrong space was the one real mistake in the spike.
- Decoding in the viewer needs WebCodecs
VideoDecoder. The decode-rate spike for JPEG (#448) showedcreateImageBitmapis fast; WebCodecs availability in Studio's macOS WebView (WKWebView) still has to be verified before committing (see Risks).
Plan
- Find or fetch scrcpy, pinned. Order: a configured path, then
scrcpy-serverfound next to an installedscrcpy, then a download of one pinned release asset into the user cache. The pinned version and its SHA-256 are constants in the CLI; the download is checked against the digest and refused on mismatch (same approach as the CI integrations). Never download silently:probe device view --video(opt in) says what it fetches and from where. Offline and CI: use a configured path or fall back. - Hub: a video source beside the agent source. The hub keeps its single-client, token and origin rules. A new source starts the server on the device, forwards a loopback port with a random socket id (
scid), reads the video socket and relays packets as binary WebSocket frames with a new frame type; the hello'scapsgainsvideo: "h264". Theadb forwardand the on-device process are cleaned up when the view closes, on takeover, and when the hub exits (including--exit-on-stdin-close). - Viewer: WebCodecs decode in the shared component. Feature-detect
VideoDecoderwithisConfigSupported; configure from the stream's codec data; decode on a worker where possible; draw to the same canvas so hover, selection and the letterbox mapping keep working. Drop frames rather than queue when the decoder falls behind; keyframe request on resume. - Input through the control socket. Taps, swipes and text go to scrcpy's control socket from the hub, mapped from the canvas into video coordinates. Selection and hover (widget tree) still come from the agent, which reports logical pixels: the mapping between agent logical pixels and video pixels is one function with tests, because the two spaces differ by the device pixel ratio and any scaling option.
- Fallback to agent frames. If scrcpy is missing, cannot start (older Android, a blocked
app_process, a physical device with USB debugging limits), or the viewer has noVideoDecoder, the hub reportscaps.videoabsent and the existing agent frames are used, with a one-line reason in the header (the capability note Studio already shows). The first release changes nothing for people who do not opt in. - Licence and notices. Apache-2.0 allows redistribution, but the first release does not bundle the jar: it is fetched and cached by the user's machine, so the repository and the release binaries carry no scrcpy code. The docs state the source, the licence and that the user's machine downloads it. If bundling is wanted later, the NOTICE text and licence copy ship with the binary and the BSL licensing page lists scrcpy as a third-party component.
- Safety. Loopback only; the existing token; random
scidper session; no new listening port reachable from the network; the pushed jar goes to a fixed temp path and is removed on close; the command lines are built from constants and validated numbers (no user text reaches a device shell).
Acceptance
- On an Android emulator: 30 fps video at 720p or better in Studio, the VS Code panel and
probe device view, with taps, swipes and typing working in the right place after rotation. - CPU on the host stays below what the JPEG path costs at the same size (measure and record on a named device).
- No scrcpy process, forward or temp file is left after close, takeover, crash or Ctrl-C.
- Fallback proven: with the server missing, with an old Android API, and without
VideoDecoder. - Tests: protocol framing, coordinate mapping, cleanup, fallback; one browser test with a recorded H.264 clip; one device run on a named emulator.
Risks and open questions
- WKWebView and WebCodecs. If Studio's WebView lacks
VideoDecoder(or only supports it behind a flag), the viewer needs a software H.264 decoder (a WASM build) or Studio stays on agent frames while VS Code (Chromium) gets video. Verify first. - Physical phones: vendor builds may block
app_process-launched servers; treat as fallback, not an error. - Rotation and resolution changes mid-stream restart the codec; the viewer must reconfigure without a blank flash.
- Version drift: scrcpy changes its protocol between releases; the pinned version is the only supported one, and the CLI refuses a mismatching server.
- Running scrcpy next to a user's own scrcpy session: one client per device for control; document it.
Order of work (small PRs)
- Detection, pinned download with digest check, and cleanup helpers, with tests (no streaming yet).
- Hub video source and framing, with a fake server in tests.
- Viewer decode in the shared component, behind feature detection, with the browser test.
- Input mapping and control socket.
- Fallback, header note, docs, device run on a named emulator.
- Lingua principale
- Go
- Stelle
- 6
- Fork
- 3
- Merge medio
- 6h 28m
- PR unite (30g)
- 362
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di AlphaWaveSystems/flutter-probe
-
agent: is checked on a wrapper holding several checkables answers from the first one in tree orderApertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
AlphaWaveSystems/flutter-probe#865 ·
I maintainer di solito rispondono entro 1 giorno
-
area/cli bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
AlphaWaveSystems/flutter-probe#424 ·
I maintainer di solito rispondono entro 1 giorno
-
area/docs documentation
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
AlphaWaveSystems/flutter-probe#409 ·
I maintainer di solito rispondono entro 1 giorno
-
Dial timeout error should say how long it waited and name the timeout settingForse già presa @jay23606 l’ha presa 2 giorni fa. Apertaarea/cli bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
AlphaWaveSystems/flutter-probe#408 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
AlphaWaveSystems/flutter-probe#888 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di AlphaWaveSystems/flutter-probe
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
IBM-Cloud/terraform-provider-ibm#7115 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
fluxcd/notification-controller#1441 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
volcengine/volcengine-go-sdk#163 · 1 commento ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
OneBusAway/maglev#2145 ·
I maintainer di solito rispondono entro 2 giorni
-
priority/medium status/ready type/chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
discobox-ai/discobox#1357 ·
I maintainer di solito rispondono entro 1 giorno