Device view: should the agent produce JPEG frames, and with what encoder?

Open
#455 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
dart, flutter, go
Domain
mobile, testing-qa

Research direction

Read the device view hub's probe.screenshot handling and the Dart agent's screenshot capture path, then compare the listed encoding options. The issue asks for a design decision and physical-device measurements before choosing; completion means selecting an approach based on PNG versus JPEG size, encode time, and Wi-Fi frame rate.

Written by the indexing model from the issue text.

Description

Question

The device view hub asks the agent for JPEG frames (probe.screenshot with format: "jpeg", quality), but the Dart agent can only produce PNG: dart:ui offers ImageByteFormat.png, rawRgba and rawStraightRgba, no JPEG. Today the hub therefore falls back to PNG and reports caps.jpeg as false. Should the agent gain a JPEG encoder, and if so, which kind?

What a JPEG saves (measured on the host, not on a device)

Same frames, converted with sips (macOS), at 603x1311:

Frame JPEG (q60) PNG Ratio
photographic 41 KiB 474 KiB 11.5x
UI screenshot 28 KiB 87 KiB 3.1x

Typical app screens sit between the two. Over USB or local Wi-Fi the PNG size is tolerable; it matters on a slow link and for the rate the hub can sustain. The WebKit decode spike (#448) shows the viewer side is not a bottleneck either way (JPEG decode 3-6 ms).

Options

  1. package:image (pure Dart). Current version 4.12.0 (published 2026-10-11, pub.dev), MIT licence, null-safe, supports every platform, depends on archive (^4.5.0, which depends on path and posix). The agent today depends on the Flutter SDK only, so this adds two third-party packages to every app that includes the agent, even though the agent is a dev dependency. Cost on the device: the frame must be read as raw RGBA (toByteData(rawRgba), about 3.2 MB at 603x1311) and encoded in Dart on the UI isolate or a helper isolate, which is CPU on the device being tested and can disturb the timing of the app under test. Not measured yet; it needs a device run.
  2. A native channel (a small platform plugin per platform: UIImage.jpegData on iOS, Bitmap.compress on Android). Fast and uses the platform encoder, but the agent currently ships no native plugin code, so this means a federated plugin, native build files in the user's app, and per-platform maintenance. It also breaks the "add one Dart dependency" install story.
  3. Do nothing in the agent; encode on the host. Keep PNG between agent and CLI and re-encode to JPEG in the hub (Go standard library image/jpeg). The viewer gets small frames, the device does no extra work, and nothing changes in the agent. The device-to-host link still carries the PNG, which is the part a slow link would feel.
  4. Lower the PNG cost instead: capture at the scale the viewer needs (already done by the scale parameter), which cuts bytes roughly with the pixel count.

Recommendation (to decide)

Option 3 first (no agent change, works with every released agent) plus option 4, which already exists. Revisit option 1 only if a measurement on a physical phone over Wi-Fi shows the PNG link is the limit; option 2 only if option 1's CPU cost disturbs tests. No code is proposed here.

To measure before choosing

  • PNG vs JPEG size and encode time on a physical phone at the pane's usual scale.
  • Frame rate the hub sustains over Wi-Fi with PNG at scale 0.5 and 1.0.
Dominant language
Go
Stars
6
Forks
3
Avg merge
6h 28m
Merged PRs (30d)
362

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AlphaWaveSystems/flutter-probe

All issues in AlphaWaveSystems/flutter-probe

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.