Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Linux X11/Xvfb: auto-sample cursor telemetry when ScreenCast METADATA is unavailable

Aberta
  • Algo errado?
#1,415 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@Jasonsu22 já está trabalhando nisso.

Desde 28/10/2026.

  • #1 de @Jasonsu22 — aberto
  • #1416 de @Jasonsu22 — aberto

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
25/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Estagnada
Stack de tecnologia
electron, linux, typescript
Domínio
backend-api-design, desktop

Direção de pesquisa

Start by reading the existing openscreen record --cursor editable-overlay and openscreen export --auto-zoom paths, then inspect the fork implementation linked in the issue. Determine how X11 pointer samples can produce the v2 sidecar when ScreenCast METADATA is unavailable, and confirm the schema with maintainers; done means automatic sampling and sidecar output work on the described X11 setup.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Summary

On some Linux cloud-agent desktops (X11 + Xvfb), the xdg-desktop-portal ScreenCast interface reports empty source/cursor modes (AvailableSourceTypes=0 / no METADATA cursor mode). OpenScreen’s live editable-cursor capture therefore cannot write a .cursor.json sidecar even though the screen can still be recorded and the pointer can be sampled separately.

Agents can still:

  1. Record with ffmpeg x11grab (draw_mouse off) or OpenScreen’s non-METADATA path
  2. Sample pointer positions + clicks (Electron screen.getCursorScreenPoint(), xdotool / xev / xinput / libinput)
  3. Drive openscreen export --auto-zoom from a v2 <video>.mp4.cursor.json sidecar

We implemented an automatic fallback on our fork and would like to contribute it upstream.

Environment (repro host)
Item Value
Host Grok Bot cloud Linux box (grok-bot-vm-316768464)
Kernel Linux 6.12.94+ x86_64
OS Debian GNU/Linux 13 (trixie)
CPU / RAM 8 vCPU / ~15 GiB
Session X11 on DISPLAY=:2 (Xvfb 1920x1200x24 + Mutter/GNOME pieces)
Wayland No
Portal xdg-desktop-portal (+ gnome/gtk) present; ScreenCast reports no usable METADATA cursor mode (AvailableSourceTypes=0 / empty cursor modes in practice; CreateSession / editable capture fails)
GPU Mesa llvmpipe (software Vulkan/GL); OpenScreen export falls back to CPU
OpenScreen Linux AppImage 2.1.2
PipeWire Present (pipewire + wireplumber)
Pointer tools xdotool, xev available
Proposed behavior

When ScreenCast METADATA cursor capture is unavailable on X11:

  • Automatically sample the pointer during record
  • Write <recording>.mp4.cursor.json (v2: timeMs, normalized cx/cy, move|click|mouseup) next to the video
  • So openscreen export --auto-zoom needs no manual import

Optional CLI for external ffmpeg tracks: openscreen cursor import --csv … --clicks … --video …

Env knobs: OPENSCREEN_DISABLE_X11_CURSOR=1, OPENSCREEN_FORCE_X11_CURSOR=1

Workaround today (without the PR)

ffmpeg x11grab + CSV/clicks → converter → sidecar → openscreen export --auto-zoom

Ask
  1. Accept auto X11 sampling when METADATA is missing (primary path for agent hosts)
  2. Document the sidecar / optional CLI import path
  3. Confirm preferred schema fields beyond { version: 2, samples: [...] }

Happy to adjust the PR to maintainer preference.

Related
Linguagem predominante
TypeScript
Estrelas
3.7k
Forks
231
Merge médio
15h 10min
PRs com merge (30d)
324

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de getopenscreen/openscreen

Todas as issues de getopenscreen/openscreen

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.