Scrawl-style feedback capture with hold-Shift drawing and Apple transcription

Ouverte
#3,152 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
35/100
Type d'issue
Fonctionnalité
Clarté
À clarifier
Activité
Active
Stack technique
macos, rust
Domaine
cli, desktop

Piste de recherche

Start by reading crates/project/src/cursor.rs for cursor/modifier handling, apps/desktop-gpui/src/transcription.rs for existing Whisper/Parakeet transcription, and apps/cli/src/caps.rs for agent context commands. Check issues #2136 and #2059 for related drawing and frame export work. Done means Shift-draw while recording, macOS SpeechAnalyzer option, and a single CLI/MCP export of annotated context.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Feature

yo @richiemcilroy, can we make this a Cap feature? https://github.com/kitlangton/scrawl

The situation

I want to record a bug or UI change, talk through it, point at the exact thing, and hand the result to a coding agent. Scrawl packages the recording into timestamped screenshots, the matching narration, clicks and drawn marks. The agent gets the context without me typing it all out again.

This feels like a natural fit for Cap. I'd rather do it in the recorder I already use.

What works today, and why it is not enough

Cap already has a lot of the pieces:

I couldn't find the full Scrawl-style handoff, Apple speech backend, or live hold-Shift drawing in current main. Screenshot freehand drawing exists (#2136), but the ask here is drawing while recording.

https://github.com/user-attachments/assets/ceae6b30-09be-4e6d-b840-c788a9c6edbc

Ask

  • Hold Shift and move the pointer to draw on screen while recording. Release Shift to return to normal interaction. Keep those marks aligned with the recording and include them in exported frames/video. Ideally the modifier is configurable so it doesn't interfere with shortcuts in the app being recorded.
  • Offer Apple's on-device transcription on Mac. Scrawl already uses SpeechAnalyzer on macOS 26 and on-device SFSpeechRecognizer on macOS 15. Could Cap offer that alongside Whisper/Parakeet? It would need OS/language/model availability handling, but this seems worth supporting as a native option.
  • One action to copy/export context for an agent: selected screenshots or a contact sheet, timestamped narration, clicks and drawn marks, with a structured manifest and readable summary. Expose the same output through the CLI/MCP so an agent can inspect a specific moment. Let this work locally without requiring an upload.

Related: #2059 asks for native frame-at-timestamp export, which would help with this.

The main thing is the loop: record, talk, hold Shift to point something out, stop, send the context to the agent. Can we get that directly in Cap?

Langage dominant
Rust
Étoiles
25k
Forks
2.4k
Merge moyen
1 j 8 h
PR mergées (30 j)
676

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CapSoftware/Cap

Toutes les issues de CapSoftware/Cap

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.