[FR]: Gemini Live API integration for voice-based interaction
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Domain
- ai, audio-video-rtc, mobile
Research direction
Start by tracing the current text-only Interactions API flow in the chat UI and the existing AppFunction-to-Gemini tool schema conversion mentioned in the issue. Review the Gemini Live API documentation and Android audio requirements before deciding how streaming, tool calls, transcripts, and voice-mode controls fit together. Done means voice input and output work end to end while AppFunctions remain callable.
Written by the indexing model from the issue text.
Description
Is there an existing issue for this?
- I have searched the existing issues
Describe the problem
AppFunctions Studio currently only supports text-based interaction with the agent. For interactive showcases, being able to speak to the agent and hear responses would be dramatically more compelling and natural — especially when demonstrating how AppFunctions enable hands-free, voice-driven workflows on Android.
Describe the solution
Motivation
AppFunctions are fundamentally about enabling agent-driven automation without opening app UIs. The most natural interface for this is voice — a user says "create a note about tomorrow's meeting" and the agent executes it via AppFunctions. But the current testing agent forces users to type, which undermines the demo narrative.
Key use cases:
- Demos: Showing executives how AppFunctions work in a hands-free, conversational way
- Developer testing: Rapidly testing AppFunction flows via voice without typing complex queries
- Accessibility: Voice-first interaction for users who can't easily type on mobile
Technical Context
The agent currently uses the Interactions API with text input exclusively. Google already offers the Gemini Live API which supports:
- Real-time audio input/output (bidirectional streaming)
- Tool use / function calling — critical for AppFunctions execution
- Low-latency audio-to-audio via gemini-3.1-flash-live-preview
- Session management for multi-turn conversations
The Live API already supports tools, meaning the existing AppFunction → Gemini tool schema conversion could be reused. The agent would need:
- Audio capture (microphone) → streaming to Live API
- Live API processes speech, calls AppFunctions as tools (same flow as today)
- Audio response streamed back → speaker output
Suggested Implementation
- Add a "Voice mode" toggle or a push-to-talk button in the chat UI
- Show a transcript in the chat alongside audio (for clarity and debugging)
Additional context
No response
Code of Conduct
- I agree to follow this project's Code of Conduct
- Dominant language
- Kotlin
- Stars
- 210
- Forks
- 45
- Avg merge
- 12h 24m
- Merged PRs (30d)
- 3
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from android/appfunctions
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 90/100
android/appfunctions#66 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
android/appfunctions#62 ·
-
Difficulty 4/5 1-2 days Newbie friendliness 55/100
android/appfunctions#64 ·
-
[FR]: Testing agent should handle AppFunctionUriGrant results and pass returned images to the modelOpen
Difficulty 4/5 3-5 days Newbie friendliness 45/100
android/appfunctions#63 ·
-
Clarify production invocation, Android Auto support, and Android 16 fallback in AppFunctions docsOpen
Difficulty 4/5 3-5 days Newbie friendliness 52/100
android/appfunctions#51 ·
All issues in android/appfunctions
Similar issues
-
pending:code-area pending:issue-category to-triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
T31n/YagniLauncher#1096 ·
Maintainers usually reply within 1 day
-
win_build: parallel PostBuildEvent race in example_app_atiopencl and example_app_nvopencl copying openclapp_kernels.clPossibly taken @qixuancao claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
OkHttp versions 5.4.0+ no longer include baseline-prof.txtPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 1/5 Under an hour Newbie friendliness 83/100
lysine-dev/okhttp#9828 ·
Maintainers usually reply within 1 day
-
bug Impact: Low Work: Low
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
oppia/oppia-android#6532 · 1 comment ·
Maintainers usually reply within 3 days