【文章推荐】豆包手机之后,开源 AI 怎么操作 iPhone 和 Android?

Open Beginner friendly
#207 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Domain
content

Research direction

Read the issue text and follow its linked HotRepo article to review the referenced story. Confirm that the Chinese article recommendation fits the weekly publication and that its link works; done when the recommendation is published with the intended article link. The issue names no repository files or tests.

Written by the indexing model from the issue text.

Description

AI 怎么操作手机?拆开看就两件事:怎么读屏,怎么把点按送进去。

模型通常看截图,也可以结合系统提供的控件信息,判断下一步该点哪里。真正区分各个方案的是:谁有权限截图、点按和输入?

Android 有三条常见路线:

  • 无障碍服务:读取界面控件,通过系统辅助功能执行点击、滑动。用户要主动授权,Agent 通常和你共用主屏。
  • 电脑 + ADB:电脑通过调试连接获取截图、发送输入。适合测试机,也适合让电脑上的编程 Agent 操作手机。
  • 无线调试 + 虚拟屏:一些项目把目标 App 放到后台虚拟屏,在那块屏上截图和点按,主屏可以继续用。后台屏能否独立接收输入、锁屏后能否继续运行,要看 Android 版本和设备。

iPhone 有四条路线:

  • WebDriverAgent:把 XCTest 的界面测试能力交给 Agent,用它读取控件、点击和输入。需要签名和开发者模式。
  • Mac 的 iPhone 镜像:Agent 截取并操作 Mac 上的镜像窗口,手机无需另装控制程序。但 iPhone 一解锁,镜像就会暂停。
  • AirPlay + 外接键鼠:投屏负责让 Agent 看见画面,外接硬件负责发送输入。有项目用约 7 美元的 ESP32 开发板模拟键鼠。
  • iOS 27 的 CoreDevice:一些项目借用 Xcode 远程操作真机的开发者通道发送触摸和键盘输入。手机上无需另装 WebDriverAgent,仍要开启开发者模式并信任电脑。
    所以选型可以先看你的设备、能接受的授权、是否允许 Agent 占用主屏。还有一个值得单独检查的点:涉及付款、转账或删除时,执行端能不能停下来等你确认?

HotRepo 平时跟踪 GitHub 热门和新兴项目。这次整理了 31 个超过 100 星的手机 Agent 仓库,原文有各方案的设备条件、对比图和完整项目清单。

阅读全文

Dominant language
No language data
Stars
612
Forks
34
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from eryajf/learning-weekly

All issues in eryajf/learning-weekly

Similar issues

More Content issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.