feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel

未关闭
  • 哪里不对？
#1,629 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@Rasaboun 已经在做这个了。

开始于 2026年11月10日。

  • #1630 来自 @Rasaboun —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
1/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
停滞
技术栈
pytorch, react-native, typescript
领域
ai, mobile

调研方向

The issue identifies src/core/model.ts and the LLM runner, task, hook, and model loading APIs as the main entry points. Start by reviewing the linked PR #1630 and its tests to see the implementation scope and expected behavior. Done means the proposed model sources and external weight loading work through the core and LLM APIs, with the cited tests and documentation updates.

由索引模型根据 Issue 内容生成。

描述

core feature request
Problem description

loadModel(modelPath, options) and the LLM task APIs (createLLMRunner, createLLMChatSession, createLLMTranslator, useLLMChatSession) currently require a plain local .pte file path on disk and do not expose ExecuTorch's data_map_loader for separated .ptd weight archives.

This blocks three common production use cases:

  1. In-memory models (ArrayBuffer): loading small .pte programs directly from memory without writing a temporary file to disk.
  2. Separated .ptd external weight archives: ExecuTorch supports exporting a small .pte program alongside an external .ptd named-data archive (Module(data_loader, ..., data_map_loader)), where backend delegates such as XNNPACK unpack named weight segments during method initialization and immediately free the source buffers.
  3. Chunked / on-demand loading (DataLoader): apps that store models in custom containers or encrypted chunks cannot materialize a 200 MB–2 GB model into a temporary plaintext file or a single full-file ArrayBuffer without doubling peak/steady-state resident memory. They need a random-access loader (size() + load(offset, size)) so ExecuTorch only requests the byte ranges it needs and frees each FreeableBuffer segment as soon as the backend unpacks it.
Proposed solution

Extend the core loadModel primitive and thread it through the LLM runner/task layer while staying 100% backward-compatible with existing path-based callers:

  1. Add ModelDataLoader and ModelSource in src/core/model.ts: 
    export type ModelDataLoader = {
  size(): number;
  load(offset: number, size: number): ArrayBuffer;
};

export type ModelSource = string | ArrayBuffer | ModelDataLoader;
  2. Extend LoadModelOptions and loadModel: 
    export type LoadModelOptions = {
  readonly eagerLoadMethods?: boolean;
  readonly dataMapPath?: ModelSource;
};

export function loadModel(modelSource: ModelSource, options?: LoadModelOptions): Model;
    • string continues to use MmapDataLoader (NoMlock).
    • ArrayBuffer uses a 64-byte-aligned owning BufferDataLoader (model.path === '<buffer>').
    • ModelDataLoader uses a JSI-backed DataLoader (model.path === '<loader>') that returns 64-byte-aligned FreeableBuffers with a deallocation callback so ExecuTorch frees each segment once unpacked, and detaches its JS function/runtime handles before loadModel() returns so no jsi::Runtime* or JS closure outlives the load call.
  3. Thread ModelSource and dataMapPath through LLM APIs & hooks:
    • createLLMRunner(modelPath: ModelSource, tokenizerPath: string, modalities?, runtime?, options?: LoadModelOptions)
    • LLMModel<TSource extends ModelSource = string> (defaulting to string so models.llm.* registry entries remain typed as string URLs, while createLLMChatSession, createLLMTranslator, and useLLMChatSession accept ModelSource and dataMapPath).
    • Preserve ArrayBuffer and ModelDataLoader reference identity in useModel and useResourceDownload config memoization, and skip ModelDataLoader objects in download() traversal.
Alternative solutions
  • Writing temporary .pte / .ptd files to disk before calling loadModel(path): undesirable for encrypted/protected model assets (writes plaintext weights to flash storage) and adds unnecessary disk I/O and storage pressure.
  • Passing only whole ArrayBuffers: works for small .pte programs, but for large models it keeps a full copy of the raw weights in memory alongside the backend's packed weights (~170+ MB extra RSS on a 180 MB model). Supporting both ArrayBuffer and ModelDataLoader plus .ptd external weights lets XNNPACK free each weight segment right after packing.
Benefits to React Native ExecuTorch
  • Unlocks separated .ptd weight archives (data_map_loader) across loadModel and LLM tasks.
  • Enables zero-temp-file in-memory loading (ArrayBuffer) and low-memory streaming/chunked loading (ModelDataLoader) for custom asset containers or encrypted models.
  • Lives cleanly in core/model with no breaking changes to existing APIs or models registry types.
Additional context

I have opened PR #1630 implementing this with host C++ GoogleTest (ModelTest), Jest API tests, and documentation updates.

主要语言
TypeScript
星标
1.8k
派生
101
平均合并
1 天 10 小时
30 天内合并 PR
51

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

software-mansion/react-native-executorch 的其他 Issue

查看 software-mansion/react-native-executorch 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。