feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel

Offen
  • Stimmt etwas nicht?
#1,629 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@Rasaboun arbeitet bereits daran.

Seit 10.11.2026.

  • #1630 von @Rasaboun — offen

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
1/100
Issue-Typ
Feature
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
pytorch, react-native, typescript
Bereich
ai, mobile

Rechercherichtung

The issue identifies src/core/model.ts and the LLM runner, task, hook, and model loading APIs as the main entry points. Start by reviewing the linked PR #1630 and its tests to see the implementation scope and expected behavior. Done means the proposed model sources and external weight loading work through the core and LLM APIs, with the cited tests and documentation updates.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

core feature request
Problem description

loadModel(modelPath, options) and the LLM task APIs (createLLMRunner, createLLMChatSession, createLLMTranslator, useLLMChatSession) currently require a plain local .pte file path on disk and do not expose ExecuTorch's data_map_loader for separated .ptd weight archives.

This blocks three common production use cases:

  1. In-memory models (ArrayBuffer): loading small .pte programs directly from memory without writing a temporary file to disk.
  2. Separated .ptd external weight archives: ExecuTorch supports exporting a small .pte program alongside an external .ptd named-data archive (Module(data_loader, ..., data_map_loader)), where backend delegates such as XNNPACK unpack named weight segments during method initialization and immediately free the source buffers.
  3. Chunked / on-demand loading (DataLoader): apps that store models in custom containers or encrypted chunks cannot materialize a 200 MB–2 GB model into a temporary plaintext file or a single full-file ArrayBuffer without doubling peak/steady-state resident memory. They need a random-access loader (size() + load(offset, size)) so ExecuTorch only requests the byte ranges it needs and frees each FreeableBuffer segment as soon as the backend unpacks it.
Proposed solution

Extend the core loadModel primitive and thread it through the LLM runner/task layer while staying 100% backward-compatible with existing path-based callers:

  1. Add ModelDataLoader and ModelSource in src/core/model.ts: 
    export type ModelDataLoader = {
  size(): number;
  load(offset: number, size: number): ArrayBuffer;
};

export type ModelSource = string | ArrayBuffer | ModelDataLoader;
  2. Extend LoadModelOptions and loadModel: 
    export type LoadModelOptions = {
  readonly eagerLoadMethods?: boolean;
  readonly dataMapPath?: ModelSource;
};

export function loadModel(modelSource: ModelSource, options?: LoadModelOptions): Model;
    • string continues to use MmapDataLoader (NoMlock).
    • ArrayBuffer uses a 64-byte-aligned owning BufferDataLoader (model.path === '<buffer>').
    • ModelDataLoader uses a JSI-backed DataLoader (model.path === '<loader>') that returns 64-byte-aligned FreeableBuffers with a deallocation callback so ExecuTorch frees each segment once unpacked, and detaches its JS function/runtime handles before loadModel() returns so no jsi::Runtime* or JS closure outlives the load call.
  3. Thread ModelSource and dataMapPath through LLM APIs & hooks:
    • createLLMRunner(modelPath: ModelSource, tokenizerPath: string, modalities?, runtime?, options?: LoadModelOptions)
    • LLMModel<TSource extends ModelSource = string> (defaulting to string so models.llm.* registry entries remain typed as string URLs, while createLLMChatSession, createLLMTranslator, and useLLMChatSession accept ModelSource and dataMapPath).
    • Preserve ArrayBuffer and ModelDataLoader reference identity in useModel and useResourceDownload config memoization, and skip ModelDataLoader objects in download() traversal.
Alternative solutions
  • Writing temporary .pte / .ptd files to disk before calling loadModel(path): undesirable for encrypted/protected model assets (writes plaintext weights to flash storage) and adds unnecessary disk I/O and storage pressure.
  • Passing only whole ArrayBuffers: works for small .pte programs, but for large models it keeps a full copy of the raw weights in memory alongside the backend's packed weights (~170+ MB extra RSS on a 180 MB model). Supporting both ArrayBuffer and ModelDataLoader plus .ptd external weights lets XNNPACK free each weight segment right after packing.
Benefits to React Native ExecuTorch
  • Unlocks separated .ptd weight archives (data_map_loader) across loadModel and LLM tasks.
  • Enables zero-temp-file in-memory loading (ArrayBuffer) and low-memory streaming/chunked loading (ModelDataLoader) for custom asset containers or encrypted models.
  • Lives cleanly in core/model with no breaking changes to existing APIs or models registry types.
Additional context

I have opened PR #1630 implementing this with host C++ GoogleTest (ModelTest), Jest API tests, and documentation updates.

Vorherrschende Sprache
TypeScript
Sterne
1.8k
Forks
101
Ø Merge
1 T. 10 Std.
Gemergte PRs (30 T.)
51

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus software-mansion/react-native-executorch

Alle Issues in software-mansion/react-native-executorch

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.