Add an embeddable Rust SDK for in-process Flare inference

Start by inspecting the Helios crate, especially server/inference.rs, ModelRuntime, and NativeModel, then check the crate feature configuration and existing HTTP server adapter. Map which runtime and request-processing pieces can be exposed without the server or terminal features. Done means a documented embedding API covers loading, typed and streaming requests, cancellation, resource ownership, shutdown, and a reproducible Cargo dependency path.

Summary

Provide a documented Rust SDK that lets independent software vendors (ISVs) load a prepared Flare package and run inference directly inside their application, without starting an HTTP listener or a separate Flare server process.

Build this on Helios's existing native runtime so embedded applications and the HTTP server share model execution, scheduling, sampling, prefix caching, and package validation. The initial SDK can be explicitly experimental, consistent with Flare's research-prototype status.

Motivation

ISVs building local assistants, desktop applications, coding tools, and custom inference services may want to own the inference lifecycle inside their application. Today, HTTP endpoints provide a convenient integration path, but an embedded library would also allow applications to:

• Submit requests and receive results through typed Rust interfaces.
• Manage model loading, cancellation, resource limits, and shutdown alongside their application lifecycle.
• Use Flare behind their own service transport or user interface without an intermediate loopback HTTP service.
• Package a local inference capability without requiring end users to launch and configure a separate server.

The motivation is integration simplicity and lifecycle control. Any latency benefit relative to HTTP should be measured rather than assumed.

Existing foundation

Helios is already a Rust library linked by the Flare CLI. At commit 5f37bd14062334f7687d168d58ea00567b9aa7fe:

• NativeModel loads prepared packages and exposes execution manifests.
• ModelRuntime can start independently of Server and exposes generate/generate_many, bounded generation-event channels, health checks, and shutdown.
• Frontend functionality, typed errors, tokenization, and output parsing already exist.
• Protocol-neutral request types exist internally in server/inference.rs.

The missing piece is a supported embedding surface that assembles these components and owns their lifecycle. ModelRuntime::start and NativeModel::load currently expose unsafe contracts, request preparation requires several internal components, and inference is gated behind the server feature. The current flare-helios crate also has publish = false.

Proposed scope

1. An application-facing engine and client handle

Expose an Engine/Model-style API, either through a new SDK crate or a documented public module in flare-helios. Naming is open for discussion.

The API should:

• Load an existing local prepared package with explicit device selection.
• Accept supported memory, concurrency, and cache settings within the package's prepared limits.
• Report model identity, effective context capacity, and available features.
• Own package files, CUDA resources, rank workers, and request lifetimes so ordinary inference calls do not require raw pointers, manifests, or unsafe blocks in application code.
• Provide shareable request-submission handles with documented Send/Sync behavior and explicit shutdown semantics.

Package verification, publisher trust, and any required package-terms acceptance should be reusable library operations with application-supplied policy. Loading must not prompt on a terminal. A Rust facade should document that packages contain trusted executable GPU code; signature verification does not establish arbitrary kernel memory safety.

2. Typed requests and results

Support text completion and chat requests with typed messages, generation limits, sampling overrides, and stop conditions. Reuse the package's tokenizer, chat template, sampling defaults, validation, and output parsers.

Provide both collected responses and incremental output. Stream events should distinguish text deltas, completion/finish reason, usage, and failures, with a documented extension path for reasoning, tool calls, images, structured output, and log probabilities where supported by the package.

Callers should not have to build an HTTP-protocol JSON body or decode raw model tokens to obtain an ordinary text response. Tool execution remains the responsibility of the embedding application.

3. Predictable request and resource lifecycle

• Explicit cancellation and documented cancellation when a request stream is dropped.
• Bounded admission and output queues, with clear overload and slow-consumer behavior.
• A reliable terminal outcome for every accepted request, including asynchronous admission failures.
• Explicit shutdown that stops admission, drains or cancels work according to a documented policy, and reports whether GPU execution has drained.
• Resource ownership that remains safe on partial startup failure, cancellation, worker failure, and failed drain. Drop must not free buffers that GPU work may still reference.
• Loading and blocking CUDA teardown must not unexpectedly block an application's asynchronous executor.

4. Separate inference from HTTP and terminal integration

Allow the embedded inference feature set to build without Axum, HTTP listeners, SSE encoding, or the terminal dashboard. Tokio synchronization may remain an implementation dependency; avoiding every asynchronous-runtime dependency is not a requirement.

Move or expose shared request preparation, output processing, and metrics at the inference boundary. Keep the existing HTTP server as an adapter over the same engine rather than duplicating serving behavior in the SDK.

Embedding should not install process signal handlers, exit the process, take over the terminal, or silently rewrite process-wide CUDA environment settings. Any initialization requirements that affect an existing CUDA application must be explicit.

5. Distribution and developer experience

• Provide an explicit SDK versioning and compatibility policy, initially allowing an experimental API.
• Document a reproducible Cargo dependency path. A tagged Git dependency is a reasonable first step; crates.io publication can follow when dependency and release requirements permit it.
• Document the supported GPU/driver/OS matrix, Rust toolchain, native build dependencies, package/runtime compatibility, and deployment requirements.
• Provide examples for a minimal text application, streaming chat, cancellation, and use inside an existing async application.
• Expose health, usage, and resource metrics directly, without requiring an HTTP endpoint.

Illustrative API

The following is API design pseudocode, not an existing or compiled Flare API. The application supplies publisher_policy, including any required package acceptance.

let package = TrustedPackage::open("model.safetensors", publisher_policy)?;

let engine = Engine::builder()
    .package(package)
    .devices([0])
    .load()
    .await?;

let request = ChatRequest::new([
    Message::user("Summarize the document below..."),
])
.max_output_tokens(256);

let mut generation = engine.generate(request).await?;

while let Some(event) = generation.next().await {
    match event? {
        GenerationEvent::TextDelta(text) => render_text(text),
        GenerationEvent::Finished(result) => record_usage(result.usage),
        _ => {}
    }
}

drop(generation);
engine.shutdown().await?;