feat: manage local LLM / image-gen servers (Ollama, llama.cpp, vLLM, MLX, ComfyUI)

Open
  • Is something wrong?
#1,622 1 comment 2 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
flutter, ollama, rust
Domain
ai, api, backend, devops

Research direction

Start with the planned Rust crate, discovery layer, and Ollama support, then trace the app's ServerFuncBtn entry and the monitor agent's /llm/* endpoints. Review how the shared layer is used by the app and web panel; done means the phased backend management flow, model handling, and panel integration described in the plan are implemented.

Written by the indexing model from the issue text.

Description

enhancement feature

Manage inference servers running on a host (or this device via Spi.local), including their models.

Approach

  • Shared Rust layer, used by both the app (FFI) and the web panel (monitor agent):
    • sans-IO clients per backend: build requests, parse responses into one normalized model (endpoint, models, loaded/running, VRAM, pull progress).
    • discovery: fingerprint known ports (GET / Ollama, /props llama.cpp, /version vLLM, /system_stats ComfyUI; MLX by process cmdline).
    • HF cache scan/delete (shared by vLLM, MLX, llama.cpp -hf).
  • Transport
    • App over SSH: ServerTcpDialer to 127.0.0.1:<port>, no port exposed; Rust parses.
    • Monitor agent: new /llm/* endpoints calling the backends locally; web panel and monitor-only servers use these.
  • Lifecycle (start/stop/restart) reuses existing systemd / container / launchd management.

Backends

Backend Model management Notes
Ollama full API: tags, ps, pull (NDJSON progress), delete, show, unload
llama.cpp router mode: /models, /models/load, /models/unload, /models/sse, download/delete (recent builds) single-model mode read-only
ComfyUI /system_stats, /queue, /interrupt, /free, /api/experiment/models/{folder}; files via exec no core download API
MLX (mlx_lm.server) HF cache macOS only
vLLM /health, /metrics, /v1/models, HF cache, optional LoRA one model per process; switching = restart with new args

Plan

  1. Rust crate + discovery + Ollama; monitor /llm/*; app ServerFuncBtn entry + page.
  2. llama.cpp + shared HF cache module.
  3. ComfyUI + MLX.
  4. vLLM.
  5. Web panel views.

Open questions

  • Manual endpoints and API keys (--api-key on llama-server / vLLM) need a new table; v1 could be discovery-only.
  • Editing launch config to switch models (vLLM, single-mode llama.cpp): out of scope for v1.
  • Optional: add a discovered endpoint as an Agent (fl_pi_llm) provider.
Dominant language
Dart
Stars
8.9k
Forks
573
Avg merge
2h 11m
Merged PRs (30d)
134

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lollipopkit/flutter_server_box

All issues in lollipopkit/flutter_server_box

Similar issues

More Dart issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.