[RFC] Make the alerting core reusable across hosted and Maple Local deployments

Read the existing AlertsService in apps/api and the scheduler host in apps/alerting, focusing on runSchedulerTick() and the persistence and query boundaries described in the issue. Compare those boundaries with the apps/cli entry point and Local server. Done means an upstream architectural decision on whether the alert core can be shared across hosted and Local deployments; implementation scope is not yet defined.

Maple’s alerting work appears to separate the Cloudflare-based scheduler host from substantial alert evaluation, lifecycle, and delivery logic in the API.
I am running Maple Local and would like that core alerting capability to work across deployment targets. The scheduler, persistence, and telemetry-query adapters could remain deployment-specific, while rule evaluation, incident lifecycle, retries, deduplication, and delivery behavior remain shared.
Would a host-neutral alerting core, with hosted and Local adapters, align with the project’s intended architecture?

To help be concrete about what path looks best I had my agent write up this nice visual presentation. (here, we are focused only on the first priority I had given it)

Maple Local: Architecture, Direction, and the Alerting Path

Format: Architectural briefing / decision aid
Deployment: srvmini2 Maple Local
Priority order: alerts and event delivery; overview and charts; traffic map
Immediate action: propose a host-neutral Local alerting design upstream

This document is intentionally diagram-heavy. It separates the Maple monorepo,
the Maple Local product, the full hosted application, and this deployment's
fork-specific operational work so that architectural decisions are made at the
right layer.

---

1 — The forest

flowchart LR
    A["Proven telemetry substrate<br/>Collector + Maple Local + chDB"]
    B["Priority 1<br/>Headless alerts and event delivery"]
    C["Priority 2<br/>Operational overview and charts"]
    D["Priority 3<br/>Graphical traffic map"]

    A --> B --> C --> D

The central architectural choice is not whether alerts belong in a browser.
They do not. The choice is whether Maple Local should host the existing alert
engine, while the Local UI becomes one control and presentation surface for it.

flowchart TB
    Goal["Desired outcome<br/>Useful local observability, not full-web parity"]
    Runtime["Always-running local runtime<br/>evaluate, persist, deliver"]
    UI["Optional browser UI<br/>configure, inspect, acknowledge"]
    Subscribers["Variable subscribers<br/>webhook first, other sinks optional"]

    Goal --> Runtime
    Runtime --> Subscribers
    UI -->|"control and read state"| Runtime
    Runtime -.->|"must work with browser closed"| UI

---

2 — One repository, several products

This is the canonical architecture diagram for the proposal. Solid arrows show
what exists today. Dashed arrows show the one proposed change: make the alert
engine reusable, then host it from Maple Local. The diagram intentionally stops
at the product boundary; section 8 shows how the proposed Local path runs in our
deployment.

flowchart TB
    Repo["Maple monorepo"]

    subgraph LocalProduct["Maple Local today"]
        CLI["apps/cli<br/>CLI + OTLP server + chDB"]
        LocalUI["apps/local-ui<br/>Traces, Logs, Sessions"]
        LocalBundle["Two-file bundle<br/>maple + libchdb"]

        subgraph ProposedLocal["Proposed additions in Maple Local (2 components)"]
            LocalAdapter["2. apps/cli alert adapter<br/>scheduler + routes + state/query wiring"]
            LocalAlertUI["3. Alert views<br/>inside apps/local-ui"]
        end

        CLI --> LocalBundle
        LocalUI -->|"embedded assets"| LocalBundle
    end

    subgraph Shared["Shared libraries"]
        Query["packages/query-engine"]
        UI["packages/ui"]
        Domain["packages/domain"]

        subgraph ProposedShared["Proposed addition in Shared libraries (1 component)"]
            AlertCore["1. Host-neutral alert core<br/>evaluation + lifecycle + delivery"]
        end
    end

    subgraph FullProduct["Full / hosted Maple today"]
        Web["apps/web"]
        API["apps/api"]
        AlertImpl["AlertsService<br/>currently owned by apps/api"]
        AlertWorker["apps/alerting<br/>Cloudflare scheduler host"]
        Ingest["apps/ingest"]
        Platform["PostgreSQL + warehouse + Cloudflare services"]

        Web --> API
        API --> AlertImpl
        AlertWorker --> AlertImpl
        AlertImpl --> Platform
        API --> Platform
        Ingest --> Platform
    end

    LocalBundle ~~~ Query ~~~ Platform

    CLI -->|"uses"| Query
    CLI -->|"uses"| Domain
    LocalUI -->|"uses"| UI
    API -->|"uses"| Query
    API -->|"uses"| Domain
    Web -->|"uses"| UI

    AlertImpl -. "extract reusable logic" .-> AlertCore
    AlertWorker -. "hosted scheduler continues to call" .-> AlertCore
    API -. "hosted adapter uses" .-> AlertCore
    CLI -. "imports" .-> LocalAdapter
    LocalAdapter -. "calls" .-> AlertCore
    LocalUI -. "adds" .-> LocalAlertUI
    LocalAlertUI -. "same-origin alert routes" .-> LocalAdapter

    Repo --> LocalProduct
    Repo --> Shared
    Repo --> FullProduct

What this says: alerting is present in the monorepo but absent from the Local
product because no current dependency path reaches it. Moving our fork to
current main, or waiting for PR #194, does not change that. The proposal adds
the dashed path; it does not put apps/api, apps/web, PostgreSQL, or the
Cloudflare worker into the Local bundle. The existing apps/cli/src/bin.ts
entry point and two-file distribution can remain.

---

3 — What is actually deployed today

flowchart LR
    subgraph LAN["Trusted homelab LAN"]
        Producers["OTel producers"]
        Browser["Browser"]
        Operator["Operator CLI"]
    end

    subgraph Container["Apple Container: maple"]
        Collector["OTel Collector<br/>0.0.0.0:4318<br/>Basic Auth"]
        Maple["Maple Local<br/>0.0.0.0:4418<br/>unauthenticated"]
        Hot["Embedded chDB<br/>hot telemetry"]
        Queue["Collector persistent queue"]
        Checkpoints["Validated checkpoints"]
        Scratch["Bounded scratch"]
    end

    Archive["External-backed archive<br/>immutable Parquet generations"]

    Producers -->|"OTLP/HTTP + Basic Auth"| Collector
    Collector --> Queue
    Collector -->|"OTLP/HTTP on loopback"| Maple
    Maple --> Hot
    Browser -->|"UI + /local/query"| Maple
    Operator -->|"checkpoint/query on 4418"| Maple
    Hot --> Checkpoints
    Checkpoints --> Scratch
    Scratch --> Archive

flowchart TB
    PublicBoundary["Port 4318 boundary<br/>authenticated Collector ingest"]
    LocalBoundary["Port 4418 boundary<br/>unauthenticated UI, raw query, direct OTLP"]
    Trust["Current safety assumption<br/>trusted LAN only"]

    PublicBoundary --> Trust
    LocalBoundary --> Trust

The telemetry substrate is already the strongest part of the system. Alerting
should reuse it without weakening checkpoint, archive, retention, or rollback
guarantees. This section describes only the deployment that exists today; the
runtime form of the section 2 proposal appears once, in section 8.

---

4 — Why alerts disappear from the Local build

flowchart LR
    Script["scripts/build-local-binary.sh"]
    BuildUI["Build apps/local-ui"]
    Embed["Embed UI assets"]
    Compile["bun build --compile<br/>apps/cli/src/bin.ts"]
    Imports["Follow CLI import graph"]
    Output["maple executable"]

    Script --> BuildUI --> Embed --> Compile --> Imports --> Output

    API["apps/api AlertsService"]
    Worker["apps/alerting Cloudflare worker"]
    WebAlerts["apps/web alert screens"]

    API -. "outside import graph" .-> Imports
    Worker -. "outside import graph" .-> Imports
    WebAlerts -. "different SPA" .-> BuildUI

The current Local HTTP surface is similarly narrow:

flowchart TB
    LocalServer["Maple Local HTTP server"]
    Health["GET /health"]
    Traces["POST /v1/traces"]
    Logs["POST /v1/logs"]
    Metrics["POST /v1/metrics"]
    Query["POST /local/query"]
    Assets["GET embedded Local UI assets"]
    Missing["No alert CRUD, incident, delivery, or scheduler endpoints"]

    LocalServer --> Health
    LocalServer --> Traces
    LocalServer --> Logs
    LocalServer --> Metrics
    LocalServer --> Query
    LocalServer --> Assets
    Missing -. "not currently routed" .-> LocalServer

PR #194 changes checkpoint/archive behavior in apps/cli; it does not import or
host apps/api, apps/alerting, or the full alert UI.

---

5 — How full Maple alerting works today

sequenceDiagram
    participant Cron as Cloudflare Cron
    participant Worker as apps/alerting worker
    participant Alerts as AlertsService
    participant DB as PostgreSQL alert state
    participant Warehouse as Telemetry warehouse
    participant Delivery as Delivery dispatcher
    participant Subscriber as Destination / subscriber

    Cron->>Worker: scheduled event
    Worker->>Alerts: runSchedulerTick()
    Alerts->>DB: load enabled rules and claim work
    Alerts->>Warehouse: evaluate time-window queries
    Warehouse-->>Alerts: observations
    Alerts->>DB: update checks, states, incidents, delivery queue
    Alerts->>Delivery: process due deliveries
    Delivery->>Subscriber: signed webhook / Slack / PagerDuty / email / Discord
    Subscriber-->>Delivery: delivery result
    Delivery->>DB: persist success, failure, or retry

The Cloudflare worker is primarily a host and scheduler adapter. The alert
engine already exposes runSchedulerTick(). This is why a Local adaptation is
plausible without reimplementing alert semantics.

---

6 — Reusable engine versus missing Local integration

flowchart LR
    subgraph Exists["Substantial code already exists"]
        Rules["Rule validation and CRUD"]
        Eval["Query-based evaluation"]
        State["Checks, incidents, retries, deduplication"]
        Lifecycle["trigger / resolve / renotify / test"]
        Delivery["Multiple destinations + signed generic webhook"]
        Tick["Host-callable runSchedulerTick()"]
    end

    subgraph Missing["Local integration still required"]
        Host["Local scheduler host"]
        LocalState["Durable local alert-state adapter"]
        LocalQuery["Local chDB warehouse adapter wiring"]
        LocalAPI["Bounded Local alert API"]
        LocalAlertUI["Local configuration and incident UI"]
        Policy["Local/LAN outbound URL policy"]
        Backup["Alert-state backup and recovery contract"]
    end

    Exists -->|"extract and adapt"| Missing

This is neither a hidden feature flag nor a greenfield system. It is an
existing engine with a missing Local host boundary.

---

7 — The architectural decision

The proposal is exactly the dashed path in section 2. The upstream question is
whether the alert engine should become shared product code that Maple Local can
host, while the hosted deployment continues to use its own scheduler and
persistence adapters.

If upstream agrees, section 8 shows the resulting runtime on srvmini2. If
upstream prefers alerting to remain owned by the hosted API, the honest
alternative is a separate consumer service; it should not be disguised as the
same architecture.

Preferred working hypothesis

Host the optional scheduler as a supervised fiber inside maple start, because
the existing process already owns the chDB connection and Local server
lifecycle. Keep a companion process as an upstream design alternative if
failure containment proves more important. That would be a different deployment
design, not an extra arrow added to sections 2 or 3.

---

8 — Runtime view of section 2's proposed Local path

This is the same architecture as the dashed path in section 2, viewed after it
has been compiled into and deployed as Maple Local. It is not a second proposal.

flowchart LR
    Collector["Collector :4318<br/>authenticated ingest"]

    subgraph MapleProcess["Maple Local process :4418"]
        Ingest["OTLP ingest"]
        Query["Local query engine"]
        AlertCore["Section 2: shared alert core"]
        LocalAdapter["Section 2: apps/cli Local alert adapter<br/>scheduler + routes + state/query wiring"]
        LocalAlertUI["Section 2: alert views<br/>inside embedded apps/local-ui"]
    end

    Hot["chDB telemetry store"]
    AlertState["Durable alert state<br/>rules, checks, incidents, deliveries"]
    Webhook["Signed webhook consumer"]
    Browser["Homelab browser"]

    Collector --> Ingest --> Hot
    LocalAdapter --> AlertCore
    AlertCore --> Query --> Hot
    AlertCore --> AlertState
    AlertCore --> Webhook
    Browser --> LocalAlertUI --> LocalAdapter

This preserves the existing container, ports, Collector authentication, chDB,
checkpoints, archives, retention, and reconciler. The new durability obligation
is alert state: rules, open incidents, and pending deliveries must survive
restart and recovery with explicit backup, restore, and migration semantics.

---

9 — Subscriber and delivery model

“Roll your own subscriber” is already partly represented by Maple's signed
generic webhook. The useful abstraction is a destination/event sink, not an
arbitrary browser plugin.

flowchart LR
    Engine["Alert lifecycle event"]

    Trigger["trigger"]
    Resolve["resolve"]
    Renotify["renotify"]
    Test["test"]

    Sink["AlertEventSink / destination dispatcher"]

    PublicHook["Public signed webhook"]
    LANHook["Private/LAN webhook<br/>explicit Local opt-in"]
    BuiltIns["Slack / PagerDuty / Discord / email"]
    Consumer["Operator-owned consumer<br/>automation, paging, ticketing"]

    Engine --> Trigger --> Sink
    Engine --> Resolve --> Sink
    Engine --> Renotify --> Sink
    Engine --> Test --> Sink

    Sink --> PublicHook
    Sink --> LANHook
    Sink --> BuiltIns
    PublicHook --> Consumer
    LANHook --> Consumer

flowchart TB
    HostedPolicy["Hosted policy<br/>block loopback and private targets"]
    LocalPolicy["Local policy<br/>same safe default, explicit allowlist for trusted LAN"]
    SharedDelivery["Shared delivery implementation"]

    HostedPolicy --> SharedDelivery
    LocalPolicy --> SharedDelivery

The hosted SSRF policy should remain strict. Local mode can add an explicit,
auditable private-network allowlist without weakening the hosted product.

---

10 — What “eventing off OTel events” can mean

Alert lifecycle events and raw OpenTelemetry records are different layers.

flowchart LR
    SDK["Application instrumentation"]
    OTLP["OTLP records<br/>traces, logs, metrics"]
    Store["Telemetry store"]
    Evaluate["Windowed rule evaluation"]
    DomainEvent["Alert domain event<br/>trigger / resolve / renotify"]
    Subscriber["Subscriber"]

    SDK --> OTLP --> Store --> Evaluate --> DomainEvent --> Subscriber

flowchart TB
    Need["What behavior is needed?"]
    Window["Aggregate over time<br/>latency, rate, count, absence"]
    Immediate["Match one incoming record immediately"]
    Alerts["Query-based alert engine"]
    Stream["Future ingest/event-routing path"]

    Need --> Window --> Alerts
    Need --> Immediate --> Stream

The first upstream proposal should focus on the query-based alert engine because
that code already exists and covers the highest-value operational cases. A
general raw-event bus is a separate follow-up and may overlap with Collector
routing/export functionality.

---

11 — UI responsibility: control plane, not execution engine

flowchart TB
    subgraph Browser["Local UI — may be closed"]
        RuleEditor["Create and edit rules"]
        IncidentView["View incidents and checks"]
        DestinationEditor["Configure destinations"]
        Status["Show scheduler and delivery health"]
    end

    subgraph Runtime["Maple Local — always running"]
        API["Bounded alert API"]
        Scheduler["Scheduler"]
        Evaluator["Evaluator"]
        Persistence["Durable state"]
        Dispatcher["Delivery dispatcher"]
    end

    RuleEditor --> API
    IncidentView --> API
    DestinationEditor --> API
    Status --> API

    API --> Persistence
    Scheduler --> Evaluator --> Persistence
    Persistence --> Dispatcher

flowchart LR
    Closed["Browser closed"] --> Running["Alerts still evaluate and deliver"]
    Open["Browser open"] --> Configure["User can configure and inspect"]

This preserves the user's highest-priority capability independently of the
lower-priority graphical work.

---

12 — Priority-aligned roadmap

flowchart LR
    P0["P0 — Upstream design issue<br/>agree on Local ownership and seams"]
    P1["P1 — Headless Local alerts<br/>scheduler + state + signed webhook"]
    P2["P2 — Alert control surface<br/>CLI/API first, Local UI second"]
    P3["P3 — Overview and charts"]
    P4["P4 — Traffic map"]
    P5["P5 — Optional broader event sinks<br/>only after real use cases"]

    P0 --> P1 --> P2 --> P3 --> P4 --> P5

flowchart TB
    UserGoals["Expressed goals in priority order"]
    G1["1. Eventing and alerts"]
    G2["2. Overview and charts"]
    G3["3. Traffic map"]

    Architecture["Architecture response"]
    R1["Headless runtime first<br/>UI-independent delivery"]
    R2["Small read-only operational UI<br/>not full-web parity"]
    R3["Bounded visualization over existing service_map_spans"]

    UserGoals --> G1 --> R1
    UserGoals --> G2 --> R2
    UserGoals --> G3 --> R3
    R1 --> Architecture
    R2 --> Architecture
    R3 --> Architecture

The existing Maple Local UI roadmap already defers
overview/charts and the traffic map until the alerting/eventing architecture is
decided. This sequence makes that dependency explicit.

---

13 — Why the upstream issue comes before implementation

flowchart TB
    Issue["Upstream proposal<br/>Host-neutral alerts for Maple Local"]

    Ownership["Does upstream want Local to own alert evaluation?"]
    Host["In-process scheduler or companion process?"]
    Store["Which local state store and migration model?"]
    Durability["How are alert state and pending deliveries backed up?"]
    Sinks["Is signed webhook sufficient as the extension boundary?"]
    Policy["How should Local private-network destinations be authorized?"]
    UIReuse["Which full-web alert components can be shared safely?"]

    Issue --> Ownership
    Issue --> Host
    Issue --> Store
    Issue --> Durability
    Issue --> Sinks
    Issue --> Policy
    Issue --> UIReuse

Proposed issue scope

flowchart LR
    subgraph InScope["In scope"]
        A["Host-neutral alert runtime"]
        B["Local scheduler adapter"]
        C["Local durable state"]
        D["Local query adapter"]
        E["Signed webhook baseline"]
        F["Bounded API and optional UI"]
    end

    subgraph OutOfScope["Not required for the first change"]
        G["Full web-app parity"]
        H["Raw OTLP event bus"]
        I["Arbitrary runtime plugins"]
        J["Every hosted notification provider"]
        K["Overview and traffic-map implementation"]
    end

The issue should invite maintainer direction before choosing the local state
store or permanently coupling the engine to the CLI process.

---

14 — Source and ownership reality

flowchart LR
    Upstream["Upstream Maple Local"]
    Merged["Checkpoint foundation<br/>PR #129 merged"]
    Review["Archive system<br/>PR #194 under review"]
    Proposed["Configurable Local listener<br/>PR #219 proposed"]
    Fork["Deployment fork"]
    LocalWork["Bounded retention<br/>LAN publication<br/>operational proof"]
    Next["Next upstream proposal<br/>Local alert runtime"]

    Upstream --> Merged
    Upstream --> Review
    Upstream --> Proposed
    Fork --> LocalWork
    Merged --> Fork
    Review --> Fork
    Proposed --> Fork
    Fork --> Next
    Next -. "community feedback shapes implementation" .-> Upstream

Maple Local is maintained, but the significant Local operational direction has
largely been driven by this deployment's work. The alerting issue should be
treated as a proposal for upstream product direction, not as a request to reveal
a feature that is already bundled.

---

15 — North-star architecture

flowchart LR
    subgraph Sources["Telemetry sources"]
        Apps["Applications"]
        Hosts["Hosts and infrastructure"]
    end

    subgraph Intake["Authenticated intake"]
        Collector["OTel Collector<br/>queue + policy"]
    end

    subgraph MapleLocal["Maple Local"]
        Ingest["OTLP ingest"]
        Hot["chDB hot telemetry"]
        Query["Shared query engine"]
        AlertEngine["Optional headless alert engine"]
        AlertState["Durable alert state"]
        LocalAPI["Bounded local APIs"]
        UI["Local UI<br/>alerts, overview, charts, traffic map"]
    end

    subgraph Durability["Operational durability"]
        Checkpoints["Checkpoints"]
        Archives["Parquet archives"]
        AlertBackup["Alert-state backup"]
    end

    subgraph Consumers["Actions and people"]
        Hook["Signed webhook"]
        Custom["Operator-owned subscriber"]
        Human["Homelab operator"]
    end

    Apps --> Collector
    Hosts --> Collector
    Collector --> Ingest --> Hot
    Hot --> Query
    Query --> AlertEngine
    AlertEngine --> AlertState
    AlertEngine --> Hook --> Custom
    LocalAPI --> AlertEngine
    UI --> LocalAPI
    Human --> UI

    Hot --> Checkpoints --> Archives
    AlertState --> AlertBackup

North-star sentence

Keep Maple Local small and operable, add headless alert intelligence as an
optional host-neutral capability, use the UI as a control and visualization
surface, and add only the overview and traffic views that produce real local
value.

---

Appendix — Current evidence and related documents

• Maple Local UI roadmap
• Deployment orientation
• Checkpoint and archive plan
• PR #129 — Local chDB checkpoints
• PR #194 — Local telemetry archives
• PR #219 — Configurable Local bind host
• Current Local build script
• Current alert worker
• Current alert service
• Current delivery dispatcher