[Feature]: Rented GPU nodes: serve a Hugging Face Inference Endpoint to the cluster like a local node

The issue names no implementation files or tests. Start by reading the existing external endpoint work in #24 and #64, then inspect the router, scheduler, node inventory, credential handling, and workload history entry points. Break the proposal into the three described slices; done means rented endpoints can be securely attached and shared, managed through the Hugging Face adapter, and routed with the stated policies and cost controls.

Area

Engine or model management (also touches Routing and scheduling)

User problem

I run PAIR on two machines: an M4 Max MacBook Pro (64 GB) and a PC with an RTX 3090 (24 GB). Most of my work fits on them. Some doesn't:

• a model larger than the 3090's 24 GB, or too large to run at a useful speed on the Mac;
• a context the 3090 can't hold (about 3% of the records in my batch workload);
• a batch that would take all night on local hardware.

Hugging Face rents GPUs for exactly these cases, billed by the minute: an L40S (48 GB) at $1.80/hr, an A100 (80 GB) at $2.50/hr, an H200 (141 GB) at $5.00/hr. They're available through Inference Endpoints or Jobs, and both serve an OpenAI-compatible API from vLLM, SGLang, llama.cpp or TGI. A two-hour batch on an L40S costs $3.60.

PAIR only routes to machines on the LAN that run PAIR. So today a rented GPU has to be a second, separate endpoint:

1. Every client has to be pointed at it. Apps use 127.0.0.1:11434 because that's where PAIR is. A model served only by the rented GPU isn't in PAIR's inventory, so a request for it through PAIR fails. The client has to switch base URL and token for that one model.
2. It can't be adopted, even with PR #64. #64 (adopt OpenAI-compatible endpoints by base URL, still open) accepts only http URLs and has no credential field. A rented endpoint is HTTPS behind a bearer token. #2 and #26 ask for credentials, but only for local engines.
3. It can't be shared. For another machine in the cluster to use the same rented GPU, that machine needs its own copy of the URL and the token. Trust between paired nodes doesn't extend to the rented GPU.
4. Nothing turns it off. A rented GPU bills while it runs, used or not. PAIR can't start it when a request needs it, pause it when it's idle, or stop at a budget. Cost control is manual and easy to forget.
5. It's invisible. Work the rented GPU serves isn't in PAIR's Jobs list or workload history, and nothing tells the user that a prompt left the LAN.

I found no existing issue for this. In PAIR's tracker, Hugging Face appears only as a source of model weights (#9, #50, #117, #125, #148) and in #24's note on model aliases. No issue or PR mentions Inference Endpoints, Jobs, Spaces, Inference Providers, or rented GPUs from any provider.

Desired outcome

A rented node: a GPU the user rents from Hugging Face, added to PAIR by one paired node (its sponsor), and served to the cluster as if it were a member.

"As if it were a member" means:

• It appears in the node list on every paired node, labeled Rented · off-LAN, with its GPU, provider, state and hourly price.
• Its model appears in the merged inventory. A request for that model to 127.0.0.1:11434 on any node is routed to the rented node and streamed back. It shows in the Jobs list and workload history like any other workload, with the rented node as the serving node.
• Failover works both ways. If the rented node is down, a local node that has the model takes the request, and the reverse (as the routing policy below allows).
• The rented machine runs no PAIR software. The sponsor holds the credential, probes health and inventory, and forwards requests over HTTPS. Peers reach the rented node only through the sponsor, over the cluster's existing mTLS, and never see its URL or token.

I'd suggest three slices. Each is useful on its own.

1. Attach an existing rented endpoint

Extend the external-endpoint contract from #24/#64 with what a rented endpoint needs:

• https base URLs, with certificate verification and no way to turn it off.
• A bearer credential held in the sponsor's OS credential store (Keychain, Windows Credential Manager, libsecret). Never in a JSON file, a manifest, a log or a peer message. The sponsor adds it only to its outbound request; clients never send or see it.
• A first-byte timeout per node. A scaled-to-zero endpoint takes minutes to cold-start, and the proxy's fixed 120 s response-header timeout (#57) turns that into a 502. For a Hugging Face endpoint, the sponsor also sends X-Scale-Up-Timeout with the same value. HF's proxy then holds the request instead of answering 503 while a replica starts.
• The Rented · off-LAN label, and a share_with_cluster switch, off by default. Until the user turns it on, only callers on the sponsor itself can use the rented node.

This slice covers any OpenAI-compatible HTTPS endpoint:

• a Hugging Face Inference Endpoint;
• a Hugging Face Job running vllm serve behind a token-protected exposed port (https://<job_id>--8000.hf.jobs);
• Hugging Face Inference Providers (https://router.huggingface.co/v1, billed per token);
• another provider's GPU rental.

2. Load a GPU and a model from PAIR (Hugging Face adapter)

An Add rented GPU action on the sponsor. It uses Hugging Face's Inference Endpoints REST API, the same calls huggingface_hub and hf endpoints make, so PAIR needs no Python.

1. The user signs in with a fine-grained Hugging Face token.
2. They pick a model, either from Hugging Face's endpoint catalog (models with tested hardware and engine recipes) or by repository id. Then they pick a GPU from the list Hugging Face returns, which carries the hourly price and the account's quota.
3. PAIR creates a token-protected endpoint, never a public one. The node shows Initializing (billing starts here: HF bills by the minute while initializing and running), then Running, and the model joins the cluster inventory.
4. Node actions: Pause (no charge; needs an explicit resume), Resume, and Delete (irreversible on Hugging Face, so ask for confirmation).

How PAIR treats the node in each Hugging Face state (for nodes it created):

Hugging Face state	PAIR treats the node as
running	eligible
scaledToZero	eligible with a cold start; sends X-Scale-Up-Timeout
paused	not eligible. With wake on request on, the sponsor resumes it and the request waits up to the cold-start budget
pending, initializing, updating	starting; not eligible
failed	unhealthy; node detail shows Hugging Face's message

Cost controls, all enforced by the sponsor:

• Idle pause after N minutes with no PAIR workload. Pause, not scale-to-zero: HF bills neither, but a scaled-to-zero endpoint still counts against the account's GPU quota and wakes on any request.
• Budget cap per day or month, computed from HF's hourly price. At the cap, PAIR pauses the endpoint and marks the node ineligible, with the reason.
• Pause on sponsor exit, on by default, so quitting PAIR doesn't leave a GPU billing. As a backstop for a sponsor that crashes or goes offline, PAIR also turns on HF's own scale-to-zero (for example, after 15 minutes) when it creates the endpoint.
• Running cost for the session and the month, shown in the node detail.

Hugging Face Jobs could be a second adapter, for batch windows. Same per-minute GPU prices, a hard --timeout as the safety stop, and a token-protected exposed port. But no scale-to-zero, and the model loads again on every start.

3. Routing policy for rented nodes

PAIR's scheduler routes on the exact model tag and current load. To it, a free local GPU and a $5/hr rented one look the same. Each rented node needs a policy:

use	Behavior
when-no-local-node (default)	only for models no local node advertises
overflow	also when every local owner of the model is busy
prefer	ahead of local nodes, for a batch the user rented the GPU for
never	attached but not routed to (testing, or budget reached)

This is the priority idea from #24 (items 4–5) and the placement signals in #7. Rented nodes only need a cost class the scheduler can see.

Example declaration (illustrative only):

rented_node:
  name: hf-l40s
  provider: huggingface-endpoint      # huggingface-job | openai-compatible
  endpoint: <namespace>/<endpoint-name>
  credential: os-credential-store:pair/hf
  share_with_cluster: false
  routing:
    use: when-no-local-node
  timeouts:
    cold_start_s: 600                 # first-byte timeout and X-Scale-Up-Timeout
  lifecycle:
    managed: true
    wake_on_request: true
    idle_pause_min: 15
    pause_on_sponsor_exit: true
  budget:
    usd_per_day: 10

Alternatives considered

• Point clients at the rented endpoint directly. Works for one app, but loses PAIR's single endpoint, inventory, failover and Jobs list, and puts the token in every client.
• A loopback reverse proxy that adds the token, adopted through #64's openai_base_url. This should work on that branch today (I haven't tested it). But PAIR then sees what looks like a free LAN machine: no off-LAN label, no cost, no wake or pause. And the token sits in the reverse proxy's config file.
• A separate router in front of PAIR (LiteLLM or similar, as #24's author does with their own router). Duplicates discovery, inventory and failover, and peers still can't share the rented GPU without the token.
• Install PAIR on the rented machine. Hugging Face endpoints and Jobs are containers behind HF's proxy: no mDNS, no inbound ports for PAIR's mTLS services, no PIN pairing. A full rented VM elsewhere could run PAIR. But it would then be a paired node with engine control over the whole cluster (port 14323), which is far more trust than burst capacity needs.
• Hugging Face Spaces and ZeroGPU. Built for Gradio apps and per-call GPU allocation, not a long-running OpenAI-compatible server. Out of scope.

Compatibility and security implications

• Prompts leave the LAN. This changes what PAIR promises, so it has to be explicit:

  • opt-in for each rented node;
  • share_with_cluster off by default;
  • the off-LAN label everywhere a node or workload appears.

  A peer's prompt should not go off-LAN unless that peer's user can see that it will. Hugging Face states that it doesn't store endpoint payloads or tokens, encrypts traffic in transit, and keeps logs for 30 days. Its terms and the chosen region apply.

• Credential. A fine-grained token with the least scope that works: calling and managing endpoints, or running Jobs. It stays on the sponsor, in the OS credential store, and is never sent to peers or logged. The secret handling designed for #2/#26 should cover local engines and rented nodes, rather than PAIR growing two mechanisms.

• No public endpoints. PAIR creates endpoints as token-protected and exposes Job ports with --expose, never --expose-public. It warns when an attached endpoint answers without a credential.

• Transport. No new listening ports. The sponsor makes outbound HTTPS requests only, with certificate verification. Peer traffic uses the existing mTLS path.

• Cost is a safety property. A runaway client on any peer, such as an agent loop, can keep a $5/hr GPU busy. Budget caps, idle pause and pause-on-exit are part of the feature, not polish. Billing someone for usage requires knowing who sent the request, and workload history's requesterId is always null today. At minimum, record the origin node and an estimated cost for each workload.

• Audit. The proxy rightly doesn't log prompts. For off-LAN work the metadata record matters more: which node asked, which rented node served it, which model, when, for how long, at what estimated cost. An optional model allowlist for each rented node would let an operator limit what can be rented.

• Migration. Purely additive. Clusters without rented nodes behave exactly as today.

Validation approach

All of this costs under $2 on a 1× L4 Inference Endpoint ($0.80/hr) serving a small model:

1. Attach and share. Register the endpoint on node A. With sharing off, a request from node B for its model gets a clear refusal naming the rented node. Turn sharing on, and the same request is served, streams, and appears in the Jobs list on both nodes, with the rented node as the serving node. Search node B's data directory and logs for the token: it should be absent.
2. Cold start. Let HF scale the endpoint to zero. The next request is served after the cold start, not cut off at 120 s.
3. Idle pause and wake. Set a 5-minute idle pause; afterwards HF reports paused. With wake off, a request fails with a message naming the paused node, not model inventory unavailable (see #2). With wake on, PAIR resumes the endpoint and serves the request.
4. Budget. Set a cap of $0.10. After about 7½ minutes on the L4, PAIR pauses the endpoint and marks the node ineligible, with the reason.
5. Sponsor exit. Quit PAIR on node A. hf endpoints describe reports paused.
6. Precedence. Serve the same model locally and on the rented node. Under the default policy, the local node serves. Under overflow, with the local owner busy, the rented node serves.
7. Refusals. Each of these produces a specific error: an http:// URL to a non-loopback host, a public endpoint, and a wrong token.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

Related, but not duplicates:

• #24 and PR #64: external OpenAI-compatible endpoints. This request adds HTTPS, credentials, lifecycle and cost on top.
• #2 and #26: credentials for engines.
• #7: placement signals.
• #57 and PR #93: timeouts.
• PR #38: API keys for LAN access.
• PR #90: third-party workload ingress.

Appendix: Hugging Face rental options (prices checked 2026-10-11)

Inference Endpoints, AWS, one GPU. Billed by the minute while initializing and running; a paused endpoint costs nothing. Multi-GPU sizes scale linearly (8× H200 is $40/hr).

GPU	VRAM	$/hr
T4	16 GB	0.50
L4	24 GB	0.80
L40S	48 GB	1.80
A100	80 GB	2.50
RTX PRO 6000	96 GB	2.75
H100	80 GB	4.50
H200	141 GB	5.00
B200	—	9.25

How each Hugging Face service fits:

Service	Billing	Interface	Fit
Inference Endpoints	per minute, by GPU	OpenAI-compatible (vLLM, SGLang, llama.cpp, TGI); API to create, pause, resume and scale to zero	slices 1–3
Jobs	per minute, by GPU (same prices), plus $0.01/hr for an exposed port; default timeout 30 min	whatever the container serves, on a token-protected HTTPS port	slice 1; adapter for batch windows
Inference Providers	per token at provider rates, no HF markup; PRO includes $2/month	router.huggingface.co/v1, chat completions only	slice 1 only; no GPU to manage
Spaces GPU / ZeroGPU	per hour / free quota	Gradio app	out of scope

Sources:

• Hugging Face pricing
• Inference Endpoints pricing
• Inference Endpoints autoscaling
• Inference Endpoints security
• Managing endpoints with huggingface_hub
• Jobs pricing
• Jobs configuration
• Inference Providers