Experiment: lossless weight scheduling for models larger than RAM

Start by reviewing the reported madvise experiments, the scheduled-streaming measurements, and the controlled fixed-kernel comparisons. A next step requires deciding whether this experimental direction belongs in dev and whether its controls should remain environment variables or become a memory-budget option; done is an agreed implementation scope and validation plan.

Hi @jan-wassenberg, following your suggestion in google/gemma.cpp#982,
I tested madvise policies for mapped weights and activations.

I tried the suggested advice modes and safe combinations: MADV_WILLNEED,
MADV_HUGEPAGE, MADV_POPULATE_READ, and MADV_COLLAPSE. Advice alone did
not produce a meaningful repeatable end-to-end gain. For example, activation
huge pages reduced minor faults but changed throughput by only about 0–1%, and
WILLNEED did not improve either 4B or 27B. I did not populate the entire 27B
mapping because that would force severe reclaim on this machine.

Here are my findings. The 27B test exposed a different problem: its 27.27 GiB
checkpoint has about 25.2 GiB of mapped weights active during text inference,
but the machine has 15.5 GiB RAM. Each token cycles through the weights,
evicting pages needed by the next token and rereading almost the entire active
set from storage.

I prototyped an opt-in, lossless policy:

• retain 8.45–8.81 GiB of exact weight copies across tokens;
• distribute the retained layers through the network;
• stream nonresident layers through two reusable O_DIRECT buffers;
• read the next nonresident layer while the CPU computes the current layer;
• keep a memory-headroom guard and fall back to a smaller cache when needed.

No weight precision, batching, or decoding algorithm changes.

On an i5-12400F with 15.5 GiB RAM, medians of three interleaved cold runs with
Gemma 3 27B SFP, batch 1, and eight output tokens were:

That is 3.07x faster steady decoding, 2.42x faster cold requests, and 33.6%
fewer physical reads. Controlled fixed-kernel runs matched baseline decode-logit
hashes exactly. The final normal-autotuning runs produced identical text, though
autotuning changes floating-point accumulation order even between baseline runs.

This is only intended for Linux systems where the model's active weights exceed
safe available RAM. Models that already fit, such as 4B on this machine, become
slower because explicit copying and streaming add overhead.

Would this experimental, opt-in direction be useful in dev? In particular,
feedback would be helpful on whether the controls should remain experimental
environment variables or become a smaller user-facing memory-budget option.