DS4.1 Flash disk Engram: 8–32 GiB cache leaves long first-touch prefill unchanged; warmed controls recover throughput

Start with the pinned source revision’s helios/src/engram/reader.rs and model.rs, then inspect the published CSV and reproducer artifacts linked in the issue. Measure effective outstanding reads and GPU wait time for gathered rows across first-touch and warmed cases. Done means identifying a trace or experiment that distinguishes cache misses from delivery or synchronization costs and clarifies the TP4 metric scope.

Summary

On the released DS4.1-Flash-1017 TP4 package, long first-touch prefill is substantially slower with disk Engram than the prior RAM control. Increasing the CLI row-cache budget from 8 to 16 to 32 GiB does not materially improve first-touch TTFT, although it improves warmed reuse for the larger random workload.

This is a reproducible performance investigation, not a claim that disk mode should equal RAM, and not a demonstrated defect in a specific I/O scheduling mechanism. I am asking which trace/experiment would best distinguish unavoidable misses from a remediable delivery/synchronization bottleneck.

Pinned environment

• 4 × RTX PRO 6000 Blackwell Workstation Edition, 96 GB each; Intel Xeon w9-3495X; ~251 GiB host RAM; Linux 7.2.10-arch1-1.
• GPU order 0,1,3,2; configured GPU power limits 600 W, unchanged throughout. No power/clock tuning during the campaign.
• Model on local NVMe/Btrfs; benchmark server isolated on loopback, C1 requests.
• Image: ghcr.io/local-inference-lab/flare@sha256:67124437a73011d63c2b25c939b9c48581eba90e51c6e565d4c808d161d10fc6.
• OCI source revision: 2193455651794642debdaad0dda91e77ab675031.
• Package: local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, gb202 commit 2fd901741cf227b7d620feea43a8cc05f02fbe6b; primary SHA begins ec4b7027707c.
• TP4/C32 package, MTP7 adaptive, alpha 1, throughput scheduler, prepared 4096-token prefill chunks, resident Engram scales enabled, row-cache flag swept 8/16/32 GiB. No disk-prefix-cache flag.
• Container: seccomp unconfined for io_uring; unlimited memlock; memory/swap ceiling both 64 GiB.

Protocol

108 validated requests: 3 cache budgets × 3 seeds × 2 input types × 2 lengths × 3 conditions.

For each cache/seed/input/length case, start a fresh server, perform the same small arithmetic smoke, then run:

1. First presentation of the long body (fresh application row cache apart from the smoke).
2. Same body with the opening marker changed, defeating useful KV reuse while preserving Engram locality.
3. Related-but-new input and a new opening marker.

All 108 windows had exactly one completed request and one reused KV token. Identical payload hashes and actual prompt token counts were verified across cache budgets. max_tokens=2; TTFT detects content, reasoning_content, or reasoning; SSE completion and usage are required. Request timing is monotonic.

Seeds: 101, 202, 303. Cache order was balanced across seeds: 8/16/32, 16/32/8, 32/8/16. The random corpus follows the LiL thread's pseudoword generator, seeded; related input replaces every tenth word while keeping word lengths. Code is a seeded ordering of Python 3.14 stdlib source; related code uses deterministic identifier-style text substitutions, not a claimed executable program revision. Actual sizes are approximately 248k–253k / 499k–509k; actual counts are in the CSV. No host-global page-cache flush; direct-I/O server settings are held fixed.

TTFT results

Seconds, median of 3 seeded cases. Observed min/max and per-request data are published; these are not confidence intervals.

The 500k random warmed case improves from 40.91 s at 8 GiB to 37.85 s at 16 GiB and 37.19 s at 32 GiB; the first-touch case remains ~61 s. First-touch code likewise stays ~24/~51 s at all three budgets. This separates a modest warm-retention benefit from the persistent first-touch miss cost. I would not recommend 32 GiB as a general cure based on these results.

Counters and scope caveats

• Fresh random ~500k, median exported disk-wait counter deltas: 45.16 / 45.06 / 44.99 worker-seconds at 8/16/32 GiB; exported disk reads ~4.65M each. Gather deltas ~48.34 / 48.22 / 48.14 seconds.
• For the same-body/new-prefix ~500k random request, median exported row-cache hit rates rise 92.07% / 98.97% / 99.90%; eviction counts fall substantially. More cache is being used, but first-touch latency remains unchanged.
• Worker-time counters are not additive slices of request wall time and should not be treated as complete TP4 accounting without confirming scope.
• Scope question: startup flare_engram_row_cache_bytes is approximately 2/4/8 GiB for CLI flags 8/16/32 GiB; resident-scale gauge is ~1.43 GiB. The tested model.rs divides the budget among Engram groups, and its startup summary filters bindings by rank 0. Can you confirm whether /metrics describes one group or the complete TP4 set? The public CSV preserves the exported values without multiplying by four.
• I also sampled /sys/block/nvme1n1/stat and inflight. Fresh random ~500k gives whole-device mean read await ~0.315 ms and mean queue depth ~48. These are coarse, whole-device interval observations, can include other root-device traffic, and are not per-Flare latency distributions or proof of saturation.

Earlier matched RAM/disk control (separate campaign)

Same host, image and package, one random ~500k request/arm: Flare disk TTFT 59.12 s vs RAM 37.07 s. Interior GPU samples averaged 72.4% utilization / 317 W with disk versus 99.9% / 437 W with RAM. This is one observation per arm, not part of the three-seed ladder, and not a claim about other rigs' power limits. The prior vLLM disk control also slowed on random long inputs. Exact-repeat KV hits were tested separately and are excluded from the throughput conclusions.

Source check and questions

The reader at the image's exact source revision already defines DEPTH=128, submits ReadFixed batches, and drains each batch before moving on. Linux rejects disk workers other than one; I am not proposing “enable QD24” or the Windows workers flag.

1. What trace would best measure effective outstanding reads and time spent waiting for gathered rows on the GPU critical path?
2. Is batch draining, completion handling, or gather/compute synchronization a plausible contributor worth testing, or is this expected first-touch behavior for this package?
3. Are additional prefetch/overlap or cache-retention controls already available on a newer revision? This report is deliberately pinned to the release above; I have not benchmarked current main.
4. Would per-gather latency/miss-count histograms and explicit metric scope/rank labels be useful additions?

This runs at C1 with no competing decode requests; it does not reproduce the mixed ordinary-decode starvation scenario in #42. Random strings are a stress case; the code control is included to avoid generalizing them to every chat workload.

Reproduce and inspect the data

Public artifacts: https://gist.github.com/jayinsine/ec4c0e2799d813e15a19e3fe51bb5eb8

• launch-reproducer.sh: pinned image/package and the exact serving flags; no automatic shutdown of any existing services.
• reproduce_random_prefill.py: deterministic random first-touch / prefix-busted / related trio. Its generated bodies match the measured files byte-for-byte, and its SSE client was smoke-tested on a live endpoint.
• cache-ladder-108-requests.csv: all measurements, token counts, hashes, cache/Engram deltas and coarse hardware samples.
• aggregates.json: median/min/max for each condition.
• cases.json: payload hashes, seeds, counts and related-input transformations.

Example (after downloading the pinned package, on an idle GPU group):

CACHE_GIB=8 ./launch-reproducer.sh
# In another terminal:
python reproduce_random_prefill.py --url http://127.0.0.1:18080 --target 500000 --seed 101

Restart the server before changing the cache/seed/length case. Repeat with 16/32 GiB, seeds 101/202/303 and target 250000. The published ladder additionally includes the code corpus described above. A small arithmetic smoke preceded each measured trio; it warms only a small number of rows.

Originating discussion: LiL Disk engram prefill problem thread. Measurement collection and source review were AI-assisted; observed data, source facts and hypotheses are distinguished above.