Grow a cell with its streams: scale the fleet on what fills, and split and merge shards

Start by reading the D3 description in this issue, then inspect the fleet loop in src/fleet.rs, topology creation in src/registry.rs, and the referenced src/split.rs and src/merge.rs work on slate-codex. Review the crash, stale-owner, race, and takeover CI scripts before considering implementation. Done means the one-shard cell can scale out and back in through the stated ramp while meeting its success and latency thresholds.

A cell has to grow with the number of streams it holds. Today it can't do that by itself, for two reasons:

1. The fleet autoscaler never sees this workload. The cell refuses work while every signal the autoscaler reads stays under its target.
2. A cell's shard count is fixed when its topology is created. An instance beyond the shard count owns nothing. So a cell that may grow to 64 instances must start with at least 64 shards and pay for all of them while it runs on a few instances.

We want a cell that starts on one instance with one shard, splits shards and adds instances as its streams grow, and merges shards and removes instances as they go.

What we measured

On 2026-11-02 we ramped a cell from 10 to 10,000,000 tiny streams, ten times more at each step. Each stream was touched once an hour, by one 400-byte append and one read of it, with auth enforced. The streams were owned by one tenant, or each by its own. The build was slate at e1ba5757, with #146, #147 and #148.

Because the autoscaler did not react, each step pinned FLEET_MIN = FLEET_MAX = INITIAL_SHARDS = N. When a step did not hold, N was doubled, up to 64. A step held when at least 98% of touches succeeded with a p99 of at most 2 s, and nothing was shed.

Streams	One tenant	A tenant each
10 to 10,000	1 instance	1 instance
100,000	1 instance, p99 453 ms	8 instances, p99 420 ms
1,000,000	1 instance, p99 490 ms, 341 MB RSS	not held on 64: every instance OOM-killed 17 to 20 s after it started serving
10,000,000	16 instances, p99 472 ms; not held on 8	not run

Queues ran the same ramp, each queue touched by one send, one claim and one acknowledgement. One tenant needed 2 cells at 100,000 queues and 32 at 1,000,000. 10,000,000 queues did not hold on 64.

1. The autoscaler does not trigger

The fleet loop sizes the fleet in src/fleet.rs:519-575. It takes the largest of five signals:

• cores in use against the 75% target;
• requests in flight against SCALE_EDGE_SLOTS;
• sustained ack p50 above SCALE_LATENCY_MS;
• edge p50 above its bound;
• one instance's CPU sustained above the hot threshold.

All of these measure CPU or request concurrency. The tiny-stream workload hits a different bound each time, while CPU stays low:

• Read memory. At 10,000,000 streams on 8 instances, only 79% of touches succeeded. Reads were refused 503 read_memory_busy, and p99 was 3.2 s. The fleet used 5.3 cores, under the 6 cores (75% of 8) at which it scales out. A read without maxBytes reserves 8 MiB of the 125 MiB read budget while it waits on the store.
• Project tracker. An instance tracks at most 32,768 active projects, and refuses a create beyond that with project_tracker_capacity. With a tenant per stream, 100,000 streams created only 32,768 on 1 instance, 43,550 on 2 and 74,585 on 4, all at low CPU. One reason it took 8 instances: the instance that receives a create counts the project, as well as the owner.
• Descriptor cache. The cache holds REGISTRY_CACHE_MAX (65,536) descriptors. Beyond that, every touch reads its descriptor from the store: 300 Class B reads a second for 278 touches a second at 1,000,000 streams. This shows up as store latency, not as CPU.
• Store waits on a queue cell's cold path. A queue cell held about 50,000 queues touched once an hour, about 14 touches a second. A cell of 62,500 held 55%, with a median of 13 s, at 0.04 of a core. Each touch waited on about three store reads.
• RSS. RSS grew about 0.26 KB per stream on one instance, plus about 5 KB per tenant for the feeds. Every instance was OOM-killed at 1,000,000 tenants on 64.

Proposal

Report the following per instance in the heartbeat, and add each as a signal to the desired count:

• Refusals. Sustained refusals of any kind, such as read_memory_busy, project_tracker_capacity or an admission shed, ask for live + 1. They are damped like the hot-CPU signal. While refusals are sustained, the fleet does not scale in, as edge congestion already blocks it.
• Occupancy. A resource that fills up asks for ceil(total / (target × per-instance capacity)). These are resident streams against the descriptor cache and the handle limit, active projects against the tracker's capacity, and RSS against ADMIT_RSS_SHED_MB.
• Store wait. A sustained p50 for store reads on the touch path asks for live + 1. Ack p50 misses this, because a refused or slow read is not an ack.

Today, every signal except latency applies only to an instance serving at least 5 requests a second. The occupancy signals should apply to every instance, because a cell full of tiny streams can be full at a low request rate.

Some of these bounds should be raised rather than scaled around, and they are separate work. One is a configurable descriptor cache that revalidates with If-None-Match. Another is a read reservation sized by the stream rather than 8 MiB. A third is counting a project only on its owner. Even with those, the fleet needs a signal for each resource that can fill up.

Adding an instance only helps if the instance gets something to own. That is the second half of this issue.

2. Shards cannot split or merge

topology.json is written once, from INITIAL_SHARDS (src/registry.rs:1250-1300), and never changes. The ring assigns those shards to instances. An instance beyond the shard count owns no shard and adds no capacity, so a cell's largest useful fleet is fixed at creation. The config notice for INITIAL_SHARDS > FLEET_MAX states the reverse cost: every shard with traffic pays its own WAL PUTs, whatever it carries.

What a shard costs, as measured:

• A shard with traffic writes about 10 to 12 WAL PUTs a second at the 100 ms flush interval. At 10,000,000 streams, 16 shards wrote 184.6 Class A requests a second. Over the run, these PUTs and the instances made up most of the cell's cost.
• An idle shard writes no WAL, but it reads about 0.9 Class B a second and may hold up to MAX_UNFLUSHED_BYTES (16 MiB).

To be able to reach 64 or 128 instances, a cell must therefore be created with 64 or 128 shards. A small tenant's traffic is then hashed across all of them, so it pays for many WAL writers instead of one. This defeats the point of a small cell.

The specification already describes what we want, as D3, dynamic shard topology. The service starts at one shard. It splits a hot shard into its two children by bit prefix of the stream hash, and merges cold siblings. Each split or merge is a metadata-only change to one shard, never a global reshard. The PUT cost and the footprint follow actual usage, not a provisioned shard count.

It was built, and never merged

D3 was implemented on the slate-codex branch on 2026-07-18:

• eadaa2d6 "add crash-safe automatic online shard splitting": src/split.rs, the shard and registry changes, and the online-split, auto-split and stale-owner CI scripts.
• d1f8afb0 "add crash-safe online sibling merge": src/merge.rs, src/reconfiguration.rs, and the merge crash matrix, the stale-owner, takeover-GC and split/merge race scripts.
• 16c09ee0 "automate conservative cold sibling merges": the fleet loop's merge trigger, with the auto-merge and remote-owner scripts.
• 4990f345 "decouple reconfiguration fences from durable ack latency", on 2026-07-19, which removed a fence GET from the ack path.

On 2026-07-21, 8e8d77bb carried the branch's safety core into slate and abandoned the rest. The branch as a whole had crash-looped at the platform's memory kill line. Split and merge were deferred with it. AWS-readyness.md §4, item 5, calls the design sound, with renewable intents beside the shard data, fences after durability, a topology published by one CAS, and a 6× gap between the split and merge triggers. It says the CI matrix is worth porting wholesale "when shard counts demand it". These measurements show that shard counts now demand it.

The per-stream segment splits of docs/ROUTING-V3.md are a different mechanism, for one hot stream rather than many small ones. They are also declined whenever a ring is published (src/scaler3/controller.rs:114-139). Nothing in slate today changes the number of shards a cell has.

Proposal

• Port. Bring split and merge from slate-codex onto slate's current code: the registry and shard directory, the ring with its overrides, and fair-share shard moves. Keep the 4990f345 rule that a liveness check never adds a store round trip to the ack path. Port the crash, stale-owner, race and takeover CI matrix with it.
• Trigger. Split a shard when its load passes a threshold, whether CPU, appends, resident streams or unflushed bytes. Also split when the fleet's desired count exceeds the shard count, so the next instance has a shard to take. Merge cold siblings with the 6× hysteresis. Couple the two: the fleet never wants more instances than there are shards, and splits before it scales out.
• Propagate. Non-owners must learn a new topology version before they forward to a child. A stale forward should be answered with a redirect to the new owner, not an error.
• Footprint. Measure the memory of a split and of each resident shard against ADMIT_RSS_SHED_MB before this lands, because memory is what ended the branch.

Done when

• A cell starts with FLEET_MIN=1, FLEET_MAX=64 and one shard.
• The same ramp, from 10 to 10,000,000 tiny streams with one tenant and with a tenant each, holds at least 98% of touches with a p99 of at most 2 s at every step. The cell grows its shards and its instances by itself.
• At the low steps, the cell runs on one instance with few shards.
• When the load goes, the cell merges shards and scales back in.