aarch64: paired ldp/stp on adjacent mi_page_t fields makes malloc+free 27–29% slower than v3.3.2

Start with src/static.c and the mi_page_t field accesses described in the issue, then compare the aarch64 disassembly for mi_malloc and mi_free with and without load/store pairing. Reproduce the 32-byte allocation/free loop on aarch64 using the provided commands. Done means identifying a change that restores the short-lived allocation performance without causing regressions in the other reported benchmarks.

On aarch64, a mi_malloc immediately followed by mi_free (the pattern of a short-lived object) is 27–29% slower in v3.5.3 than in v3.3.2, and still slower on dev3 (e34c9d38). The slowdown goes away when the compiler is told not to pair loads and stores, so it appears to come from how the fast paths access mi_page_t rather than from extra work.

Results

Graviton5 (Neoverse-V3), Linux 6.12, isolated core, ASLR off. Each line is the best of 7 rounds of 50M mi_malloc(32) + mi_free pairs, repeated twice with identical results (±0.02 ns).

"No load/store pairing" compiles static.c with --param=aarch64-ldp-policy=never --param=aarch64-stp-policy=never (GCC 14+) or -mllvm -aarch64-enable-ldst-opt=false (Clang). On an Apple M1 Max (Apple clang 21), v3.5.3 through a Rust binding showed the same direction: 6.7–7.0 ns → 5.1–5.4 ns per pair.

A batched pattern (64 allocations, then 64 frees) is not affected, because consecutive calls then access the same addresses with the same widths.

Disassembly

In v3.5.3 and dev3, free (offset 0x10), xused (0x18, 64-bit) and local_free (0x20) are adjacent in mi_page_t (with MI_PAGE_META_IS_ALIGNED). Both compilers merge the two field accesses of each fast path into one 16-byte pair (dev3, -O3):

GCC 14.2                               Clang 19.1.7
mi_malloc:                             mi_malloc:
  ldp  x0, x2, [x3, #0x10]  ; free, xused   ldp  x8, x10, [x9, #0x10]
  ...                                    ...
  stp  x1, x2, [x3, #0x10]               stp  x11, x10, [x9, #0x10]
mi_free:                               mi_free:
  ldp  x1, x3, [x2, #0x18]  ; xused, local_free   ldp  x8, x9, [x0, #0x18]
  ...                                    ...
  stp  x1, x0, [x2, #0x18]               stp  x8, x1, [x0, #0x18]

So each call's 16-byte load overlaps only half of the 16-byte store made by the previous call on the same page. Our reading (inferred from the disassembly and from the effect of disabling the pairing; not confirmed with PMU events) is that such a partially overlapping load cannot be served by store-to-load forwarding and has to wait for the store to complete. In v3.3.2 the used count was a uint16_t accessed with ldrh/strh at its own address, so each load matched an earlier store exactly.

In a larger build we also tried -DMI_FREE_USE_PAGEMAP=1, -DMI_PROFILE=0 and -DMI_STATS=0; none of them changed the result (with the page map the offsets shift, but the accesses are still paired).

Reproduction

// repro.c: one mi_malloc immediately followed by mi_free, in a loop.
#include <mimalloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

static double now(void) {
  struct timespec t;
  clock_gettime(CLOCK_MONOTONIC, &t);
  return (double)t.tv_sec + (double)t.tv_nsec * 1e-9;
}

int main(int argc, char** argv) {
  size_t size = argc > 1 ? strtoul(argv[1], NULL, 10) : 32;
  const long iters = 50000000;
  void* volatile sink;
  for (long i = 0; i < 1000000; i++) { void* p = mi_malloc(size); sink = p; mi_free(p); }
  double best = 1e9;
  for (int round = 0; round < 7; round++) {
    double start = now();
    for (long i = 0; i < iters; i++) { void* p = mi_malloc(size); sink = p; mi_free(p); }
    double ns = (now() - start) / (double)iters * 1e9;
    if (ns < best) best = ns;
  }
  (void)sink;
  printf("size %zu: %.2f ns per mi_malloc + mi_free (best of 7)\n", size, best);
  return 0;
}

FLAGS="-O3 -DNDEBUG -DMI_DEBUG=0 -ftls-model=initial-exec -fvisibility=hidden"
gcc $FLAGS -Iinclude -c src/static.c -o mi.o
gcc $FLAGS --param=aarch64-ldp-policy=never --param=aarch64-stp-policy=never -Iinclude -c src/static.c -o mi-nopair.o
gcc -O2 -Iinclude repro.c mi.o -lpthread -o repro && ./repro 32
gcc -O2 -Iinclude repro.c mi-nopair.o -lpthread -o repro-nopair && ./repro-nopair 32

Possible fixes

• (Untested) Change the field layout or the access widths so that a fast path's loads never partially overlap the stores of the other fast path on the same page. Simply reordering the three fields may not be enough, since a 16-byte load can still half-overlap an 8-byte store; narrower accesses to the used count, as in v3.3.2, avoid the pairing altogether.
• (Tested) Build with the compiler flags above on aarch64. They apply to the whole library; through our Rust binding on Neoverse-V3, a batched alloc/free benchmark changed by −3% to +11% depending on size and compiler, and a cross-thread free benchmark by −1.5% to +1.4%.

We have not checked other aarch64 cores or x86_64 (where these paired instructions do not exist).