perf: inlined receive future, faster on every row but fan_out/workers/16 on Neoverse-N2

Open
  • Is something wrong?
#183 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@hseeberger is already working on this.

Since Oct 27, 2026.

Assessment

This issue has not been assessed yet.

Description

Found while investigating the cost of #174: a hand-written receive future makes every messaging row of the stack sweep faster, except fan_out/workers/16 on Neoverse-N2. This issue keeps that change and its open question separate from the timer work.

The change

One change to mailbox.rs: Mailbox::recv returns a Recv future whose #[inline(always)] poll wraps flume's RecvFut and releases the quota in the same poll. Under hotpath an async fn wrapper keeps the instrumentation. The diff applies to 39aec83.

Diff 
diff --git a/tellus/src/mailbox.rs b/tellus/src/mailbox.rs
index 5df9522..a669df5 100644
--- a/tellus/src/mailbox.rs
+++ b/tellus/src/mailbox.rs
@@ -3,10 +3,14 @@ use crate::{
     quota::{CountedSendError, CountedSender, Full, Quota},
     watch::{ActorTerminated, TerminatedHandler, WatcherRegistration, WatcherRegistry},
 };
-use flume::Receiver;
 #[cfg(feature = "test-util")]
 use flume::TryRecvError;
-use std::sync::Arc;
+use flume::{Receiver, r#async::RecvFut};
+use std::{
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll, ready},
+};
 use thiserror::Error;
 
 /// Sending mailbox half.
@@ -65,14 +69,15 @@ pub(crate) struct Mailbox<M> {
 }
 
 impl<M> Mailbox<M> {
-    /// `&mut self` to enforce one consumer per mailbox.
-    #[cfg_attr(feature = "hotpath", hotpath::measure)]
+    #[cfg(feature = "hotpath")]
+    #[hotpath::measure]
     pub(crate) async fn recv(&mut self) -> Option<Incoming<M>> {
-        let incoming = self.incoming_rx.recv_async().await.ok()?;
-        if matches!(incoming, Incoming::Message(_)) {
-            self.quota.unreserve();
-        }
-        Some(incoming)
+        self.recv_future().await
+    }
+
+    #[cfg(not(feature = "hotpath"))]
+    pub(crate) fn recv(&mut self) -> Recv<'_, M> {
+        self.recv_future()
     }
 
     #[cfg(feature = "test-util")]
@@ -92,6 +97,32 @@ impl<M> Mailbox<M> {
     pub(crate) fn split(self) -> (Receiver<Incoming<M>>, WatcherRegistration) {
         (self.incoming_rx, self.watcher_registration)
     }
+
+    /// `&mut self` to enforce one consumer per mailbox.
+    fn recv_future(&mut self) -> Recv<'_, M> {
+        Recv {
+            incoming: self.incoming_rx.recv_async(),
+            quota: &self.quota,
+        }
+    }
+}
+
+pub(crate) struct Recv<'a, M> {
+    incoming: RecvFut<'a, Incoming<M>>,
+    quota: &'a Quota,
+}
+
+impl<M> Future for Recv<'_, M> {
+    type Output = Option<Incoming<M>>;
+
+    #[inline(always)]
+    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
+        let incoming = ready!(Pin::new(&mut self.incoming).poll(cx)).ok();
+        if matches!(incoming, Some(Incoming::Message(_))) {
+            self.quota.unreserve();
+        }
+        Poll::Ready(incoming)
+    }
 }
 
 #[derive(Debug, Error)]

Measured

Stack sweep run 40410138410 against 39aec83 (four Neoverse-N2 jobs; x86_64 three EPYC 7763, one EPYC 9V74):

Row N2 x86_64
queued/* (4 rows) pass, -5.0 to -5.7% pass, -6.9 to -7.5%
flood/unbounded, /bounded pass, -6.0%, -3.6% spread, -8.3%, -9.1%
fan_out/workers/4 pass, -2.5% pass, -17.4%
fan_out/workers/16 fail, +9.0% (every job +8.2 to +11.4%) pass, -10.0%
ping_pong/pairs/1, /4 pass, -0.7%, -1.0% spread -2.3%, pass -1.2%

The Xeon 6973P-C, which showed the /16 cost in earlier sweeps, was not drawn. Timed actors run 7 to 13% faster with it on top of the timer cursor.

What is known about /16

The cost is per message, not per actor. Its workers wait for a message on more of their receives than on main. Call-chain profiles put the added time in flume's channel mutex (the root's send against the worker's recv) and in tokio's wake and scheduling paths; the stop channel's Notify mutex costs the same in both. Producer bursts cut the waiting but not the penalty. The local machine does not reproduce the row; only the N2 runner decides it. Details: the cursor with an inlined recv future through main with the inlined recv future.

Open question

Either change how a woken worker drains its mailbox (several items per lock acquisition) so the row passes, or accept the row with the gains elsewhere. The merge criterion is unchanged: every messaging row passes the stack sweep on both runner types.

How to measure

  • Candidate as exactly one commit on its base, pushed as a branch: gh workflow run stack-sweep.yml -f base=<full ID> -f candidate=<full ID>, then python3 -I .github/stack-sweep/report.py <run ID>. Borderline rows get a second run; spread rows stay inconclusive.
  • Before the sweep: clippy on the three feature sets, just profile showing mailbox::recv at 110000 calls, just profile-alloc-gate.
  • Generated code: nm -C for out-of-line recv::{closure symbols, .github/stack-sweep/fncounts.py, fndiff.py and fnmnem.py on the bench binaries of both commits, both architectures (x86_64 cross build flags in the README's "How to measure").
Dominant language
Rust
Stars
37
Forks
1
Avg merge
6h 36m
Merged PRs (30d)
123

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tellus-rs/tellus

All issues in tellus-rs/tellus

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.