Sync with a peer stays wedged after one AlreadySyncing refusal (local state never leaves Running)

Aperta
#121 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust

Direzione di ricerca

Start with on_sync_via_connect_finished in src/engine/live.rs (around line 498), where AlreadySyncing is only logged, then read the sync state helpers in src/engine/state.rs. The failing test is sync_is_asked_again_after_an_already_syncing_refusal in tests/sync.rs; run cargo test --test sync sync_is_asked_again. Done when that test passes and the rest of tests/sync.rs still passes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Disclaimer - this issue was mostly prepared by AI - Claude Code with Opus 5.5 - under my supervision - it's an issue that was revealed in my project in tests - from what Claude has described looks like real problem.

After a single AbortReason::AlreadySyncing refusal from a peer, a node never syncs with that peer again until it restarts.

on_sync_via_connect_finished (src/engine/live.rs:498) only logs the refusal. It never calls on_sync_finished, so state.finish is never called, and the local PeerState for that peer stays SyncState::Running { origin: Connect(..) } for good. As a result:

  • every later start_connect for the peer is dropped as abort connect: sync already running (src/engine/state.rs:196-205). That includes an explicit start_sync, a NewNeighbor, and a SyncReport. A SyncReport sets resync_requested, but the finish that would act on it never comes;
  • no SyncFinished event is emitted for that round, so a subscriber cannot tell anything went wrong;
  • an incoming request from the peer is accepted only when expected_sync_direction picks Accept for the pair. Otherwise it is refused with AlreadySyncing too (state.rs:224-225), and the pair stays stuck in both directions.

How the refusal happens in practice

The remote is still finishing the previous round. When the accepting side has received entries, it broadcasts a SyncReport to its neighbours before it calls state.finish (live.rs:561-593). That leaves a window of tens of milliseconds in which its state is still Running { origin: Accept }. A round that the connecting side asks for as soon as it sees SyncFinished lands in that window and is refused with AlreadySyncing (state.rs:219).

The early return at live.rs:498 looks written for the simultaneous-dial case, where the other connection finishes the state. In the Origin::Accept case there is no other connection, so nothing ever resets the state.

We hit this in an application that asks for one more round with a peer once a round ends: a restarted node never caught up, in about 1 run in 7–20. As a workaround we now wait a second before asking.

Failing test

Branch: izderadicka/iroh-docs@test/already-syncing-wedge (one commit on main @ 8cfeacb, tests/sync.rs only). The test uses the repo's own lockfile, so no version pins are involved.

  • The peer refuses every request through iroh-docs' own net::handle_connection, with a chosen AbortReason. This makes the refusal deterministic.
  • A real node asks it for a sync three times. Each ask is repeated every 100ms until the peer sees it.
  • sync_is_asked_again_after_a_refusal is the control: it refuses with InternalServerError, and every ask reaches the peer.
$ cargo test --test sync sync_is_asked_again
test sync_is_asked_again_after_a_refusal ... ok
test sync_is_asked_again_after_an_already_syncing_refusal ... FAILED
Error: ask 2: the peer was not asked again within 10s

It fails at ask 2 every time: the first ask reaches the peer, and nothing does after that.

The test
/// A peer that refuses every sync request with `reason`, and says each time it
/// is asked.
#[derive(Debug, Clone)]
struct Refusing {
    sync: SyncHandle,
    reason: AbortReason,
    asked: mpsc::UnboundedSender<()>,
}

impl ProtocolHandler for Refusing {
    async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
        let (reason, asked) = (self.reason, self.asked.clone());
        let open = connection.clone();
        handle_connection(
            self.sync.clone(),
            connection,
            move |_, _| {
                asked.send(()).ok();
                async move { AcceptOutcome::Reject(reason) }
            },
            None,
        )
        .await
        .ok();
        // As a real peer does: the connection stays up until the dialer is done.
        open.closed().await;
        Ok(())
    }
}

/// Asks a peer that refuses with `reason` for a sync three times. Each ask is
/// repeated every 100ms until the peer sees it, so a round still finishing
/// on our side does not count against us.
async fn ask_a_refusing_peer_three_times(reason: AbortReason, seed: &[u8]) -> Result<()> {
    const ASK_TIMEOUT: Duration = Duration::from_secs(10);
    let mut rng = test_rng(seed);
    let node = spawn_node(0, &mut rng).await?;
    let doc = node.client().docs().create().await?;

    let (asked, mut was_asked) = mpsc::unbounded_channel();
    let peer = empty_endpoint().await?;
    let _router = Router::builder(peer.clone())
        .accept(
            iroh_docs::ALPN,
            Refusing {
                sync: SyncHandle::spawn(Store::memory(), None, "refusing".to_string()),
                reason,
                asked,
            },
        )
        .spawn();

    for attempt in 1..=3 {
        n0_future::time::timeout(ASK_TIMEOUT, async {
            loop {
                doc.start_sync(vec![peer.addr()]).await?;
                tokio::select! {
                    Some(()) = was_asked.recv() => return anyhow::Ok(()),
                    _ = n0_future::time::sleep(Duration::from_millis(100)) => {}
                }
            }
        })
        .await
        .with_context(|| {
            format!("ask {attempt}: the peer was not asked again within {ASK_TIMEOUT:?}")
        })??;
    }

    node.shutdown().await?;
    Ok(())
}

/// A refused sync is asked for again: the control for the test below.
#[tokio::test]
#[traced_test]
async fn sync_is_asked_again_after_a_refusal() -> Result<()> {
    ask_a_refusing_peer_three_times(
        AbortReason::InternalServerError,
        b"sync_is_asked_again_after_a_refusal",
    )
    .await
}

/// The same after an `AlreadySyncing` refusal, which a peer gives while it is
/// still finishing the round before.
///
/// Fails today: `on_sync_via_connect_finished` only logs that refusal, so the
/// local state for the peer stays `Running` and every later `start_sync`,
/// `NewNeighbor` or `SyncReport` for it is dropped as "sync already running".
#[tokio::test]
#[traced_test]
async fn sync_is_asked_again_after_an_already_syncing_refusal() -> Result<()> {
    ask_a_refusing_peer_three_times(
        AbortReason::AlreadySyncing,
        b"sync_is_asked_again_after_an_already_syncing_refusal",
    )
    .await
}

A possible fix

On RemoteAbort(AlreadySyncing), reset the peer's state if it is still the refused connect. If the state is Running { origin: Accept }, the concurrent incoming connection owns it, so leave it alone. With this, the test above passes 10 runs of 10, and so does the rest of tests/sync.rs. Queueing a resync, since the remote was busy and not uninterested, or emitting a SyncFinished with the error, might be wanted too.

Diff
diff --git a/src/engine/live.rs b/src/engine/live.rs
index db25dc3..ebb9d14 100644
--- a/src/engine/live.rs
+++ b/src/engine/live.rs
@@ -497,6 +497,7 @@ impl LiveActor {
         match result {
             Err(ConnectError::RemoteAbort(AbortReason::AlreadySyncing)) => {
                 debug!(?reason, "remote abort, already syncing");
+                self.state.abort_connect(&namespace, peer);
             }
             res => {
                 self.on_sync_finished(
diff --git a/src/engine/state.rs b/src/engine/state.rs
index 6e040d2..3150366 100644
--- a/src/engine/state.rs
+++ b/src/engine/state.rs
@@ -99,6 +99,15 @@ impl NamespaceStates {
         state.accept_request(me, &node)
     }
 
+    /// Forget a connect the remote refused, if it is still the running sync.
+    pub fn abort_connect(&mut self, namespace: &NamespaceId, node: EndpointId) {
+        if let Some(state) = self.entry(namespace, node) {
+            if matches!(state.state, SyncState::Running { origin: Origin::Connect(_), .. }) {
+                state.state = SyncState::Idle;
+            }
+        }
+    }
+
     /// Insert a finished sync operation into the state.
     ///
     /// Returns the time when the operation was started, and a `bool` that is true if another sync

Transport note

With noq 1.3.0, the close sequence after run_alice (net.rs:47-52) fails before the refusal is read. Every refusal, the control's included, is then reported as ConnectError::Close ("Failed to close connection1"), and the state is reset. So the wedge does not show there, but the refusal reason is lost instead. It reproduces with noq 1.0.0 (this repo's lockfile) and 1.1.1 (what iroh 1.0.3 resolves to).

Versions

iroh-docs 0.101.0 and main @ 8cfeacb; iroh 1.0.x, iroh-blobs 0.103, iroh-gossip 0.101. Reproduced on Linux.

Lingua principale
Rust
Stelle
81
Fork
19
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di n0-computer/iroh-docs

Tutte le issue di n0-computer/iroh-docs

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.