Sync with a peer stays wedged after one AlreadySyncing refusal (local state never leaves Running)
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- Mezza giornata
- Idoneità per principianti
- 62/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- rust
- Ambito
- distributed-systems, networking
Direzione di ricerca
Start with on_sync_via_connect_finished in src/engine/live.rs (around line 498), where AlreadySyncing is only logged, then read the sync state helpers in src/engine/state.rs. The failing test is sync_is_asked_again_after_an_already_syncing_refusal in tests/sync.rs; run cargo test --test sync sync_is_asked_again. Done when that test passes and the rest of tests/sync.rs still passes.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
Disclaimer - this issue was mostly prepared by AI - Claude Code with Opus 5.5 - under my supervision - it's an issue that was revealed in my project in tests - from what Claude has described looks like real problem.
After a single AbortReason::AlreadySyncing refusal from a peer, a node never syncs with that peer again until it restarts.
on_sync_via_connect_finished (src/engine/live.rs:498) only logs the refusal. It never calls on_sync_finished, so state.finish is never called, and the local PeerState for that peer stays SyncState::Running { origin: Connect(..) } for good. As a result:
- every later
start_connectfor the peer is dropped asabort connect: sync already running(src/engine/state.rs:196-205). That includes an explicitstart_sync, aNewNeighbor, and aSyncReport. ASyncReportsetsresync_requested, but thefinishthat would act on it never comes; - no
SyncFinishedevent is emitted for that round, so a subscriber cannot tell anything went wrong; - an incoming request from the peer is accepted only when
expected_sync_directionpicksAcceptfor the pair. Otherwise it is refused withAlreadySyncingtoo (state.rs:224-225), and the pair stays stuck in both directions.
How the refusal happens in practice
The remote is still finishing the previous round. When the accepting side has received entries, it broadcasts a SyncReport to its neighbours before it calls state.finish (live.rs:561-593). That leaves a window of tens of milliseconds in which its state is still Running { origin: Accept }. A round that the connecting side asks for as soon as it sees SyncFinished lands in that window and is refused with AlreadySyncing (state.rs:219).
The early return at live.rs:498 looks written for the simultaneous-dial case, where the other connection finishes the state. In the Origin::Accept case there is no other connection, so nothing ever resets the state.
We hit this in an application that asks for one more round with a peer once a round ends: a restarted node never caught up, in about 1 run in 7–20. As a workaround we now wait a second before asking.
Failing test
Branch: izderadicka/iroh-docs@test/already-syncing-wedge (one commit on main @ 8cfeacb, tests/sync.rs only). The test uses the repo's own lockfile, so no version pins are involved.
- The peer refuses every request through iroh-docs' own
net::handle_connection, with a chosenAbortReason. This makes the refusal deterministic. - A real node asks it for a sync three times. Each ask is repeated every 100ms until the peer sees it.
sync_is_asked_again_after_a_refusalis the control: it refuses withInternalServerError, and every ask reaches the peer.
$ cargo test --test sync sync_is_asked_again
test sync_is_asked_again_after_a_refusal ... ok
test sync_is_asked_again_after_an_already_syncing_refusal ... FAILED
Error: ask 2: the peer was not asked again within 10s
It fails at ask 2 every time: the first ask reaches the peer, and nothing does after that.
The test
/// A peer that refuses every sync request with `reason`, and says each time it
/// is asked.
#[derive(Debug, Clone)]
struct Refusing {
sync: SyncHandle,
reason: AbortReason,
asked: mpsc::UnboundedSender<()>,
}
impl ProtocolHandler for Refusing {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (reason, asked) = (self.reason, self.asked.clone());
let open = connection.clone();
handle_connection(
self.sync.clone(),
connection,
move |_, _| {
asked.send(()).ok();
async move { AcceptOutcome::Reject(reason) }
},
None,
)
.await
.ok();
// As a real peer does: the connection stays up until the dialer is done.
open.closed().await;
Ok(())
}
}
/// Asks a peer that refuses with `reason` for a sync three times. Each ask is
/// repeated every 100ms until the peer sees it, so a round still finishing
/// on our side does not count against us.
async fn ask_a_refusing_peer_three_times(reason: AbortReason, seed: &[u8]) -> Result<()> {
const ASK_TIMEOUT: Duration = Duration::from_secs(10);
let mut rng = test_rng(seed);
let node = spawn_node(0, &mut rng).await?;
let doc = node.client().docs().create().await?;
let (asked, mut was_asked) = mpsc::unbounded_channel();
let peer = empty_endpoint().await?;
let _router = Router::builder(peer.clone())
.accept(
iroh_docs::ALPN,
Refusing {
sync: SyncHandle::spawn(Store::memory(), None, "refusing".to_string()),
reason,
asked,
},
)
.spawn();
for attempt in 1..=3 {
n0_future::time::timeout(ASK_TIMEOUT, async {
loop {
doc.start_sync(vec![peer.addr()]).await?;
tokio::select! {
Some(()) = was_asked.recv() => return anyhow::Ok(()),
_ = n0_future::time::sleep(Duration::from_millis(100)) => {}
}
}
})
.await
.with_context(|| {
format!("ask {attempt}: the peer was not asked again within {ASK_TIMEOUT:?}")
})??;
}
node.shutdown().await?;
Ok(())
}
/// A refused sync is asked for again: the control for the test below.
#[tokio::test]
#[traced_test]
async fn sync_is_asked_again_after_a_refusal() -> Result<()> {
ask_a_refusing_peer_three_times(
AbortReason::InternalServerError,
b"sync_is_asked_again_after_a_refusal",
)
.await
}
/// The same after an `AlreadySyncing` refusal, which a peer gives while it is
/// still finishing the round before.
///
/// Fails today: `on_sync_via_connect_finished` only logs that refusal, so the
/// local state for the peer stays `Running` and every later `start_sync`,
/// `NewNeighbor` or `SyncReport` for it is dropped as "sync already running".
#[tokio::test]
#[traced_test]
async fn sync_is_asked_again_after_an_already_syncing_refusal() -> Result<()> {
ask_a_refusing_peer_three_times(
AbortReason::AlreadySyncing,
b"sync_is_asked_again_after_an_already_syncing_refusal",
)
.await
}
A possible fix
On RemoteAbort(AlreadySyncing), reset the peer's state if it is still the refused connect. If the state is Running { origin: Accept }, the concurrent incoming connection owns it, so leave it alone. With this, the test above passes 10 runs of 10, and so does the rest of tests/sync.rs. Queueing a resync, since the remote was busy and not uninterested, or emitting a SyncFinished with the error, might be wanted too.
Diff
diff --git a/src/engine/live.rs b/src/engine/live.rs
index db25dc3..ebb9d14 100644
--- a/src/engine/live.rs
+++ b/src/engine/live.rs
@@ -497,6 +497,7 @@ impl LiveActor {
match result {
Err(ConnectError::RemoteAbort(AbortReason::AlreadySyncing)) => {
debug!(?reason, "remote abort, already syncing");
+ self.state.abort_connect(&namespace, peer);
}
res => {
self.on_sync_finished(
diff --git a/src/engine/state.rs b/src/engine/state.rs
index 6e040d2..3150366 100644
--- a/src/engine/state.rs
+++ b/src/engine/state.rs
@@ -99,6 +99,15 @@ impl NamespaceStates {
state.accept_request(me, &node)
}
+ /// Forget a connect the remote refused, if it is still the running sync.
+ pub fn abort_connect(&mut self, namespace: &NamespaceId, node: EndpointId) {
+ if let Some(state) = self.entry(namespace, node) {
+ if matches!(state.state, SyncState::Running { origin: Origin::Connect(_), .. }) {
+ state.state = SyncState::Idle;
+ }
+ }
+ }
+
/// Insert a finished sync operation into the state.
///
/// Returns the time when the operation was started, and a `bool` that is true if another sync
Transport note
With noq 1.3.0, the close sequence after run_alice (net.rs:47-52) fails before the refusal is read. Every refusal, the control's included, is then reported as ConnectError::Close ("Failed to close connection1"), and the state is reset. So the wedge does not show there, but the refusal reason is lost instead. It reproduces with noq 1.0.0 (this repo's lockfile) and 1.1.1 (what iroh 1.0.3 resolves to).
Versions
iroh-docs 0.101.0 and main @ 8cfeacb; iroh 1.0.x, iroh-blobs 0.103, iroh-gossip 0.101. Reproduced on Linux.
- Lingua principale
- Rust
- Stelle
- 81
- Fork
- 19
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Nessuna guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di n0-computer/iroh-docs
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 52/100
n0-computer/iroh-docs#124 ·
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
n0-computer/iroh-docs#122 ·
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 58/100
n0-computer/iroh-docs#120 ·
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
n0-computer/iroh-docs#109 ·
-
Sync with other peers never re-downloads blobsForse già presa @cbenhagen l’ha presa 235 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 38/100
n0-computer/iroh-docs#82 · 2 commenti · 3 reazioni ·
Tutte le issue di n0-computer/iroh-docs
Issue simili
-
area:web kind:bug priority:p2 ready
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
sims1253/ry#931 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 84/100
sims1253/r-typeshed#140 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
storytold/craft-libs#2867 ·
I maintainer di solito rispondono entro 1 giorno
-
help wanted needs investigation priority: low
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100