Asset hot reload can deadlock when the Asset Processor reprocesses an asset the asset server is reloading

Aperta
  • Qualcosa non va?
#26,407 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
data

Direzione di ricerca

The deadlock involves get_meta_loader_and_reader in crates/bevy_asset/src/server/mod.rs and the gated readers in crates/bevy_asset/src/io/processor_gated.rs, where read and read_meta each take a read guard on the file_transaction_lock. The processor's write_arc().await in crates/bevy_asset/src/processor/mod.rs is the writer that blocks. Start by reproducing the race with a test that changes a source file right after its first load, then check whether the second read guard can be avoided. Done means the asset reloads reliably in that timing window without the 1 s workaround.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

A-Assets C-Bug D-Async D-Modest S-Ready-For-Implementation X-Uncontroversial

Bevy version

0.20.0, and the code in question is unchanged on main (ae6152152d).

Relevant system information

Windows 11, Rust stable. The race is in platform-independent code.

What you did

An App with AssetPlugin { mode: AssetMode::Processed, use_asset_processor_override: Some(true), watch_for_changes_override: Some(true), .. }, so the Asset Processor runs inside the app and both the source folder and the processed folder are watched. Load an asset, then change its source file shortly after the first processing has finished.

What went wrong

About 2% of the time the changed asset never reloads, nothing is logged, and that asset never reloads again until restart. Other assets keep hot reloading.

It is a deadlock on that asset's file_transaction_lock, an async_lock::RwLock. async_lock holds new readers back while a writer waits:

  1. The asset server reloads the asset. get_meta_loader_and_reader keeps the meta reader alive while it opens the asset reader (L1477), so that the asset reader can "take over" the lock. Through ProcessorGatedReader, each of the two readers takes its own read guard (read, read_meta). So the server holds one read guard and asks for a second one.
  2. In between, the processor starts processing the same asset again and waits for the write guard (write_arc().await).
  3. The second read waits behind the waiting writer, and the writer waits for the first read guard, which is only dropped after the second read. Neither proceeds.

What makes the two meet: about 300 ms (the debounce) after the first processing, the processed folder's watcher reports every file the processor wrote, and the asset server reloads them. A source change in that window is reported by the source watcher at about the same instant, so the processor reprocesses the file while the server reloads it.

A trace (bevy_asset at TRACE) of a stuck run shows, for the stuck asset, all at the same millisecond:

DEBUG bevy_asset::processor: Processing graphs\restart.graph.ron
TRACE bevy_asset::io::processor_gated: Processing finished with Processed, reading meta for graphs/restart.graph.ron
TRACE bevy_asset::io::processor_gated: Processing finished with graphs/restart.graph.ron, reading Processed

After that there is no Finished processing, Skipping processing or Reloaded line for that asset, while the other assets reload normally. The second processor_gated line is logged just before get_transaction_lock, where the read blocks.

Additional information

  • How often: with a test that changes a source right after its first load, 9 failures in 400 runs. With the app updated for 1 s after the load before the change (past the debounce), 0 in 400. We use that wait as a workaround in our tests.
  • We don't have a standalone minimal repro yet. The setup above is the whole recipe; the timing window is narrow.
  • Possible fixes: let the asset reader reuse the meta reader's guard instead of taking a second one (for example by passing the guard through, or by taking one guard per load in the gated reader), or take both readers under a single guard. Any change that avoids a recursive read acquisition on a lock that prefers writers would do.
Lingua principale
Rust
Stelle
49.7k
Fork
5k
Merge medio
3g 19h
PR unite (30g)
134

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di bevyengine/bevy

Tutte le issue di bevyengine/bevy

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.