Builds randomly fail on Apple silicon using bind mounts with VirtioFS enabled

The report names no repository file or test to modify. First reproduce the failure with the provided docker run command and repeated cargo build loop, comparing VirtioFS with gRPC FUSE and a non-bind-mounted directory; done would require a clearly identified repository-side fix and a regression check.

I have an issue on Apple Silicon where building Rust projects will randomly fail in bind-mounted volumes when VirtioFS is enabled in Docker.

I've found it pretty easy to reproduce. First make sure VirtioFS is enabled on the Docker engine and run the container:

docker run -itv /tmp/failing-builds:/failing-builds rust

Inside the container, if you try building in the bind-mounted volume, builds will randomly fail. In this example, the first build succeeds and the second fails. I'm making sure to rm -fr target/ every time to make sure the build actually happens.

$ cd /failing-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
   Compiling failing-builds v0.1.0 (/failing-builds)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "/tmp/rustcaNiWwj/symbols.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.iyl3saptcr8eyus.rcgu.o" "-Wl,--as-needed" "-L" "/failing-builds/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-1af56df70101340f.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-2d201e9ec773e2b2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-11edaa83ea3f42ca.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-be5d62f450379333.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-48892539e9a2c4de.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-f8eb1df1397eacbf.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-2e09868052a8b2d5.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-a7e259ed513c618c.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-791329d15b754d4d.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-00676b12b0ca4248.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-77a22a58722b5fb2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-e9ab6f5f15790e88.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-fab82cfa2048cd45.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-21f6d61e1c83d48b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-bac7f74953c2c715.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-9b674ebb751cb2fa.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-088fae249a1081ae.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-29f211c7751ad68b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-26783adfa77949da.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o: No such file or directory
          collect2: error: ld returned 1 exit status
          

error: could not compile `failing-builds` (bin "failing-builds") due to previous error

If you try in a normal directory, not a bind-mounted volume, building works fine.

$ mkdir ~/working-builds
$ cd ~/working-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
~/working-builds# cargo init
     Created binary (application) package
root@90751c34564f:~/working-builds# while rm -fr target/ && cargo build; do :; done
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
^C

And if you disable VirtioFS and use the old gRPC FUSE file sharing implementation in Docker, building in the bind-mounted directory then works fine as well although much slower, compare 0.94s here to 0.31s from earlier. 🙁

$ cd /failing-builds
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
^C

Since this only fails with VirtioFS enabled, I'm guessing it's an issue on their end. I can open an issue on their repo but I figured it might be useful to have this issue tracked here anyways until VirtioFS fixes it.

Also if there's any more Rust/Linux debugging information I could collect and provide to them, or other things I could try before opening the issue there, that would be great.

(edit) Reached out to VirtioFS people and they said it's something to do with MacOS's VIRTIO implementation. I tried submitting something via Mac's feedback assistant back in October, but it hasn't gone anywhere.