Flock 0.8.0 group-file race crashes a subscriber during concurrent joins (push_fanout / detector_narrowing benchmarks)

Aperta
#1 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
1/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
c
Ambito
backend, testing

Direzione di ricerca

CI is already building Flock from commit 7b484c5 via 02c28cf; start by checking that CI pin and the na+sm runs for push_fanout_benchmark and detector_narrowing_benchmark. Done means several green runs with both benchmarks passing, then CI can move to a Flock release newer than 0.8.0; if failures persist, symbolize the crash with debug info as the issue describes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Symptom

push_fanout_benchmark and detector_narrowing_benchmark intermittently fail in the na+sm CI job. The writer reports subscriber N never signaled ready, with a different N each time:

Run Mercury Case Subscriber
35791025169 2.3.1 push_fanout, 4 subscribers 2
35791025169 2.3.1 detector_narrowing 8
35974637262 2.4.1 push_fanout, 8 subscribers 4

Cause

That subscriber didn't time out. It crashed while joining. Both push_fanout logs show a SIGSEGV in an Argobots thread inside libflock-server.so, at the same offset (+0x471d) under both Mercury versions:

b_push_fanout:45640 terminated with signal 11 ...
/home/runner/mochi-install/lib/libflock-server.so.0(+0x471d)
/home/runner/mochi-install/lib/libflock-server.so.0(+0xe14f)
...
/home/runner/mochi-install/lib/libabt.so.1(+0x1a6bc)

The 2.4.1 run also shows the writer failing to update the group file several times while subscribers join:

[error] [flock] Could not rename file b_push_fanout_4.h5.vsgroup.swp into b_push_fanout_4.h5.vsgroup: No such file or directory

This is a Flock 0.8.0 bug. On every membership change, the writer's provider rewrites the group file (dispatch_member_update -> serialize_view_to_file, src/provider.c), and nothing serializes those rewrites:

  • file_serializer() (src/view-serialize.c) always writes to the same <file>.swp and then renames it. When two updates run at once, one rename moves the other's half-written file into place (a torn group file), and the other rename fails with ENOENT.
  • flock_group_view_serialize() walks v->members without the view lock, and the SWIM gossip handler can reallocate that array (flock_group_view_add_member) at the same moment.

The ENOENT rename errors prove that two updates really do run concurrently in one process. The crash matches unlocked access to the member array, but I haven't mapped +0x471d to a symbol, so that part is inferred, not confirmed.

vol-stream already works around the read side of the same problem (tr_mercury.c, vs_tr_reader_join_group(): a retry because flock_group_view_from_file() ignores fread()'s return value).

Fix

Upstream fixed all three points in mochi-flock 7b484c5 ("better access to group files", 2026-08-28): an mkstemp temp file, the view lock held while serializing, and a checked fread. It's the only commit after v0.8.0 and hasn't been released yet.

02c28cf builds Flock from that commit in CI.

To close this

  • Several green na+sm runs with both benchmarks passing.
  • Move CI back to a release tag once Flock ships one after 0.8.0, and consider raising vol-stream's flock >= 0.8.0 floor, because users building against the 0.8.0 release will still hit this.
  • If it still happens with the pin, symbolize the crash (build Flock with debug info and resolve the offset) before looking elsewhere.

My first version of this issue blamed a tight 10-second wait in the test and CPU contention on the runner. That was wrong: I had only read the writer's warnings, not the subscriber's crash.

Lingua principale
C
Stelle
0
Fork
0
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.