Flock 0.8.0 group-file race crashes a subscriber during concurrent joins (push_fanout / detector_narrowing benchmarks)

Open
#1 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
15/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
c
Domain
backend, testing

Research direction

CI is already building Flock from commit 7b484c5 via 02c28cf; start by checking that CI pin and the na+sm runs for push_fanout_benchmark and detector_narrowing_benchmark. Done means several green runs with both benchmarks passing, then CI can move to a Flock release newer than 0.8.0; if failures persist, symbolize the crash with debug info as the issue describes.

Written by the indexing model from the issue text.

Description

Symptom

push_fanout_benchmark and detector_narrowing_benchmark intermittently fail in the na+sm CI job. The writer reports subscriber N never signaled ready, with a different N each time:

Run Mercury Case Subscriber
35791025169 2.3.1 push_fanout, 4 subscribers 2
35791025169 2.3.1 detector_narrowing 8
35974637262 2.4.1 push_fanout, 8 subscribers 4

Cause

That subscriber didn't time out. It crashed while joining. Both push_fanout logs show a SIGSEGV in an Argobots thread inside libflock-server.so, at the same offset (+0x471d) under both Mercury versions:

b_push_fanout:45640 terminated with signal 11 ...
/home/runner/mochi-install/lib/libflock-server.so.0(+0x471d)
/home/runner/mochi-install/lib/libflock-server.so.0(+0xe14f)
...
/home/runner/mochi-install/lib/libabt.so.1(+0x1a6bc)

The 2.4.1 run also shows the writer failing to update the group file several times while subscribers join:

[error] [flock] Could not rename file b_push_fanout_4.h5.vsgroup.swp into b_push_fanout_4.h5.vsgroup: No such file or directory

This is a Flock 0.8.0 bug. On every membership change, the writer's provider rewrites the group file (dispatch_member_update -> serialize_view_to_file, src/provider.c), and nothing serializes those rewrites:

  • file_serializer() (src/view-serialize.c) always writes to the same <file>.swp and then renames it. When two updates run at once, one rename moves the other's half-written file into place (a torn group file), and the other rename fails with ENOENT.
  • flock_group_view_serialize() walks v->members without the view lock, and the SWIM gossip handler can reallocate that array (flock_group_view_add_member) at the same moment.

The ENOENT rename errors prove that two updates really do run concurrently in one process. The crash matches unlocked access to the member array, but I haven't mapped +0x471d to a symbol, so that part is inferred, not confirmed.

vol-stream already works around the read side of the same problem (tr_mercury.c, vs_tr_reader_join_group(): a retry because flock_group_view_from_file() ignores fread()'s return value).

Fix

Upstream fixed all three points in mochi-flock 7b484c5 ("better access to group files", 2026-08-28): an mkstemp temp file, the view lock held while serializing, and a checked fread. It's the only commit after v0.8.0 and hasn't been released yet.

02c28cf builds Flock from that commit in CI.

To close this

  • Several green na+sm runs with both benchmarks passing.
  • Move CI back to a release tag once Flock ships one after 0.8.0, and consider raising vol-stream's flock >= 0.8.0 floor, because users building against the 0.8.0 release will still hit this.
  • If it still happens with the pin, symbolize the crash (build Flock with debug info and resolve the offset) before looking elsewhere.

My first version of this issue blamed a tight 10-second wait in the test and CPU contention on the runner. That was wrong: I had only read the writer's warnings, not the subscriber's crash.

Dominant language
C
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.