Central new-partition zero-progress seed can overwrite another process's committed progress and resend rows

Open
  • Is something wrong?
#2,786 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, node.js
Domain
backend, databases

Research direction

Start with hypaware-core/plugins-workspace/central/src/sink.js, especially ensureOpenDatasetPartition and withDatasetRolloutLock, then trace the watermark file operations and upload claim used for progress commits. Reproduce the stated interleaving with the deterministic probe described in the issue. Done means zero-progress seeding cannot overwrite progress committed by another process, and the regression case no longer resends row 1.

Written by the indexing model from the issue text.

Description

Found in review of #2785. This is pre-existing: it reproduces on master before that PR (7973e82e) and on the PR head.

When a central sink sees a new partition of an open dataset, ensureOpenDatasetPartition in hypaware-core/plugins-workspace/central/src/sink.js reads the partition's progress and, if none exists, writes zero progress. The read and the write are separate steps, and the only lock around them is in-process (withDatasetRolloutLock). A foreground hyp sync and the daemon run in separate processes, so neither excludes the other.

Interleaving that resends rows:

  1. Process A reads progress for the new partition: absent. A pauses.
  2. Process B reads absent, writes zero, uploads row 1 and commits progress seq 1.
  3. Process A writes zero over B's committed seq 1.
  4. A's upload reloads seq 0 and sends rows 1 and 2, so row 1 goes to central twice.

The #2785 upload claim does not cover this: rollout preparation runs before the claim is taken, so the claimed pass faithfully reloads the stale zero.

Reproduction: a deterministic probe runs two independent sink instances, each with its own per-runtime rollout map, sharing real watermark files, and pauses one instance's progress read. That models the cross-process interleaving without spawning two OS processes. It reads 0 then 0, posts row1, then row1+row2, final seq 2, with the same result at the PR head and at the parent. Whether central stores the resent row twice was not tested.

Expected: seeding zero progress must never overwrite progress another process has committed. Two candidate fixes: write the seed create-only (fail or no-op if a progress file already exists), or run the new-partition seed inside the same cross-process upload claim.

🤖 Generated with Claude Code

Dominant language
JavaScript
Stars
11
Forks
0
Avg merge
2h 55m
Merged PRs (30d)
294

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hyparam/hypaware

All issues in hyparam/hypaware

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.