Name wedged flushers in shutdown warnings and pin the shutdown order on the batch writer

Open
  • Is something wrong?
#980 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
64/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
clickhouse, java
Domain
backend, databases, testing

Research direction

Start at BatchingFlowRepository.stop() and trace the flusher lifecycle, shutdown warnings, and queue sweep ordering. Find the validation and documentation for riptide.clickhouse.batch.flushers, then run the existing batch-writer tests. Done means warnings report live flushers out of the total, the cap error explains the valid range and connection headroom, and tests prove the sweep waits for flusher shutdown.

Written by the indexing model from the issue text.

Description

enhancement priority: low

Problem statement

Three small gaps were left in the batch writer after #976 added several flushers. An operator whose shutdown hits the grace period reads Batch flusher did not drain within ... without learning whether one or all of the N flushers wedged. The cap of 8 is explained as leaving connections for "schema and drift queries", but those run in delegate.start() before any flusher starts, and no test pins that the queue sweep runs only after every flusher has ended.

User story

  • As an operator running riptide with several batch flushers,
  • I want to read from the shutdown log how many flushers wedged, and trust the cap's stated reason,
  • so that I can tell one stuck insert from a stuck ClickHouse.

Proposed solution

  • Both shutdown warnings in BatchingFlowRepository.stop() name how many of the N flushers were still alive.
  • The riptide.clickhouse.batch.flushers error text and docs give a reason for the cap that holds at runtime (headroom in the client's 10-connection default pool, including dead-letter writes).
  • A test asserts the sweep never runs while a flusher is alive.

Alternatives considered

  • Do nothing: shutdown logs stay ambiguous for N > 1, and the cap's reason stays slightly wrong; no rows are lost either way.

Acceptance criteria

Scenario: one of four flushers wedges

  • Given four flushers, one blocked in an insert that ignores interrupts,
  • When stop() runs past the grace period,
  • Then the warning names 1 of 4 flushers still alive.

Scenario: cap error text

  • Given riptide.clickhouse.batch.flushers=9,
  • When riptide starts,
  • Then startup fails with a message naming the key, the value, the range 1 to 8, and a reason that does not cite startup-only queries.

Scenario: sweep after join

  • Given two flushers and a delegate that records when the sweep's insert happens,
  • When stop() runs,
  • Then no sweep insert starts before both flusher threads have ended or been given their unwind window.

Business impact & priority

  • Value: faster shutdown diagnosis for operators running more than one flusher
  • Frequency: Unknown, only on shutdowns that hit the grace period
  • Target metric: Unknown
  • Priority: P4, cosmetic and test hygiene, no data loss

Open questions

  • Are the two log lines pinned by any operator alerting or log-based dashboards that a wording change would break?
Dominant language
Java
Stars
2
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Riptide-Labs/riptide

All issues in Riptide-Labs/riptide

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.