Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Automatic bounded JOIN mode under memory pressure: measured not to help yet; design recorded for when it would

Aperta
  • Qualcosa non va?
#2,186 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
c
Ambito
backend, performance

Direzione di ricerca

This is a recorded design, not a task: the measurements say automatic bounded JOIN mode is not implemented and should not be built yet. Read the trigger rule b and the latch design (sess->join_batch_effective, set in eval_plan's JOIN case), plus the WIRELOG_JOIN_BATCH_BYTES parsing, to understand the proposal. Nothing here defines a done state for a newcomer; wait for a maintainer to reopen it with a concrete change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Bounded JOIN production (WIRELOG_JOIN_BATCH_BYTES, #1370) is opt-in.
This issue records:

  • a design for turning it on automatically when an enforcing memory budget
    comes under pressure;
  • why it was not implemented: on every measured pressured workload it
    would not have changed the outcome;
  • the conditions under which it should be revisited.

The open question is not the threshold value. The measured budget
failures arrive as single large requests inside one operator, at low
occupancy:

  • CSPA's own JOIN outputs;
  • DOOP's 80.5 GB CONSOLIDATE scratch request (#2182).

No dispatch-time occupancy trigger, at any threshold, can see such a
request coming. Tuning T will not change that.

Measurements

All on x86_64, main plus PR #2183, release builds. Each run is a single
process with a memory budget, through WIRELOG_MEMORY_BUDGET or a cgroup.

Fixed reservations
  • Every session reserves about 128.2 MiB at creation (eval arena plus
    delta pool): measured 134,418,560-134,419,566 B at the end of session
    creation.
  • At the 256 MiB minimum budget that is over 50% of usable before any data.
A shadow census of three trigger rules (logged at every JOIN dispatch; behaviour unchanged)

The rules:

  • old: reserved ≥ 50% of usable;
  • a: headroom < max(16 MiB, 25% of usable);
  • b: (reserved − session-start baseline) ≥ 50% of (usable − baseline).
Run JOIN dispatches old a b
tc, cspa, galen, polonius, andersen, crdt-small at 256 MiB 100-1851 each every dispatch 0 0
The same at 512 MiB and 1 GiB – 0 0 0
DOOP (zxing) under a 90 GiB cgroup, until it stopped at 866 s on #2182 52,638 0 0 0

For DOOP, the highest (reserved − baseline)/(usable − baseline) at any JOIN
dispatch was 0.365 (35.26 GB). The run then failed in one CONSOLIDATE
scratch request of 80.5 GB.

Does bounded mode rescue a failing run?

CSPA:

Budget Result
256, 384, 512, 768, 832, 896, 960 MiB FAIL with the knob off, at 1 MiB, and at 64 KiB where tried
1 GiB OK in both modes (20,381 tuples)

Bounded mode does not move the minimum budget. The pressure is the JOIN
outputs themselves, which the relation sink must materialise, so
retry-once-in-bounded-mode would not help either.

Cost of bounded mode where it would engage

Budget pinned at 64 GiB in both arms, 5 interleaved rounds, governor
schedutil (indicative):

Workload Change Note
Polonius +13.2% (7.6 → 8.6 ms)
Galen −0.5%
CSPA −24.4% Result digest identical; cause not established

An always-on mode would cost small workloads more than #1370's 3%
criterion allows.

Design to use if this is revisited

  1. Trigger: rule b. Measure (reserved − baseline) against (usable −
    baseline).

    • The baseline is the reserved bytes at the end of session creation,
      stored on the session.
    • No baseline (test mock sessions) means automatic mode is off.
    • Do not trigger on raw occupancy; at small budgets the fixed arenas
      alone exceed 50%.

  2. Latch: decide the effective batch size once per JOIN dispatch.

    • Set sess->join_batch_effective in eval_plan's JOIN case before the
      pipeline block, and clear it after the operator.
    • Every reader uses it: eval_plan's pipeline gate; the pipeline
      preflight and its producer call; bounded-mode eligibility; the
      operator's bounded check and producer call; the differential gate,
      condition and producer call.
    • The TDD bound-run refusal, the insertion writer and the env parser
      stay raw.
    • Re-reading a live helper mid-JOIN can flip the mode after the output
      and arrangement were admitted. The differential producer returns
      EINVAL for a zero batch size.

  3. Configuration:

    WIRELOG_JOIN_BATCH_BYTES Meaning
    unset automatic allowed
    0 off
    a positive integer that size
    unparsable off, with a warning

    Strict mode stays explicit-only.

  4. Exclusions:

    • worker sessions (coordinator);
    • TDD bound runs (tdd_input_run);
    • sessions with join_output_shared_count set. That is where the
      shared row cap counts calls, and bounded sinks call it once per batch
      (#2174).

    owns_evaluation_control is not a usable discriminator: it is
    false for every session created without an evaluation control.

  5. Recording: an internal count of automatic dispatches. No per-JOIN
    warnings.

  6. Tests:

    • an explicit test-built governor;
    • a test-only threshold hook instead of a release env var;
    • a guard: zero automatic dispatches at the 256 MiB minimum budget with
      a small program;
    • a clone-sharing-cap case that asserts the clones ran;
    • a mutation that re-reads the helper mid-JOIN must fail a test.

  7. Claim scope: automatic mode would bound the producer and its consumed
    pipeline, not a relation sink's output.

Revisit when

  • P1: a workload exists where rule b fires before a JOIN-path budget
    denial and the explicit knob changes that run's outcome (fail → OK)
    or its peak by ≥ 10%.
  • P2: reclaim (#1371) or spill (#1372) makes reservations fall back
    mid-run, so switching modes per JOIN becomes useful.
  • P3: bounded mode reaches worker sessions (#1373/#1383).

Prerequisite

The bounded paths must report budget denials correctly first: the #2177
follow-up and #2182.

Context

Split from #1370 (d), which delivers bounded JOIN production behind the
explicit knob.

Lingua principale
C
Stelle
18
Fork
2
Merge medio
7h 53m
PR unite (30g)
300

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di semantic-reasoning/wirelog

Tutte le issue di semantic-reasoning/wirelog

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.