Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Bounded JOIN pipeline and differential paths report some budget denials as non-budget errors (follow-up to #2177)

Closed
  • Is something wrong?
#2,185 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
c
Domain
backend, databases

Research direction

Start from the budget-denial classification in wl_columnar_session_budget_denied() and the bounded JOIN paths it covers: the pipeline in wl_columnar_join_pipeline_try_eval, the non-differential wl_columnar_join_op, and wl_columnar_join_diff_op. Use the budget sweep bands in the table to find the denial sites that return ENOMEM or ENOSPC without setting the flag. Done means one fail-first test per row 1-8 that passes once the denial is reported as ENOSPC with the public check true, plus the listed guard tests staying green.

Written by the indexing model from the issue text.

Description

Summary

#2177 was fixed for the non-differential JOIN operator's bounded output
growth: that band is now reported as a budget denial. Several other
governor denials on the bounded JOIN paths still are not.

"Reported as a budget denial" here means what callers see:

  • the rc is ENOSPC; and
  • wl_columnar_session_budget_denied() returns true. This function backs
    wl_session_budget_denied and the public budget-error classification. It
    checks the session's memory_budget_denied, live relations' and
    differential arrangements' memory_budget_denial_pending, and the TDD
    workers.

These paths are reachable today by any user who sets
WIRELOG_JOIN_BATCH_BYTES together with a memory budget. That is the
documented way to run bounded joins. The fix does not depend on any future
automatic bounded mode, but such a mode would route more JOINs onto these
paths and must not land before it.

# Path Denied reservation (call site) rc Public check
1 JOIN pipeline (a JOIN consumed with its FILTER/MAP/probe tail) Producer scratch relation: col_join_batch_producer_create → governed $join_batch constructor (wl_columnar_relation_new_auto_governed) ENOSPC false
2 JOIN pipeline Pipeline output's governor attachment: col_rel_attach_memory_governor(out, …) in wl_columnar_join_pipeline_try_eval ENOSPC false
3 JOIN pipeline Pipeline output growth: pipeline_sink_reserve → col_join_batch_sink_grow returns RESERVATION_DENIED, sink->error stays 0, and the publish loop returns its default ENOMEM after destroying out ENOMEM false
4 Non-differential JOIN operator Producer scratch relation (same constructor as row 1, reached from wl_columnar_join_op) ENOSPC false
5 Non-differential JOIN operator Producer scratch column types: col_join_set_output_types → col_rel_set_column_types → metadata reservation, inside col_join_batch_producer_create ENOSPC false
6 Differential bounded JOIN (wl_columnar_join_diff_op, join_batch_bytes > 0) bounded_out column types: col_join_set_output_types(bounded_out, …) after wl_columnar_join_new_output ENOSPC false
7 Differential bounded JOIN Differential arrangement hash-table growth: index_right → col_diff_arrangement_ensure_ht_capacity ENOMEM false
8 Differential bounded JOIN bounded_out growth: col_join_batch_run_to_relation → col_join_batch_sink_grow; bounded_out is destroyed and wl_columnar_join_dispose_left sets no flag ENOSPC false (except a ~0.5 KiB band at its lower edge, where the check is true)

Not misreported to callers, so not in scope. The rc is ENOMEM but the
public check is true for the differential arrangement's creation and
deep-copy reservations in index_right
(col_diff_arrangement_create_with_memory_governor_status,
col_diff_arrangement_deep_copy_with_memory_governor_status). The errno
there disagrees with the classification. Normalising it to ENOSPC is
optional cleanup.

Reproduction

Each row was attributed with a backtrace at the governor's denial point,
resolved with addr2line, during a budget sweep. The bands were measured on
x86_64 at main plus PR #2183. Byte values depend on struct sizes, so tests
must derive budgets from measured admissions, not copy these numbers.

Fixture sketch (C, internal test API):

/* fresh session per budget; one left row (k=0) matching 4096 right rows */
sess = pipeline_session();                 /* or make_mock_session() */
sess->join_batch_bytes = 224;              /* 7 rows per batch, 4 cols */
add right(k, r) with 4096 rows, all k = 0; build its arrangement;
gov = enforcing governor with budget_bytes == usable_bytes == B;
sess->memory_governor = gov;

/* pipeline:  JOIN(k) -> MAP(project 1,3)   via col_eval_relation_plan()  */
/* operator:  JOIN(k) alone                 via col_eval_relation_plan()  */
/* diff:      col_op_join_diff() with delta_mode WL_DELTA_FORCE_FULL      */
rc  = ...;
pub = wl_columnar_session_budget_denied(sess);   /* before teardown */

Bands observed (budget B, rc, public check):

Path Budget band rc Public check Row
Pipeline 1-3 KiB ENOSPC false 1
Pipeline 4-5 KiB ENOSPC false 2
Pipeline 6 KiB ENOSPC true MAP output; correct
Pipeline 7-100 KiB ENOMEM false 3
Pipeline 101 KiB OK –
Operator 1-3 KiB ENOSPC true Output creation; correct
Operator 4-6 KiB ENOSPC false 4
Operator 7 KiB ENOSPC false 5
Operator 8-196 KiB ENOSPC true Output growth; fixed by #2177
Operator 197 KiB OK –
Differential 512-3583 B ENOSPC true Output creation; correct
Differential 3584-4095 B ENOSPC false 6
Differential 4096-20991 B ENOMEM true Arrangement create/copy; errno only
Differential 20992-70143 B ENOMEM false 7
Differential 70144-70655 B ENOSPC true
Differential 70656-251391 B ENOSPC false 8
Differential 251392 B OK –

Expected

Every governor denial on these paths returns ENOSPC, and
wl_columnar_session_budget_denied() returns true. A non-budget failure
stays unclassified: for example, a sink resize-preparation failure, as in
test_join_batch_resume's resize-preparation test.

Acceptance

  • One fail-first test per row 1-8. Each sets one budget inside the band,
    derived from measured admissions, and asserts ENOSPC and
    wl_columnar_session_budget_denied(). Each fails on main.
  • Guard: the resize-preparation (non-budget) sink failure stays
    unclassified.
  • Guard: bands already reported correctly stay correct. These are the
    operator output growth (#2177's test), the output creations, and the MAP
    output.
  • Optional: the index_right arrangement create/copy denials return
    ENOSPC instead of ENOMEM.

Context

Found while planning #1370 (d). Related: #2177 (closed; operator output
growth) and #2182 (CONSOLIDATE k-way scratch, the same class of misreport
on another operator).

Dominant language
C
Stars
18
Forks
2
Avg merge
6h 19m
Merged PRs (30d)
280

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from semantic-reasoning/wirelog

All issues in semantic-reasoning/wirelog

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.