Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Expose the row address prefilter (Scanner::with_row_addr_prefilter) in the C/C++ API

Open
  • Is something wrong?
#95 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
c, cpp, rust
Domain
api, search

Research direction

Start with the rewrite_prepared_fts_plan path, where MatchQueryExec and PhraseQueryExec are rebuilt and the external_mask is dropped, and with PreparedScalarSegment::configure, which overwrites the prefilter. Decide between the row-id array and serialized treemap setter shapes, then add the LanceScanner setter in lance-c and its C++ wrapper. Done means the allowlist and blocklist restrict plain, prepared FTS and nearest-neighbor results with scores unchanged, and the new tests cover a multi-fragment dataset with stable row ids.

Written by the indexing model from the issue text.

Description

Motivation

We use lance-c as the storage and search backend of a C++ search service. Several of our queries need to restrict a full-text or vector search to a set of rows we already know, before ranking:

  • Compound FTS. Without a Boolean query binding (#35), we compose MUST/SHOULD sub-queries from single-column Match scans. The MUST scans usually match a few hundred rows. A SHOULD scan with common terms, though, scores every row that matches any of its terms (up to 1.7M rows on our data) unless we can restrict it to the MUST candidates.
  • Externally computed pre-filters. Some row sets come from our own in-memory bitmaps rather than from a filter expression Lance can evaluate.

Lance core already supports this through Scanner::with_row_addr_prefilter(RowAddrMask), and pylance exposes it as scanner(row_addr_allowlist=..., row_addr_blocklist=...). lance-c has no equivalent. The only option we have is a _rowid IN (...) filter (SQL, or via lance_scanner_additional_sql_filter). Lance evaluates that with a full _rowid read plus an IN-list parse, so it is slow even for a few ids.

Measured with pylance 11 on a 2.48M-row, 3-fragment dataset (stable row ids, 2 GB inverted index), restricting an OR Match scan to the candidate rows of a preceding AND Match scan (290 real queries, warm cache):

SHOULD scan p50 p90 p99
unrestricted 48.5 ms 523 ms 1231 ms
_rowid IN (...) + prefilter 34.4 ms 61 ms 179 ms
row_addr_allowlist 1.4 ms 7.5 ms 32 ms

_rowid IN costs ~25–35 ms even for 3–5 ids. For a 1.5M-id set, planning the IN list alone took ~3 s and a filtered nearest search ~6.7 s, against 43 ms with row_addr_allowlist. Both restricted variants returned results identical to the unrestricted scan (same rows, same BM25 scores).

Requested API

A setter (or pair of setters) on LanceScanner that maps to Scanner::with_row_addr_prefilter, with C++ wrappers on lance::Scanner. Two shapes seem reasonable, and we're happy with either:

A. Row id arrays (in the style of lance_scanner_set_fragment_ids):

int32_t lance_scanner_set_row_addr_allowlist(LanceScanner* scanner, const uint64_t* row_ids, size_t len);
int32_t lance_scanner_set_row_addr_blocklist(LanceScanner* scanner, const uint64_t* row_ids, size_t len);

B. Serialized RowAddrTreeMap payloads (the same encoding pylance accepts), reusing RowAddrMask::from_serialized_parts:

int32_t lance_scanner_set_row_addr_prefilter(LanceScanner* scanner,
                                             const uint8_t* allowlist, size_t allowlist_len,
                                             const uint8_t* blocklist, size_t blocklist_len);

A is simpler for C callers. B is compact for large sets (a 1.5M-row mask is ~300 KiB against 12 MB of uint64_t) and matches pylance. With B, a helper that serializes a uint64_t array (like pylance's serialize_row_addrs) would spare C callers from reimplementing the treemap framing.

Expected semantics, matching pylance and core:

  • Values are in the dataset's _rowid space: the stable row id when stable row ids are enabled, otherwise the row address.
  • The mask applies before ranking. A nearest or full-text search, including one run through a prepared FTS query context, computes top-k over the allowed rows only, and scores are unchanged. A plain scan reads only the allowed rows.
  • The result is allowed AND NOT blocked, ANDed with any filter expression.
  • An empty allowlist selects no rows; not setting one means no restriction.
  • Must be set before the scan starts.

Implementation notes

We tried this locally on main and hit two places where the mask can be silently lost:

  1. Prepared FTS contexts. rewrite_prepared_fts_plan rebuilds each MatchQueryExec / PhraseQueryExec with new_with_segments(...).with_base_scorer(...). The rebuilt exec does not carry the original exec's external_mask, and core has no getter for it. A search with a prepared context therefore ignores the mask and returns every matching row. Re-applying .with_external_mask(...) on the rebuilt exec fixes it.
  2. Segment-scoped scalar scans. PreparedScalarSegment::configure calls with_row_addr_prefilter itself, which would replace a caller's mask. It should either reject the combination or intersect the masks (RowAddrMask::intersect).

Tests worth having: a multi-fragment dataset with stable row ids (so _rowid ≠ row address), covering plain scans, plain and prepared FTS (AND and OR operators, scores equal to the unrestricted search), and nearest-neighbor top-k over the allowed rows.

Related: #35 (compound FTS queries). A native Boolean query would cover our compound-FTS case, but the row prefilter is independent and also serves vector search and externally computed row sets.

Dominant language
Rust
Stars
8
Forks
16
Avg merge
1d 6h
Merged PRs (30d)
9

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance-c

All issues in lance-format/lance-c

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.