Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Relation filters (semi/anti-joins) force a full-source load on orderBy+limit queries — forward them to loadSubset instead

Geschlossen
  • Stimmt etwas nicht?
#1,898 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
typescript
Bereich
backend-api-design, databases, performance

Rechercherichtung

Start in query/compiler/order-by.ts at processOrderBy, then trace OrderedSourceLoader, LoadSubsetOptions, loadSubset forwarding, and getLoadSubsetDemandKey. Run the prototype tests against fake on-demand backends, including honored and ignored relation hints and live child insert/delete cases. Done means limited relation-filtered queries avoid full-source loads while preserving exact pages and live-window updates.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Problem

A windowed live query (orderBy + limit) that filters its root rows by the existence of related rows loads the entire root collection.

Example: "seeks, newest first, 20 per page, that have at least one run by a user outside a hidden set". Expressed as a leftJoin of the child collection (or a distinct child subquery) plus where(not(isUndefined(j.key))) (semi-join), or isUndefined(j.key) (anti-join).

In query/compiler/order-by.ts (processOrderBy), requiresFullSource is set whenever any where references an alias other than the orderBy alias. OrderedSourceLoader then calls loadFullSource(), so the root's loadSubset receives an unlimited request with no limit/cursor. On-demand collections pull the whole table. Adapters that refuse unbounded loads return nothing.

Why it is there (as far as I can tell)

With a limit, the joined side loads lazily by key (SubsetDemandController). Root rows filtered out while their children are still in flight make dataNeeded positive, so loadMore keeps paging. With a slow child side this walks the whole table anyway, so forcing a full load is the safe fallback.

Proposal
  1. Structured, backend-agnostic hint. Add LoadSubsetOptions.relations?: Array<{ quantifier: 'some' | 'none'; collectionId: string; on: { parent: string[]; child: string[] }; where?: BasicExpression<boolean> }>. every(p) is none(not p). A collection may apply it (return only root rows that pass) or ignore it (return a superset), but must never drop a passing row. Local evaluation stays authoritative, so correctness never depends on the collection.
  2. Compiler. In processOrderBy, split the and conjuncts. Treat not(isUndefined(j.k)) / isUndefined(j.k) as a semi/anti-join when j is a LEFT join on eq(root.x, j.k) and j is a collection or a simple single-collection subquery. Those conjuncts no longer set requiresFullSource; they populate info.relations, which the root subscription forwards on every loadSubset (including requestLimitedSnapshot / tie loads).
  3. Loader. loadMore waits while any lazy join demand is unsettled or a joined subscription is loadingSubset. settleDemand already reruns the graph and the loaders.
  4. Joined-side changes. Changes on the joined side must also run the ordered loaders. Removing a child can shrink the window, and today that path (scheduleGraphRun(undefined)) passes no loader callback, so the window stays short.
  5. Dedupe. getLoadSubsetDemandKey includes relations, so a relation-narrowed load never dedupes (covers) the same load without the relation.

Backends can honor the hint exactly where they support it. For example, PostgREST: select=*,children!inner(userId)&children.userId=not.in.(…) for some, and a filtered embed with children=is.null for none. An SQL backend can use EXISTS (…).

Prototype

I implemented this as a patch on 0.9.2: about 150 lines of logic per build format, esm and cjs, plus types. It is covered by tests against fake on-demand backends, 40 root rows:

Case Today Hint honored Hint ignored
some, take 5 1 unlimited request, all 40 rows 2 requests, 6 rows 6 requests, 13 rows fetched, exact page
some, take 10 all 40 rows 2 requests, 11 rows 10 requests, 26 rows fetched, exact page
none / every, take 5 all 40 rows 2 requests, 6 rows 10 requests, 16 rows fetched, exact page

(The honored case's second request is the boundary-tie load.)

Tests cover:

  • collections that honor the hint (exact pages, no full-source load);
  • collections that ignore it (exact pages, stops at the minimal prefix, never loads the full source);
  • eager local-only collections;
  • live child insert and delete moving a parent into or out of the window, for some / none / every.

Without the demand gate in step 3, a slow child side (40 ms vs 0 ms) makes the ignoring case load 40 of 40 rows. Happy to open a PR from it if the direction looks right.

Out of scope / open questions
  • Should relations also flow for unlimited queries? The prototype only forwards them on limited windows.
  • Should demand-gating be scoped per join instead of query-wide? Query-wide is simple, but an unrelated include's pending demand delays paging (latency only).
  • Relation predicates under or / not, and joins through subqueries that themselves join, still force a full source in the prototype.
  • Adapters that page by offset count local rows, which other queries may have inflated. Relation-narrowed loads probably need to page by cursor.

Related: #1657 (loadSubset / pagination core RFC), #1887 (join key segments for remote sources).

Vorherrschende Sprache
TypeScript
Sterne
4k
Forks
277
Ø Merge
1 T. 2 Std.
Gemergte PRs (30 T.)
213

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus TanStack/db

Alle Issues in TanStack/db

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.