`fn.select` result validation walks all nested data in every row, in production

Đang mở
#2,399 0 bình luận 1 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, typescript
Lĩnh vực
performance

Hướng nghiên cứu

Start in packages/db/src/query/compiler/index.ts, where getUnsupportedFnSelectResultDescription recursively walks fn.select results and validateFnSelectResult is called; read the related query value construction points in packages/db/src/query/ir.ts and packages/db/src/query/builder/index.ts. Review packages/db/tests/query/functional-variants.test.ts and its nested data test. Done means validation avoids walking passed-through nested row data while still rejecting query values returned by callbacks.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

What happens

validateFnSelectResult recurses into everything reachable from each fn.select result, on every emission, in production builds too. A fn.select that computes a few fields and passes the rest of the record through, such as ({ r }) => ({ ...r, label: format(r) }), pays for walking that record's nested data on every row and every emission. Data read from the input row can't contain a query value, so the walk finds nothing.

The overhead is proportional to rows × nested objects per row. Each item below is 2 objects (NODE_ENV=production, 1.0.0-rc.1.4, Node 26, medians of 5 runs; initial load / update of every row, ms):

Rows Items per row fn.select Same projection as expression select
100 10 5.2 / 5.3 4.4 / 4.4
1,000 10 16.0 / 29.8 9.0 / 21.4
10,000 10 86.6 / 225.2 38.2 / 144.3
100 100 12.0 / 18.1 4.6 / 7.9
1,000 100 59.5 / 132.6 11.5 / 48.8
10,000 100 471.2 / 1,260.2 48.6 / 410.8

At 10,000 rows of 10 items, fn.select takes 86.6 ms to load instead of 38.2 ms. At 100 items per row it takes 471.2 ms instead of 48.6 ms, about 90% of it in the walk.

Repro
// NODE_ENV=production node repro.mjs fn|expr [rows] [itemsPerRow]
import { createCollection, localOnlyCollectionOptions, createLiveQueryCollection, upper } from '@tanstack/db'

const [variant = 'fn', rowCount = '10000', items = '10'] = process.argv.slice(2)
const rows = Array.from({ length: Number(rowCount) }, (_, i) => ({
  id: i,
  name: `item ${i}`,
  rev: 0,
  data: { items: Array.from({ length: Number(items) }, (_, k) => ({ k, tags: ['a', 'b'] })) },
}))
const source = createCollection(localOnlyCollectionOptions({ getKey: (r) => r.id, initialData: rows }))

const query = variant === 'fn'
  ? (q) => q.from({ r: source }).fn.select(({ r }) => ({ id: r.id, name: r.name.toUpperCase(), rev: r.rev, data: r.data }))
  : (q) => q.from({ r: source }).select(({ r }) => ({ id: r.id, name: upper(r.name), rev: r.rev, data: r.data }))

const t0 = performance.now()
const view = createLiveQueryCollection({ query, getKey: (r) => r.id })
await view.preload()
const initial = performance.now() - t0

const t1 = performance.now()
source.update(rows.map((r) => r.id), (drafts) => drafts.forEach((d) => { d.rev += 1 }))
await new Promise((r) => setTimeout(r, 0))
console.log(JSON.stringify({ variant, initialMs: +initial.toFixed(1), updateMs: +(performance.now() - t1).toFixed(1) }))
When fn.select is the only option

The docs call the functional variants "intended for use in rare cases where the standard API is not sufficient". Expressions offer comparison, logic, null checks, upper / lower / length / concat, arithmetic, coalesce, caseWhen and aggregates. Apps that keep API records in collections often need something beyond that, and the UI usually still wants the whole record:

  • String, date and number logic. The docs' own example, row.user.email.split('@')[1], has no expression equivalent. Neither do Intl formatting, relative times or parsing a version or an IP address: { ...row.order, placed: formatRelative(row.order.placedAt) }.
  • Values derived from nested arrays. Expressions have no array operators: { ...row.device, portsUp: row.device.ports.filter((p) => p.up).length }, where the UI still renders ports.
  • Merging two sources of the same entity. A REST snapshot joined with a realtime stream, with per-field precedence or newest-wins. coalesce covers only "first non-null": { ...rest, ...live, status: live.updatedAt > rest.updatedAt ? live.status : rest.status }.
  • Discriminated unions. switch (row.item.kind) picks a per-kind shape or helper and returns the original payload with it.
  • Existing domain code. Permission checks, schema transforms and shared formatters already exist as functions, not as expressions.

In each case the callback computes a few fields and hands the source data back by reference, and that hand-back is what the walk pays for.

Cause

getUnsupportedFnSelectResultDescription (packages/db/src/query/compiler/index.ts) recurses into every enumerable property of the result. #1760 made the walk unbounded on purpose ("runtime validation remains unbounded and cycle-safe"), so that JavaScript callers and any-typed results get the same error as typed ones. But a query value can reach the result in only two ways: the callback builds one, or it captures one from outside.

Proposal

Walk the whole result only when the callback built a query value. Otherwise, check the result and its own properties. Development and production behave the same, and there is no flag.

  1. A counter in a new internal module (not ir.ts, which is public as IR), incremented wherever a query value is constructed: the BaseExpression constructor (every expression class), registerWrapper (all five helper wrappers) and the BaseQueryBuilder constructor.

    // query/query-value-creations.ts
    export const queryValueCreations = { count: 0 }
    
    // query/ir.ts
    abstract class BaseExpression<T = any> {
      constructor() {
        queryValueCreations.count++
      }
    }
    
  2. The fn.select call site compares the counter before and after the callback:

    const createdBefore = queryValueCreations.count
    const selectResults = query.fnSelect!(callbackRow)
    validateFnSelectResult(selectResults, queryValueCreations.count !== createdBefore)
    
  3. getUnsupportedFnSelectResultDescription takes a depth: Infinity when deep, otherwise 1, which checks the result and its own properties.

Full diff against main (`ea3c920c`); the four changed source files are unchanged since 1.0.0-rc.1.4
diff --git a/packages/db/src/query/builder/index.ts b/packages/db/src/query/builder/index.ts
index fee87500..18c2322a 100644
--- a/packages/db/src/query/builder/index.ts
+++ b/packages/db/src/query/builder/index.ts
@@ -17,6 +17,7 @@ import {
   collectSourceRefs,
   isExpressionLike,
 } from '../ir.js'
+import { queryValueCreations } from '../query-value-creations.js'
 import {
   IncludeOperatorRequiredError,
   InvalidSourceError,
@@ -150,6 +151,7 @@ export class BaseQueryBuilder<TContext extends Context = Context> {
     /** @internal Whether the builder may keep `query` without copying it. */
     owned = false,
   ) {
+    queryValueCreations.count++
     this.query = owned ? query : { ...query }
   }
 
diff --git a/packages/db/src/query/builder/wrapper-identity.ts b/packages/db/src/query/builder/wrapper-identity.ts
index 6657f542..9957d223 100644
--- a/packages/db/src/query/builder/wrapper-identity.ts
+++ b/packages/db/src/query/builder/wrapper-identity.ts
@@ -1,6 +1,9 @@
+import { queryValueCreations } from '../query-value-creations.js'
+
 const wrapperNames = new WeakMap<object, string>()
 
 export function registerWrapper(value: object, name: string): void {
+  queryValueCreations.count++
   wrapperNames.set(value, name)
 }
 
diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts
index 5fb83aec..0b60be1a 100644
--- a/packages/db/src/query/compiler/index.ts
+++ b/packages/db/src/query/compiler/index.ts
@@ -56,6 +56,7 @@ import {
   isBasicOrAggregateExpression,
   isExpressionLike,
 } from '../ir.js'
+import { queryValueCreations } from '../query-value-creations.js'
 import { ensureIndexForField } from '../../indexes/auto-index.js'
 import { createSourceRecord } from '../../utils/source-record.js'
 import { deepEquals } from '../../utils.js'
@@ -123,6 +124,7 @@ export { INCLUDES_PUBLIC_KEY, INCLUDES_ROUTING } from './route-metadata.js'
 function getUnsupportedFnSelectResultDescription(
   value: unknown,
   seen: Set<object> = new Set(),
+  depth = Infinity,
 ): string | undefined {
   if (value instanceof IncludeWrapper) return `include()`
   if (value instanceof BaseQueryBuilder) return `a child query builder`
@@ -138,7 +140,12 @@ function getUnsupportedFnSelectResultDescription(
       ? `${value.name}()`
       : `a query expression`
   }
-  if (value === null || typeof value !== `object` || seen.has(value)) {
+  if (
+    value === null ||
+    typeof value !== `object` ||
+    seen.has(value) ||
+    depth === 0
+  ) {
     return undefined
   }
 
@@ -148,15 +155,26 @@ function getUnsupportedFnSelectResultDescription(
   )
   for (const key of keys) {
     const entry = (value as Record<PropertyKey, unknown>)[key]
-    const unsupported = getUnsupportedFnSelectResultDescription(entry, seen)
+    const unsupported = getUnsupportedFnSelectResultDescription(
+      entry,
+      seen,
+      depth - 1,
+    )
     if (unsupported) return unsupported
   }
   return undefined
 }
 
-export function validateFnSelectResult(value: unknown): void {
-  const unsupportedValueDescription =
-    getUnsupportedFnSelectResultDescription(value)
+/**
+ * `deep: false` checks only the result and its own properties: for when the
+ * callback built no query value, so anything deeper is data it read.
+ */
+export function validateFnSelectResult(value: unknown, deep = true): void {
+  const unsupportedValueDescription = getUnsupportedFnSelectResultDescription(
+    value,
+    new Set(),
+    deep ? Infinity : 1,
+  )
   if (unsupportedValueDescription) {
     throw new UnsupportedFnSelectResultError(unsupportedValueDescription)
   }
@@ -1058,8 +1076,14 @@ function compileQueryInternal(
         const callbackRow = stripInternalCallbackMetadata(
           namespacedRow,
         ) as NamespacedRow
+        // Row data can't hold a query value, so only a callback that built one
+        // needs its whole result walked.
+        const createdBefore = queryValueCreations.count
         const selectResults = query.fnSelect!(callbackRow)
-        validateFnSelectResult(selectResults)
+        validateFnSelectResult(
+          selectResults,
+          queryValueCreations.count !== createdBefore,
+        )
         let selected = selectResults
         const routing = (namespacedRow as any)[INCLUDES_ROUTING]
         if (routing && isPublicContainer(selectResults)) {
diff --git a/packages/db/src/query/ir.ts b/packages/db/src/query/ir.ts
index 65f200d4..f2ecb1e6 100644
--- a/packages/db/src/query/ir.ts
+++ b/packages/db/src/query/ir.ts
@@ -1,5 +1,6 @@
 import { codedMessage, devBuild } from '../error-message.js'
 import { isRefProxy } from './builder/ref-proxy-identity.js'
+import { queryValueCreations } from './query-value-creations.js'
 
 /*
 This is the intermediate representation of the query.
@@ -86,6 +87,9 @@ abstract class BaseExpression<T = any> {
   public abstract type: string
   /** @internal - Type brand for TypeScript inference */
   declare readonly __returnType: T
+  constructor() {
+    queryValueCreations.count++
+  }
 }
 
 export class CollectionRef extends BaseExpression {
diff --git a/packages/db/src/query/query-value-creations.ts b/packages/db/src/query/query-value-creations.ts
new file mode 100644
index 00000000..d7035080
--- /dev/null
+++ b/packages/db/src/query/query-value-creations.ts
@@ -0,0 +1,6 @@
+/**
+ * Counts constructed query values (expressions, helper wrappers and query
+ * builders), so `fn.select` can tell whether its callback built one. Kept out
+ * of `ir.ts`, which is public as the `IR` namespace.
+ */
+export const queryValueCreations = { count: 0 }
diff --git a/packages/db/tests/query/functional-variants.test.ts b/packages/db/tests/query/functional-variants.test.ts
index 8833be09..f76b9cb2 100644
--- a/packages/db/tests/query/functional-variants.test.ts
+++ b/packages/db/tests/query/functional-variants.test.ts
@@ -368,6 +368,26 @@ describe(`Functional Variants Query`, () => {
         }),
       ).toThrow(`fn.select() cannot return caseWhen()`)
     })
+
+    test(`does not walk nested data that fn.select passes through`, () => {
+      let nestedReads = 0
+      const nested = {
+        get deep() {
+          nestedReads++
+          return { deeper: [1, 2, 3] }
+        },
+      }
+      const liveCollection = createLiveQueryCollection({
+        startSync: true,
+        query: (q) =>
+          q
+            .from({ user: usersCollection })
+            .fn.select((row) => ({ id: row.user.id, data: { nested } })),
+      })
+
+      expect(liveCollection.size).toBe(5)
+      expect(nestedReads).toBe(0)
+    })
   })
 
   describe(`fn.where`, () => {

Applied to the rc.1.4 build, it rejects every case the existing fn.select rejection test covers, including builders created only on a later emission. The repro's fn.select runs at expression speed:

Rows Items per row fn.select today fn.select with this change
100 10 5.2 / 5.3 3.7 / 4.7
1,000 10 16.0 / 29.8 8.2 / 21.2
10,000 10 86.6 / 225.2 36.9 / 133.7
100 100 12.0 / 18.1 3.9 / 7.7
1,000 100 59.5 / 132.6 11.1 / 48.6
10,000 100 471.2 / 1,260.2 44.1 / 407.2

Trade-off: the runtime check no longer catches a query value created outside the callback, captured, and placed two or more levels deep in the result ({ a: { b: capturedEq } }). The type-level check from #1760 still rejects it in typed code, and a captured value at the top level is still caught. If that gap is too much, the same counter could keep the full walk in development builds only.

Happy to send this as a PR, or whichever variant you prefer.

Ngôn ngữ chính
TypeScript
Star
4k
Fork
286
Merge trung bình
1 ngày 8 giờ
Pull request đã merge (30 ngày)
105

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của TanStack/db

Tất cả issue của TanStack/db

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.