`fn.select` result validation walks all nested data in every row, in production
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 42/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- node.js, typescript
- 领域
- performance
调研方向
Start in packages/db/src/query/compiler/index.ts, where getUnsupportedFnSelectResultDescription recursively walks fn.select results and validateFnSelectResult is called; read the related query value construction points in packages/db/src/query/ir.ts and packages/db/src/query/builder/index.ts. Review packages/db/tests/query/functional-variants.test.ts and its nested data test. Done means validation avoids walking passed-through nested row data while still rejecting query values returned by callbacks.
由索引模型根据 Issue 内容生成。
描述
What happens
validateFnSelectResult recurses into everything reachable from each fn.select result, on every emission, in production builds too. A fn.select that computes a few fields and passes the rest of the record through, such as ({ r }) => ({ ...r, label: format(r) }), pays for walking that record's nested data on every row and every emission. Data read from the input row can't contain a query value, so the walk finds nothing.
The overhead is proportional to rows × nested objects per row. Each item below is 2 objects (NODE_ENV=production, 1.0.0-rc.1.4, Node 26, medians of 5 runs; initial load / update of every row, ms):
| Rows | Items per row | fn.select |
Same projection as expression select |
|---|---|---|---|
| 100 | 10 | 5.2 / 5.3 | 4.4 / 4.4 |
| 1,000 | 10 | 16.0 / 29.8 | 9.0 / 21.4 |
| 10,000 | 10 | 86.6 / 225.2 | 38.2 / 144.3 |
| 100 | 100 | 12.0 / 18.1 | 4.6 / 7.9 |
| 1,000 | 100 | 59.5 / 132.6 | 11.5 / 48.8 |
| 10,000 | 100 | 471.2 / 1,260.2 | 48.6 / 410.8 |
At 10,000 rows of 10 items, fn.select takes 86.6 ms to load instead of 38.2 ms. At 100 items per row it takes 471.2 ms instead of 48.6 ms, about 90% of it in the walk.
Repro
// NODE_ENV=production node repro.mjs fn|expr [rows] [itemsPerRow]
import { createCollection, localOnlyCollectionOptions, createLiveQueryCollection, upper } from '@tanstack/db'
const [variant = 'fn', rowCount = '10000', items = '10'] = process.argv.slice(2)
const rows = Array.from({ length: Number(rowCount) }, (_, i) => ({
id: i,
name: `item ${i}`,
rev: 0,
data: { items: Array.from({ length: Number(items) }, (_, k) => ({ k, tags: ['a', 'b'] })) },
}))
const source = createCollection(localOnlyCollectionOptions({ getKey: (r) => r.id, initialData: rows }))
const query = variant === 'fn'
? (q) => q.from({ r: source }).fn.select(({ r }) => ({ id: r.id, name: r.name.toUpperCase(), rev: r.rev, data: r.data }))
: (q) => q.from({ r: source }).select(({ r }) => ({ id: r.id, name: upper(r.name), rev: r.rev, data: r.data }))
const t0 = performance.now()
const view = createLiveQueryCollection({ query, getKey: (r) => r.id })
await view.preload()
const initial = performance.now() - t0
const t1 = performance.now()
source.update(rows.map((r) => r.id), (drafts) => drafts.forEach((d) => { d.rev += 1 }))
await new Promise((r) => setTimeout(r, 0))
console.log(JSON.stringify({ variant, initialMs: +initial.toFixed(1), updateMs: +(performance.now() - t1).toFixed(1) }))
When fn.select is the only option
The docs call the functional variants "intended for use in rare cases where the standard API is not sufficient". Expressions offer comparison, logic, null checks, upper / lower / length / concat, arithmetic, coalesce, caseWhen and aggregates. Apps that keep API records in collections often need something beyond that, and the UI usually still wants the whole record:
- String, date and number logic. The docs' own example,
row.user.email.split('@')[1], has no expression equivalent. Neither doIntlformatting, relative times or parsing a version or an IP address:{ ...row.order, placed: formatRelative(row.order.placedAt) }. - Values derived from nested arrays. Expressions have no array operators:
{ ...row.device, portsUp: row.device.ports.filter((p) => p.up).length }, where the UI still rendersports. - Merging two sources of the same entity. A REST snapshot joined with a realtime stream, with per-field precedence or newest-wins.
coalescecovers only "first non-null":{ ...rest, ...live, status: live.updatedAt > rest.updatedAt ? live.status : rest.status }. - Discriminated unions.
switch (row.item.kind)picks a per-kind shape or helper and returns the original payload with it. - Existing domain code. Permission checks, schema transforms and shared formatters already exist as functions, not as expressions.
In each case the callback computes a few fields and hands the source data back by reference, and that hand-back is what the walk pays for.
Cause
getUnsupportedFnSelectResultDescription (packages/db/src/query/compiler/index.ts) recurses into every enumerable property of the result. #1760 made the walk unbounded on purpose ("runtime validation remains unbounded and cycle-safe"), so that JavaScript callers and any-typed results get the same error as typed ones. But a query value can reach the result in only two ways: the callback builds one, or it captures one from outside.
Proposal
Walk the whole result only when the callback built a query value. Otherwise, check the result and its own properties. Development and production behave the same, and there is no flag.
-
A counter in a new internal module (not
ir.ts, which is public asIR), incremented wherever a query value is constructed: theBaseExpressionconstructor (every expression class),registerWrapper(all five helper wrappers) and theBaseQueryBuilderconstructor.// query/query-value-creations.ts export const queryValueCreations = { count: 0 } // query/ir.ts abstract class BaseExpression<T = any> { constructor() { queryValueCreations.count++ } } -
The
fn.selectcall site compares the counter before and after the callback:const createdBefore = queryValueCreations.count const selectResults = query.fnSelect!(callbackRow) validateFnSelectResult(selectResults, queryValueCreations.count !== createdBefore) -
getUnsupportedFnSelectResultDescriptiontakes adepth:Infinitywhen deep, otherwise1, which checks the result and its own properties.
Full diff against main (`ea3c920c`); the four changed source files are unchanged since 1.0.0-rc.1.4
diff --git a/packages/db/src/query/builder/index.ts b/packages/db/src/query/builder/index.ts
index fee87500..18c2322a 100644
--- a/packages/db/src/query/builder/index.ts
+++ b/packages/db/src/query/builder/index.ts
@@ -17,6 +17,7 @@ import {
collectSourceRefs,
isExpressionLike,
} from '../ir.js'
+import { queryValueCreations } from '../query-value-creations.js'
import {
IncludeOperatorRequiredError,
InvalidSourceError,
@@ -150,6 +151,7 @@ export class BaseQueryBuilder<TContext extends Context = Context> {
/** @internal Whether the builder may keep `query` without copying it. */
owned = false,
) {
+ queryValueCreations.count++
this.query = owned ? query : { ...query }
}
diff --git a/packages/db/src/query/builder/wrapper-identity.ts b/packages/db/src/query/builder/wrapper-identity.ts
index 6657f542..9957d223 100644
--- a/packages/db/src/query/builder/wrapper-identity.ts
+++ b/packages/db/src/query/builder/wrapper-identity.ts
@@ -1,6 +1,9 @@
+import { queryValueCreations } from '../query-value-creations.js'
+
const wrapperNames = new WeakMap<object, string>()
export function registerWrapper(value: object, name: string): void {
+ queryValueCreations.count++
wrapperNames.set(value, name)
}
diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts
index 5fb83aec..0b60be1a 100644
--- a/packages/db/src/query/compiler/index.ts
+++ b/packages/db/src/query/compiler/index.ts
@@ -56,6 +56,7 @@ import {
isBasicOrAggregateExpression,
isExpressionLike,
} from '../ir.js'
+import { queryValueCreations } from '../query-value-creations.js'
import { ensureIndexForField } from '../../indexes/auto-index.js'
import { createSourceRecord } from '../../utils/source-record.js'
import { deepEquals } from '../../utils.js'
@@ -123,6 +124,7 @@ export { INCLUDES_PUBLIC_KEY, INCLUDES_ROUTING } from './route-metadata.js'
function getUnsupportedFnSelectResultDescription(
value: unknown,
seen: Set<object> = new Set(),
+ depth = Infinity,
): string | undefined {
if (value instanceof IncludeWrapper) return `include()`
if (value instanceof BaseQueryBuilder) return `a child query builder`
@@ -138,7 +140,12 @@ function getUnsupportedFnSelectResultDescription(
? `${value.name}()`
: `a query expression`
}
- if (value === null || typeof value !== `object` || seen.has(value)) {
+ if (
+ value === null ||
+ typeof value !== `object` ||
+ seen.has(value) ||
+ depth === 0
+ ) {
return undefined
}
@@ -148,15 +155,26 @@ function getUnsupportedFnSelectResultDescription(
)
for (const key of keys) {
const entry = (value as Record<PropertyKey, unknown>)[key]
- const unsupported = getUnsupportedFnSelectResultDescription(entry, seen)
+ const unsupported = getUnsupportedFnSelectResultDescription(
+ entry,
+ seen,
+ depth - 1,
+ )
if (unsupported) return unsupported
}
return undefined
}
-export function validateFnSelectResult(value: unknown): void {
- const unsupportedValueDescription =
- getUnsupportedFnSelectResultDescription(value)
+/**
+ * `deep: false` checks only the result and its own properties: for when the
+ * callback built no query value, so anything deeper is data it read.
+ */
+export function validateFnSelectResult(value: unknown, deep = true): void {
+ const unsupportedValueDescription = getUnsupportedFnSelectResultDescription(
+ value,
+ new Set(),
+ deep ? Infinity : 1,
+ )
if (unsupportedValueDescription) {
throw new UnsupportedFnSelectResultError(unsupportedValueDescription)
}
@@ -1058,8 +1076,14 @@ function compileQueryInternal(
const callbackRow = stripInternalCallbackMetadata(
namespacedRow,
) as NamespacedRow
+ // Row data can't hold a query value, so only a callback that built one
+ // needs its whole result walked.
+ const createdBefore = queryValueCreations.count
const selectResults = query.fnSelect!(callbackRow)
- validateFnSelectResult(selectResults)
+ validateFnSelectResult(
+ selectResults,
+ queryValueCreations.count !== createdBefore,
+ )
let selected = selectResults
const routing = (namespacedRow as any)[INCLUDES_ROUTING]
if (routing && isPublicContainer(selectResults)) {
diff --git a/packages/db/src/query/ir.ts b/packages/db/src/query/ir.ts
index 65f200d4..f2ecb1e6 100644
--- a/packages/db/src/query/ir.ts
+++ b/packages/db/src/query/ir.ts
@@ -1,5 +1,6 @@
import { codedMessage, devBuild } from '../error-message.js'
import { isRefProxy } from './builder/ref-proxy-identity.js'
+import { queryValueCreations } from './query-value-creations.js'
/*
This is the intermediate representation of the query.
@@ -86,6 +87,9 @@ abstract class BaseExpression<T = any> {
public abstract type: string
/** @internal - Type brand for TypeScript inference */
declare readonly __returnType: T
+ constructor() {
+ queryValueCreations.count++
+ }
}
export class CollectionRef extends BaseExpression {
diff --git a/packages/db/src/query/query-value-creations.ts b/packages/db/src/query/query-value-creations.ts
new file mode 100644
index 00000000..d7035080
--- /dev/null
+++ b/packages/db/src/query/query-value-creations.ts
@@ -0,0 +1,6 @@
+/**
+ * Counts constructed query values (expressions, helper wrappers and query
+ * builders), so `fn.select` can tell whether its callback built one. Kept out
+ * of `ir.ts`, which is public as the `IR` namespace.
+ */
+export const queryValueCreations = { count: 0 }
diff --git a/packages/db/tests/query/functional-variants.test.ts b/packages/db/tests/query/functional-variants.test.ts
index 8833be09..f76b9cb2 100644
--- a/packages/db/tests/query/functional-variants.test.ts
+++ b/packages/db/tests/query/functional-variants.test.ts
@@ -368,6 +368,26 @@ describe(`Functional Variants Query`, () => {
}),
).toThrow(`fn.select() cannot return caseWhen()`)
})
+
+ test(`does not walk nested data that fn.select passes through`, () => {
+ let nestedReads = 0
+ const nested = {
+ get deep() {
+ nestedReads++
+ return { deeper: [1, 2, 3] }
+ },
+ }
+ const liveCollection = createLiveQueryCollection({
+ startSync: true,
+ query: (q) =>
+ q
+ .from({ user: usersCollection })
+ .fn.select((row) => ({ id: row.user.id, data: { nested } })),
+ })
+
+ expect(liveCollection.size).toBe(5)
+ expect(nestedReads).toBe(0)
+ })
})
describe(`fn.where`, () => {
Applied to the rc.1.4 build, it rejects every case the existing fn.select rejection test covers, including builders created only on a later emission. The repro's fn.select runs at expression speed:
| Rows | Items per row | fn.select today |
fn.select with this change |
|---|---|---|---|
| 100 | 10 | 5.2 / 5.3 | 3.7 / 4.7 |
| 1,000 | 10 | 16.0 / 29.8 | 8.2 / 21.2 |
| 10,000 | 10 | 86.6 / 225.2 | 36.9 / 133.7 |
| 100 | 100 | 12.0 / 18.1 | 3.9 / 7.7 |
| 1,000 | 100 | 59.5 / 132.6 | 11.1 / 48.6 |
| 10,000 | 100 | 471.2 / 1,260.2 | 44.1 / 407.2 |
Trade-off: the runtime check no longer catches a query value created outside the callback, captured, and placed two or more levels deep in the result ({ a: { b: capturedEq } }). The type-level check from #1760 still rejects it in typed code, and a captured value at the top level is still caught. If that gap is too much, the same counter could keep the full walk in development builds only.
Happy to send this as a PR, or whichever variant you prefer.
- 主要语言
- TypeScript
- 星标
- 4k
- 派生
- 286
- 平均合并
- 1 天 8 小时
- 30 天内合并 PR
- 105
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
TanStack/db 的其他 Issue
-
extractSimpleComparisons returns value: null when the right side of a comparison isn't a literal未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
`avg`, `min`, `max` and `sum` treat `null` values as `0`可能已有人在做 @oyobbeb 于 11 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
`Collection.currentStateAsChanges` JSDoc examples don’t match `CurrentStateAsChangesOptions`可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 1/5 1-3 小时 新手友好度 88/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
Two @tanstack/db tests time out under CPU load because a cold module import counts against the 5 s default timeout可能已有人在做 @jp-knj 于 23 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
fb55/htmlparser2#2579 ·
维护者通常 1 天内回复
-
bug good first issue needs-review
难度 1/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复
-
documentation public
难度 1/5 1-3 小时 新手友好度 68/100
midnightntwrk/umbradb#65 ·
维护者通常 2 天内回复
-
area/channels kind/bug priority/medium
难度 2/5 1-3 小时 新手友好度 78/100
nanocoai/nanoclaw#4501 · 1 条评论 ·
维护者通常 1 天内回复
-
needs verification
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 13 天内回复