[Bug] Spark vector search returns a short top-K when a residual filter is not pushed down

Start by tracing SparkV2FilterConverter and PaimonBaseScanBuilder.pushPredicates to see how non-convertible predicates become postScan filters during vector_search. Then inspect the lateral path in PushDownLateralVectorSearchFilter and compare the behavior with issue #9883. Done means residual filters no longer produce incomplete top-k results, either by refusing the pushdown or by refilling results after filtering.

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

Spark (primary-key / data-evolution vector search via the vector_search TVF)

Minimal reproduce step

Run a Spark vector_search(...) with a LIMIT k, and a WHERE conjunct that SparkV2FilterConverter cannot convert to a Paimon predicate: a UDF call, a column-to-column comparison (a > b), or an unresolvable cast. For example a filter like udf(col) = 1 combined with the vector search, limit = k.

What doesn't meet your expectations?

The result can miss qualifying rows.

The vector search returns exactly k rows, the top-k by vector distance, computed over the pushed-down (convertible) predicates only. A non-convertible conjunct is not pushed into Paimon (PaimonBaseScanBuilder.pushPredicates keeps it in postScan), so Spark applies it as a FilterExec above the already-truncated top-k. Post-filtering the top-k can only drop rows, never refill the ones displaced out of it, so rows that satisfy the residual but ranked just outside the returned k are lost, and the result is short.

This is distinct from #9883, which is about a predicate that reaches Paimon but the scalar index cannot evaluate. Here the predicate never reaches Paimon at all, so the evaluator-level fix in #9883 does not cover it. The Flink vector_search procedure does not have this problem: it parses the whole where string into the Paimon predicate, so an inexpressible conjunct fails the call rather than silently becoming a residual.

Anything else?

Found while tracing the vector-search filter pushdown for #9883. Fix direction: when a vector search has a non-convertible residual, either refuse the pushdown / fail fast, or over-retrieve and re-apply the residual to refill the top-k. The lateral / dynamic-query-vector path (PushDownLateralVectorSearchFilter) has the same exposure.

Are you willing to submit a PR?

• I'm willing to submit a PR!