lance-format/lance

Allow filtering of projected fields

Open

#2.971 geöffnet am 3. Okt. 2024

Auf GitHub ansehen
 (4 Kommentare) (1 Reaktion) (0 zugewiesene Personen)Rust (695 Forks)github user discovery
good first issue

Repository-Metriken

Stars
 (6.582 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 6T 1h) (219 gemergte PRs in 30 T)

Beschreibung

Users can introduce new fields using projection. However, they cannot turn around and filter on these fields. This can perhaps wait until we have general SQL support but it could be implemented earlier.

The challenge is that we compile filters against the scan input schema and this does not contain the projected fields. We should compile the filter against the final projected schema. Then we can subtract fields that don't exist in the scan input schema and add a final FilterExec after the project.

This is a little bit tricky because, if we do the above directly, then the take step will try and take these fields and not find them. So we need to change the take logic slightly as well.

Contributor Guide