"from" with globs can be challenging with common filenames

Aperta
  • Qualcosa non va?
#7,271 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
go
Ambito
backend, databases

Direzione di ricerca

Start by reading the SuperSQL glob documentation linked in the issue and tracing how the from clause parses file paths; the issue does not identify parser files or tests. Compare parsing of the dated glob, the escaped single filename, and the quoted filename in the examples. Done means choosing and implementing a more straightforward way to support common filename globs, with tests covering those cases.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

With the filenames of the GitHub Archive data we often test with, a user attempting to use from with a glob currently hits a parse error.

$ super -version &&
  super -c "from 2023-02-08-*.json.gz | count()"

Version: v0.3.0-361-g7a03e7f05

parse error at line 1, column 10:
from 2023-02-08-*.json.gz | count()
     === ^ ===

While the docs disclose the reasons why this wouldn't work, super would benefit from a more straightforward way to allow glob wildcards here.

Details

Repro is with super commit 7a03e7f. These filenames were encountered "in the wild" and were not a contrived torture test designed to illustrate the challenge.

The docs on SuperSQL queries have a section on globs that does disclose why the above doesn't work. However, many users tend to hack at what they think of as intuitive UX before resorting to the docs, and if they try multiple things without luck they may simply give up.

For instance, if they go off the error message above referencing column 10 and the pointer to the first -, they might first try escaping the hyphens.

$ super -c "from 2023\-02\-08\-*.json.gz | count()"
parse error at line 1, column 10:
from 2023\-02\-08\-*.json.gz | count()
     === ^ ===

Perhaps next they'd try quoting, but of course this disables the globbing.

$ super -c "from '2023-02-08-*.json.gz' | count()"
file does not exist at line 1, column 6:
from '2023-02-08-*.json.gz' | count()
     ~~~~~~~~~~~~~~~~~~~~~~

"Bare" glob is ultimately the way to go, but the leading digit was foiling us, so the combination of a different prefix combined with the escaping of the hyphens does the trick.

$ super -c "from ./2023\-02\-08\-*.json.gz | count()"
4434953

Sadly, if the user manages to figure this out, if they try to modify it to target just one of the files, now that doesn't work.

$ super -c "from ./2023\-02\-08\-0.json.gz | count()"
parse error at line 1, column 31:
from ./2023\-02\-08\-0.json.gz | count()
                          === ^ ===

For that they'd need to go back to quoted. 🤪

$ super -c "from '2023-02-08-0.json.gz' | count()"
172049

Ideas

A couple ideas I've heard kicked around when this has been discussed:

  1. Perhaps the parser could be relaxed (specifically for the from case?) to be more permissive of common filenames, such as these that lead with datestamps.

  2. Perhaps a function could be used to wrap a quoted string but treat it as a glob once unwrapped (ike read_json() in DuckDB or file() in ClickHouse).

Lingua principale
Go
Stelle
1.6k
Fork
84
Merge medio
10h 17m
PR unite (30g)
156

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di superdb/super

Tutte le issue di superdb/super

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.