"from" with globs can be challenging with common filenames

Abierto
  • ¿Algo no está bien?
#7,271 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
go
Área
backend, databases

Línea de trabajo

Start by reading the SuperSQL glob documentation linked in the issue and tracing how the from clause parses file paths; the issue does not identify parser files or tests. Compare parsing of the dated glob, the escaped single filename, and the quoted filename in the examples. Done means choosing and implementing a more straightforward way to support common filename globs, with tests covering those cases.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

With the filenames of the GitHub Archive data we often test with, a user attempting to use from with a glob currently hits a parse error.

$ super -version &&
  super -c "from 2023-02-08-*.json.gz | count()"

Version: v0.3.0-361-g7a03e7f05

parse error at line 1, column 10:
from 2023-02-08-*.json.gz | count()
     === ^ ===

While the docs disclose the reasons why this wouldn't work, super would benefit from a more straightforward way to allow glob wildcards here.

Details

Repro is with super commit 7a03e7f. These filenames were encountered "in the wild" and were not a contrived torture test designed to illustrate the challenge.

The docs on SuperSQL queries have a section on globs that does disclose why the above doesn't work. However, many users tend to hack at what they think of as intuitive UX before resorting to the docs, and if they try multiple things without luck they may simply give up.

For instance, if they go off the error message above referencing column 10 and the pointer to the first -, they might first try escaping the hyphens.

$ super -c "from 2023\-02\-08\-*.json.gz | count()"
parse error at line 1, column 10:
from 2023\-02\-08\-*.json.gz | count()
     === ^ ===

Perhaps next they'd try quoting, but of course this disables the globbing.

$ super -c "from '2023-02-08-*.json.gz' | count()"
file does not exist at line 1, column 6:
from '2023-02-08-*.json.gz' | count()
     ~~~~~~~~~~~~~~~~~~~~~~

"Bare" glob is ultimately the way to go, but the leading digit was foiling us, so the combination of a different prefix combined with the escaping of the hyphens does the trick.

$ super -c "from ./2023\-02\-08\-*.json.gz | count()"
4434953

Sadly, if the user manages to figure this out, if they try to modify it to target just one of the files, now that doesn't work.

$ super -c "from ./2023\-02\-08\-0.json.gz | count()"
parse error at line 1, column 31:
from ./2023\-02\-08\-0.json.gz | count()
                          === ^ ===

For that they'd need to go back to quoted. 🤪

$ super -c "from '2023-02-08-0.json.gz' | count()"
172049

Ideas

A couple ideas I've heard kicked around when this has been discussed:

  1. Perhaps the parser could be relaxed (specifically for the from case?) to be more permissive of common filenames, such as these that lead with datestamps.

  2. Perhaps a function could be used to wrap a quoted string but treat it as a glob once unwrapped (ike read_json() in DuckDB or file() in ClickHouse).

Lenguaje dominante
Go
Estrellas
1.6k
Forks
84
Merge medio
12 h 23 min
PR fusionados (30 d)
142

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de superdb/super

Todos los issues de superdb/super

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.