rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't Format.CSV

Aperta Adatta ai principianti
  • Qualcosa non va?
#892 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@feiiiiii5 ci sta già lavorando.

Dal 30/9/2026.

  • #893 di @feiiiiii5 — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
88/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
tooling

Direzione di ricerca

Inizia in sqlite_utils/utils.py, in rows_from_file(), ed esamina tests/test_rows_from_file.py, in particolare il test CSV esistente alla riga 44. Esegui i test di rows_from_file, quindi copri i percorsi TSV e rilevati automaticamente menzionati esplicitamente, mantenendo il comportamento predefinito di RowError. Il lavoro è completato quando ignore_extras ed extras_key funzionano in modo coerente per questi percorsi senza modificare il valore predefinito documentato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't explicitly format=Format.CSV.

The docstring says:

If a CSV or TSV file includes rows with more fields than are declared in the header a sqlite_utils.utils.RowError exception will be raised when you loop over the generator.

You can instead ignore the extra data by passing ignore_extras=True.

Or pass extras_key="rest" to put those additional values in a list in a key called rest.

In practice those two options only work for format=Format.CSV. For TSV — which the docstring names explicitly — and for the auto-detected path (format omitted) they are silently discarded, and the caller's own explicit request is contradicted by a RowError.

Both branches that recurse into the CSV branch call rows_from_file() without forwarding the two arguments, so the inner call uses its own defaults (ignore_extras=False, extras_key=None) and raises before the outer _extra_key_strategy() wrapper — which is dead code — ever sees the row:

>>> from io import BytesIO
>>> from sqlite_utils.utils import rows_from_file, Format
>>> rows, _ = rows_from_file(
...     BytesIO(b"id\tname\r\n1\tCleo\toops"),
...     format=Format.TSV,
...     ignore_extras=True,
... )
>>> list(rows)
Traceback (most recent call last):
  ...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']

The same happens with extras_key="_rest":

>>> rows, _ = rows_from_file(
...     BytesIO(b"id\tname\r\n1\tCleo\toops"),
...     format=Format.TSV,
...     extras_key="_rest",
... )
>>> list(rows)
Traceback (most recent call last):
  ...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']

With no options passed the behaviour is correct and should not change — RowError is still raised, as documented.

git log -S"_extra_key_strategy" points at d379f43 ("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches in one commit. The recursion has never forwarded the arguments, so this looks like an oversight in the original commit rather than a decision.

The existing test hardcoded format=Format.CSV (tests/test_rows_from_file.py:44), which is why it was never caught.

Proposed fix

Forward ignore_extras= and extras_key= to the inner call in both branches, and drop the now-redundant outer _extra_key_strategy wrapper so the strategy is applied once, on the shared CSV path that already implements it.

Note on scope

I would fix the auto-detect branch in the same commit, since it is the same root cause (same dropped-argument recursion, same dead wrapper) and would otherwise be a one-line follow-up. It interacts with #891: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honour ignore_extras too.

I did not add an auto-detect regression test, because csv.Sniffer returns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The TSV test passes an explicit format, so it does not depend on the sniffer.

Lingua principale
Python
Stelle
2.2k
Fork
215
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di simonw/sqlite-utils

Tutte le issue di simonw/sqlite-utils

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.