rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't Format.CSV

未关闭 适合新手
  • 哪里不对？
#892 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

@feiiiiii5 已经在做这个了。

开始于 2026年9月30日。

  • #893 来自 @feiiiiii5 —— 未关闭

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
88/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
tooling

调研方向

从 sqlite_utils/utils.py 中的 rows_from_file() 开始，检查 tests/test_rows_from_file.py，尤其是第 44 行现有的 CSV 测试。运行 rows_from_file 测试，然后覆盖明确提到的 TSV 和自动检测路径，同时保留默认的 RowError 行为。完成的标准是：ignore_extras 和 extras_key 在这些路径中一致地工作，且不改变文档中说明的默认值。

由索引模型根据 Issue 内容生成。

描述

rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't explicitly format=Format.CSV.

The docstring says:

If a CSV or TSV file includes rows with more fields than are declared in the header a sqlite_utils.utils.RowError exception will be raised when you loop over the generator.

You can instead ignore the extra data by passing ignore_extras=True.

Or pass extras_key="rest" to put those additional values in a list in a key called rest.

In practice those two options only work for format=Format.CSV. For TSV — which the docstring names explicitly — and for the auto-detected path (format omitted) they are silently discarded, and the caller's own explicit request is contradicted by a RowError.

Both branches that recurse into the CSV branch call rows_from_file() without forwarding the two arguments, so the inner call uses its own defaults (ignore_extras=False, extras_key=None) and raises before the outer _extra_key_strategy() wrapper — which is dead code — ever sees the row:

>>> from io import BytesIO
>>> from sqlite_utils.utils import rows_from_file, Format
>>> rows, _ = rows_from_file(
...     BytesIO(b"id\tname\r\n1\tCleo\toops"),
...     format=Format.TSV,
...     ignore_extras=True,
... )
>>> list(rows)
Traceback (most recent call last):
  ...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']

The same happens with extras_key="_rest":

>>> rows, _ = rows_from_file(
...     BytesIO(b"id\tname\r\n1\tCleo\toops"),
...     format=Format.TSV,
...     extras_key="_rest",
... )
>>> list(rows)
Traceback (most recent call last):
  ...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']

With no options passed the behaviour is correct and should not change — RowError is still raised, as documented.

git log -S"_extra_key_strategy" points at d379f43 ("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches in one commit. The recursion has never forwarded the arguments, so this looks like an oversight in the original commit rather than a decision.

The existing test hardcoded format=Format.CSV (tests/test_rows_from_file.py:44), which is why it was never caught.

Proposed fix

Forward ignore_extras= and extras_key= to the inner call in both branches, and drop the now-redundant outer _extra_key_strategy wrapper so the strategy is applied once, on the shared CSV path that already implements it.

Note on scope

I would fix the auto-detect branch in the same commit, since it is the same root cause (same dropped-argument recursion, same dead wrapper) and would otherwise be a one-line follow-up. It interacts with #891: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honour ignore_extras too.

I did not add an auto-detect regression test, because csv.Sniffer returns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The TSV test passes an explicit format, so it does not depend on the sniffer.

主要语言
Python
星标
2.2k
派生
215
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

simonw/sqlite-utils 的其他 Issue

查看 simonw/sqlite-utils 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。