fix(nlp): match single-letter bullets in the enumerated bullet pattern
维护者通常 1 天内回复
评估
调研方向
阅读 unstructured/nlp/patterns.py 中的 ENUMERATED_BULLETS_RE,以及新建的 test_unstructured/nlp/test_patterns.py;issue 提供了正则表达式修改要求和复现示例。运行列出的 pytest 命令,检查正则表达式、转换器以及现有的数字和缩写用例。完成标准是:单字母列表标记被拆分为独立条目,同时数字和缩写用例仍然通过。
由索引模型根据 Issue 内容生成。
描述
ENUMERATED_BULLETS_RE cannot match a single-letter bullet.
unstructured/nlp/patterns.py:111 compiles
ENUMERATED_BULLETS_RE = re.compile(r"(?:(?:\d{1,3}|[a-z][A-Z])\.?){1,3}")
[a-z][A-Z] is two single-character classes placed next to each other, so it matches exactly two characters (aB) and never a single a or A. The two ordered-bullet helpers in this repo both accept a single letter as a marker:
unstructured/cleaners/core.py:53clean_ordered_bullets— docstring says "accounting numeric and alphanumeric types" and showsa.b This is a very important point -> This is a very important point.unstructured/cleaners/extract.py:104extract_ordered_bullets— docstring showsa.1 This is a very important point -> ("a", "1", None).
The splitter regex used by the list path cannot, so lettered lists come out as one blob:
from unstructured.partition.common.common import layout_list_to_list_items
[e.text for e in layout_list_to_list_items("a. foo\nb. bar", None, None, None, None)]
# ['a. foo\nb. bar'] expected ['foo', 'bar']
This is user visible through partition_image and partition_pdf(strategy="ocr_only"): ocr_data_to_elements -> normalize_layout_element -> layout_list_to_list_items (unstructured/partition/common/common.py:137).
Change
Keep the numeric branch byte for byte, and add a letter branch that only matches at the start of a line, only with a literal dot, and only when whitespace or the end of the text follows. The line anchor and the (?!\s[a-zA-Z]\.) guard are both required: \.? is optional, so an unanchored [a-zA-Z] turns every letter into a separator ("1. foo\n2. bar" would split into []), and without the guard "e. g. spaced example" loses its first word.
Verification
test_unstructured/nlp/test_patterns.py (new) covers both the regex and the converter, including the existing numeric behaviour and the abbreviation cases:
python -m pytest test_unstructured/nlp/test_patterns.py test_unstructured/nlp/test_tokenize.py test_unstructured/cleaners/test_core.py test_unstructured/cleaners/test_extract.py -q
pristine: 19 failed, 211 passed
patched: 10 failed, 220 passed
The 10 remaining failures are the pre-existing test_tokenize.py ones (no spaCy model in this offline environment); they are identical in both runs. The new file alone goes from 9 failed, 11 passed to 20 passed.
ruff format --check and ruff check are clean on both files.
- 主要语言
- HTML
- 星标
- 15.7k
- 派生
- 1.4k
- 平均合并
- 3 天 8 小时
- 30 天内合并 PR
- 77
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
Unstructured-IO/unstructured 的其他 Issue
-
CoordinateSystem cannot be compared or converted without a caller-set `orientation`可能已有人在做 @Eapp1e 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
Unstructured-IO/unstructured#4614 ·
维护者通常 1 天内回复
-
bug(cleaners): translate_text passes text instead of chunk in attention window loop可能已有人在做 @gunjan-l 于 12 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 82/100
Unstructured-IO/unstructured#4558 · 1 条评论 ·
维护者通常 1 天内回复
-
extract_text_before/extract_text_after raise UnboundLocalError when the pattern is absent可能已有人在做 @ayam04 于 72 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
Unstructured-IO/unstructured#4427 · 3 条评论 ·
维护者通常 1 天内回复
-
Large HTML documents cannot be partitioned using the partition_html function.可能已有人在做 @CrepuscularIRIS 于 198 天前认领。 未关闭bug
难度 1/5 1 小时以内 新手友好度 72/100
Unstructured-IO/unstructured#4289 · 1 条评论 ·
维护者通常 1 天内回复
-
bug/Dependency resolution discrepancy for numba可能重新可做 关联的 PR 已关闭且未合并。 未关闭bug
难度 2/5 1-3 小时 新手友好度 68/100
Unstructured-IO/unstructured#4216 · 4 条评论 · 7 个 reaction ·
维护者通常 1 天内回复
查看 Unstructured-IO/unstructured 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复
-
Bug: POWERTOOLS_IDEMPOTENCY_DISABLED set to an empty or non-boolean value crashes every invocation可能已有人在做 @lechakrawarthy 今天认领。 未关闭bug good first issue help wanted idempotency
难度 2/5 1-3 小时 新手友好度 85/100
aws-powertools/powertools-lambda-python#8728 · 2 条评论 ·
维护者通常 2 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 80/100
mnemosyne-oss/mnemosyne#1309 ·
维护者通常 1 天内回复
-
bug elsa 3 prio low
难度 2/5 1-3 小时 新手友好度 75/100
elsa-workflows/elsa-core#9154 ·
维护者通常 1 天内回复
-
integration: ring
难度 2/5 1-3 小时 新手友好度 85/100
home-assistant/core#189787 · 1 条评论 ·
维护者通常 1 天内回复