CSVToDocument row mode drops a real extra_columns field on ragged rows
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start at the CSVToDocument entry point in haystack.components.converters.csv and trace row-mode handling for ragged CSV rows, including the rest key used by csv.DictReader. Add a regression test with a real extra_columns header and overflow values, then verify that the original field and overflow metadata are both preserved; compare with the ragged-row fix in #11944.
Written by the indexing model from the issue text.
Description
CSVToDocument(conversion_mode="row") uses extra_columns as the csv.DictReader rest key. If the CSV also has a column named extra_columns, a row with surplus fields overwrites that column before the converter sees it. The output document keeps the overflow values but silently loses the original field.
from haystack.components.converters.csv import CSVToDocument
from haystack.dataclasses import ByteStream
source = ByteStream(data=b"text,extra_columns\r\nhello,real value,overflow\r\n")
doc = CSVToDocument(conversion_mode="row").run(
sources=[source], content_column="text"
)["documents"][0]
print(doc.content, doc.meta)
# hello {'extra_columns': "['overflow']", 'row_number': 0}
I ran this twice on current main; both runs produced the same result. The original real value cannot be recovered from the document. extra_columns is a valid CSV header, and an extra trailing delimiter or unquoted comma is enough to trigger this in row mode. File mode is unaffected.
Could the rest key use a name that cannot collide with a CSV header, or could the converter separate surplus fields before building metadata? A regression test should cover a real extra_columns column alongside overflow values. This is related to the ragged-row fix in #11944, but that fix only tests a header without this name. I can prepare the patch if the maintainers want one.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 254
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepset-ai/haystack
-
MarkdownHeaderSplitter treats headings inside longer closing fences as headersPossibly taken @julian-risch claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
deepset-ai/haystack#12954 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
P3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepset-ai/haystack#12945 ·
Maintainers usually reply within 1 day
-
Sockets.__getattribute__ references _sockets but the actual attribute is _sockets_dict — the optimization at lines 127-132 is dead code due to a typoPossibly taken @julian-risch claimed this 3 days ago. OpenP2
Difficulty 1/5 Under an hour Newbie friendliness 91/100
deepset-ai/haystack#12939 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
P3
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
deepset-ai/haystack#12784 · 3 comments ·
Maintainers usually reply within 1 day
-
P2 type:documentation
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
deepset-ai/haystack#11104 · 1 comment ·
Maintainers usually reply within 1 day
All issues in deepset-ai/haystack
Similar issues
-
ACK_WAITING HELP_WANTED UPDATE_CS
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OWASP/CheatSheetSeries#2458 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
BasedHardware/omi#19711 ·
Maintainers usually reply within 1 day
-
Qwen3_5MoeModel no longer returns router_logits, breaking aux loss with output_router_logits=TrueOpen
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/transformers#49172 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
vllm-project/vllm-metal#885 ·
Maintainers usually reply within 1 day