Incorrect "Fix bad UTF-8 char " step in Dockerfile actually corrupts the input
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- dockerfile
- Domain
- build-system
Research direction
Start with the tpcds/Dockerfile lines 17–19 in the linked datafusion-benchmarks revision and inspect the iconv command applied to tpcds.dst. Verify the source bytes are already UTF-8, then confirm the Docker build no longer double-encodes characters such as Ô. Done means the generated benchmark input preserves the original UTF-8 content.
Written by the indexing model from the issue text.
Description
I spent a large amount of time confused about this and wanted to file an issue
I found it in the context of
The Dockerfile in this repo implies that the DSGen data generator for TPCH has incorrect input:
https://github.com/apache/datafusion-benchmarks/blob/cb12c981e6608e0f2dcf919956ada8f1f1622d72/tpcds/Dockerfile#L17-L19
However, what that command actually does is corrupt the file
For example,t he originl data has a CÔTE in it: (\x43 \xc3 \x94 \x54 \x45 in utf8)
grep -n "IVOIRE" tpcds.dst | head -1 | xxd | head -3
echo "---"…)
⎿ 00000000: 3634 393a 6164 6420 2822 43c3 9454 4520 649:add ("C..TE
00000010: 4427 4956 4f49 5245 223a 3129 3b0a D'IVOIRE":1);.
iconv -f ISO-8859-14 -t UTF-8
Tells iconv "treat the input as ISO-8859-14 single-byte data." But the file was already UTF-8. So iconv took each existing UTF-8 byte (e.g. C3 and 94) as if each were a separate Latin character, and re-encoded each one into UTF-8.
This results in
tpcds.dst (original) — already correct UTF-8:
- C3 94 = the valid UTF-8 encoding for Ô (U+00D4 LATIN CAPITAL LETTER O WITH CIRCUMFLEX)
tpcds.dst2 (after iconv) — corrupted (double-encoded):
- C3 83 C2 94 = two characters: Ã (U+00C3) + a control character (U+0094)
- Dominant language
- Python
- Stars
- 23
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-benchmarks
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/datafusion-benchmarks#30 · 4 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
apache/datafusion-benchmarks#22 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
All issues in apache/datafusion-benchmarks
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100