A file scan reads every file as UTF-8, whatever its Encoding field says
Maintainers usually reply within 1 day
@kz930 is already working on this.
Since Sep 19, 2026.
Assessment
This issue has not been assessed yet.
Description
What happened?
File Scan offers an Encoding field. FileScanSourceOpDesc stores it in its own encoding property, but FileScanSourceOpExec decodes with fileEncoding, the property inherited from ScanSourceOpDesc. The class carries @JsonIgnoreProperties(Array("limit", "offset", "fileEncoding")), so fileEncoding never survives serialization into the executor and is always its default, UTF_8.
Choosing any other charset therefore changes nothing. A UTF-16 file comes back decoded as UTF-8 rather than as its text.
Expected: the executor decodes with the charset the Encoding field names.
How to reproduce?
Deserialize a File Scan descriptor carrying "encoding":"UTF_16", write it out the way getPhysicalOp does, and read it back the way FileScanSourceOpExec does. fileEncoding comes back UTF_8, and the encoding the user chose is the only place UTF-16 survives.
In the UI: upload a UTF-16 text file, drop a File Scan on it, set Encoding to UTF_16 and run. The rows hold the file's bytes read as UTF-8, not its lines.
Version/Branch
1.4.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
2ab8ee0f2
What browsers are you seeing the problem on?
No response
Relevant log output
JSON = {"operatorType":"FileScan","dummyPropertyList":[],"encoding":"UTF_16","extract":false,"outputFileName":false,"attributeType":"string","attributeName":"line",...}
exec reads desc.fileEncoding = UTF_8
- Dominant language
- Scala
- Stars
- 316
- Forks
- 189
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 165
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/texera
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/texera#8682 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/texera#8300 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/texera#8235 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)"Openbug requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#6251 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21826 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
ergoplatform/ergo#2579 ·
Maintainers usually reply within 2 days
-
x:action/improve x:size/tiny x:type/content
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
typelevel/sbt-typelevel#929 ·