`Bytes.toString` drops a leading U+FEFF, so a string does not survive `fromString` and back

Open Beginner friendly
#155 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript
Domain
backend

Research direction

Start with the _Bytes_toString implementation that constructs TextDecoder("utf-8", { fatal: true }), then compare its behavior with Stream.textDecoder. Run the supplied reproduction and add coverage for leading U+FEFF bytes; done means Bytes.toString and Bytes.Decode.string preserve the leading character while invalid UTF-8 handling remains unchanged.

Written by the indexing model from the issue text.

Description

bug

Found against: gren 0.6.6, gren-lang/core 7.4.2, node 22
Reproduction: in https://github.com/gilramir/gren-bug-reports, 2026-09-14-bytes-tostring-feff; ./run.sh prints the run
below.

Summary

U+FEFF (ZERO WIDTH NO-BREAK SPACE, or the BYTE ORDER MARK / BOM)
is a character, and Bytes.fromString writes it as the three bytes
EF BB BF (UTF-8 encoding). Reading those bytes back loses it when it is the first character:

call result expected
Bytes.length (Bytes.fromString "\u{FEFF}a") 4 4
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a" False True
Bytes.toString of EF BB BF 61 Just "a" Just "\u{FEFF}a"
Bytes.toString of EF BB BF Just "" Just "\u{FEFF}"
Bytes.toString of EF BB BF EF BB BF Just "\u{FEFF}" Just "\u{FEFF}\u{FEFF}"
Bytes.toString of 61 EF BB BF Just "a\u{FEFF}" Just "a\u{FEFF}"
Bytes.Decode.string 4 of EF BB BF 61 Just "a" Just "\u{FEFF}a"

Bytes.Decode.string calls Bytes.toString on the bytes it read, so any
length-prefixed string field whose value starts with U+FEFF comes back one
character short, and a decoder that checks a re-encoding against its input
refuses it.

The cause is the decoder's options:

function _Bytes_toString(bytes) {
  var decoder = new TextDecoder("utf-8", { fatal: true });

TextDecoder, which the WHATWG Encoding Standard
defines, has an ignoreBOM
option, default false, and with it false the decoder treats EF BB BF at the start of its input as a
byte-order mark and removes it. Only the first one is removed, which is why a
second U+FEFF, or one after another character, survives.

Why Bytes.toString shoul keep it

A byte-order mark belongs to the start of a text file or a text stream, and a
library reading one should remove it. core already has that library:
Stream.textDecoder is a TextDecoderStream, and it removes EF BB BF at the
start of the stream and keeps a U+FEFF anywhere after it. Fixing
Bytes.toString does not change that.

Bytes.toString sits below that layer, and it cannot tell whether its bytes
are the start of a file. Most of the time they are not.

Other languages decode UTF-8 the same way: the character is kept, and removing
a mark is something the caller asks for.

decoding EF BB BF 61 result
Python 3.12 bytes.decode("utf-8") U+FEFF U+0061
Python 3.12 bytes.decode("utf-8-sig"), which asks for removal U+0061
Go 1.25 string(b) U+FEFF U+0061
Java 21 new String(b, StandardCharsets.UTF_8) U+FEFF U+0061
Node 22 Buffer.toString("utf8") U+FEFF U+0061
Node 22 fs.readFileSync(path, "utf8") U+FEFF U+0061
new TextDecoder("utf-8", { ignoreBOM: true }) U+FEFF U+0061
new TextDecoder("utf-8"), what Bytes.toString uses U+0061

TextDecoder's default comes from the Encoding Standard, which decodes
whole web resources, where a leading mark is expected. That is the same job
Stream.textDecoder does, and not the job Bytes.toString does.

Reproduction

$ node app
Bytes.fromString "\u{FEFF}a" has length 4
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a": False
toString [EF BB BF 61]:          Just [97]
toString [EF BB BF]:             Just []
toString [EF BB BF EF BB BF]:    Just [65279]
toString [61 EF BB BF]:          Just [97, 65279]
Decode.string 4 [EF BB BF 61]:   Just [97]

The lists are the decoded strings' code points.

Dominant language
JavaScript
Stars
48
Forks
14
Avg merge
4h 14m
Merged PRs (30d)
1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gren-lang/core

All issues in gren-lang/core

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.