Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Socket.readerString corrupts UTF-8 characters split across TCP chunks

Closed
  • Is something wrong?
#9,486 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #9487 by @jayhemnani9910 — merged

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
nodejs, typescript
Domain
networking

Research direction

Start with the reproduction in the issue and inspect packages/effect/src/socket/Socket.ts:268, where Socket.readerString decodes each item independently. Determine whether streaming decoding should apply only to byte-stream transports or also to WebSocket binary messages, especially for BOM handling, truncated input, and normal close. Add regression tests covering UTF-8 characters split across TCP chunks and the agreed WebSocket behavior.

Written by the indexing model from the issue text.

Description

What version of Effect is running?

effect 4.0.5, @effect/platform-node 4.0.5 (same code on main 7f32606ba)

What steps can reproduce the bug?

A TCP server sends café ☕ done one byte at a time. The client reads it with Socket.readerString over NodeSocket.makeNet:

import * as net from "node:net"
import { Effect } from "effect"
import * as Socket from "effect/socket/Socket"
import * as NodeSocket from "@effect/platform-node/NodeSocket"

const bytes = Buffer.from("café ☕ done", "utf8")
const server = net.createServer(async (c) => {
  for (let i = 0; i < bytes.length; i++) {
    c.write(bytes.subarray(i, i + 1))
    await new Promise((r) => setTimeout(r, 5))
  }
  c.end()
})
await new Promise<void>((r) => server.listen(0, "127.0.0.1", r))
const { port } = server.address() as net.AddressInfo

const text = await Effect.gen(function*() {
  const socket = yield* NodeSocket.makeNet({ host: "127.0.0.1", port })
  const pull = yield* Socket.readerString(socket)
  let text = ""
  while (true) {
    const r = yield* Effect.result(pull)
    if (r._tag === "Failure") return text
    text += r.success.join("")
  }
}).pipe(Effect.scoped, Effect.runPromise)
console.log(JSON.stringify(text))
server.close()
What is the expected behavior?

"café ☕ done"

What do you see instead?

"caf�� ��� done"

Additional information

readerString creates one TextDecoder per acquisition but calls decoder.decode(item) without { stream: true } (packages/effect/src/socket/Socket.ts:268), so every chunk is decoded on its own. TCP chunk boundaries are arbitrary, so any multi-byte character can land on one. toChannelString uses the same reader, and DevTools runs NDJSON over it.

Decoding in streaming mode fixes TCP, but it changes WebSocket binary messages, which are complete on their own:

  • A UTF-8 BOM is stripped from every message today; in streaming mode only from the first. Two binary messages EF BB BF 7B 7D give ["{}", "{}"] today and ["{}", "{}"] in streaming mode.
  • A truncated message's leftover bytes would join the next message instead of becoming U+FFFD.

I have a fix that decodes in streaming mode and, like #9436 for Channel.decodeText, flushes incomplete trailing bytes as U+FFFD on a normal close (code 1000), with tests. Is that the behaviour you want for WebSocket too, or should streaming decode apply only to byte-stream transports? Happy to open a PR either way.

Dominant language
TypeScript
Stars
16.7k
Forks
808
Avg merge
10h 22m
Merged PRs (30d)
771

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Effect-TS/effect

All issues in Effect-TS/effect

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.