sql-pg: pgvector columns fail with "Invalid UTF-8 in text value"

Aperta
  • Qualcosa non va?
#10,420 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@effect-bot ci sta già lavorando.

Dal 6/11/2026.

  • #10422 di @effect-bot — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug
What version of Effect is running?

4.1.2

What steps can reproduce the bug?

Versions used

  • effect 4.1.2
  • @effect/sql-pg 4.1.2
  • Node.js 26, PostgreSQL 17.10 with pgvector 0.8.6 (pgvector/pgvector:pg17 image)

To reproduce

Run Postgres with pgvector:

docker run --rm -e POSTGRES_PASSWORD=password -p 55432:5432 pgvector/pgvector:pg17

Then run:

import { NodeRuntime } from '@effect/platform-node'
import { PgClient } from '@effect/sql-pg'
import { Effect, Redacted } from 'effect'
import { SqlClient } from 'effect/sql'

const program = Effect.gen(function* () {
  const sql = yield* SqlClient.SqlClient

  yield* sql`CREATE EXTENSION IF NOT EXISTS vector`

  const pid = () =>
    sql<{ pid: number }>`
      SELECT
        pg_backend_pid() AS pid
    `.pipe(Effect.map(rows => rows[0]!.pid))
  const before = yield* pid()

  // 1. Most vectors are not valid UTF-8, so the query fails
  const failure = yield* sql`
    SELECT
      '[1,2,3]'::vector AS embedding
  `.pipe(Effect.flip)
  console.log('1.', failure.message, failure.reason.cause)

  // 2. The physical connection was dropped
  console.log(`2. backend pid before: ${before}, after: ${yield* pid()}`)

  // 3. Some vectors happen to be valid UTF-8, so they succeed and return their binary representation as a string
  const garbled = yield* sql`
    SELECT
      '[0]'::vector AS v,
      '[1,2,3]'::halfvec AS h
  `
  console.log('3.', garbled[0])
})

program.pipe(
  Effect.provide(
    PgClient.layer({
      url: Redacted.make('postgres://postgres:password@localhost:55432/postgres'),
      maxConnections: 1,
    }),
  ),
  NodeRuntime.runMain,
)

Output:

1. PgConnection: Failed to decode row { message: 'Invalid UTF-8 in text value', _tag: 'PgTypesCodecError' }
2. backend pid before: 70, after: 71
3. {
  v: '\x00\x01\x00\x00\x00\x00\x00\x00',
  h: '\x00\x03\x00\x00<\x00@\x00B\x00'
}

Expected behavior

Would have appreciated a more actionable message mentioning:

  • the table/column causing the issue
  • column not having registered codec
  • that text decoding was a fallback

Accidentally valid UTF-8 succeeding here feels like a weird edge case. Failing early from a lack of codec would mean less surprises.

Workaround

Workaround we used:

SELECT embedding::text AS embedding FROM ...

Claude also came up with this:

Registering a codec after looking up the OID at runtime.

const types = PgTypes.makeRegistry() // passed as PgClient.layer({ types })

const [{ oid }] =
  yield *
  sql<{ oid: number }>`
    SELECT
      oid::int AS oid
    FROM
      pg_type
    WHERE
      typname = 'vector'
  `
types.register(oid, {
  encode: () => Result.fail(new PgTypes.CodecError({ message: 'Not implemented' })),
  decode: bytes => {
    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
    const dim = view.getUint16(0)
    return Result.succeed(Array.from({ length: dim }, (_, i) => view.getFloat32(4 + i * 4)))
  },
})
// SELECT '[1,2,3]'::vector → [1, 2, 3]

Related: #8237, #8240, #8304.

Lingua principale
TypeScript
Stelle
16.7k
Fork
808
Merge medio
11h 40m
PR unite (30g)
827

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Effect-TS/effect

Tutte le issue di Effect-TS/effect

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.