sql-pg: pgvector columns fail with "Invalid UTF-8 in text value"

Abierto
  • ¿Algo no está bien?
#10,420 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@effect-bot ya está trabajando en esto.

Desde el 6/11/2026.

  • #10422 de @effect-bot — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug
What version of Effect is running?

4.1.2

What steps can reproduce the bug?

Versions used

  • effect 4.1.2
  • @effect/sql-pg 4.1.2
  • Node.js 26, PostgreSQL 17.10 with pgvector 0.8.6 (pgvector/pgvector:pg17 image)

To reproduce

Run Postgres with pgvector:

docker run --rm -e POSTGRES_PASSWORD=password -p 55432:5432 pgvector/pgvector:pg17

Then run:

import { NodeRuntime } from '@effect/platform-node'
import { PgClient } from '@effect/sql-pg'
import { Effect, Redacted } from 'effect'
import { SqlClient } from 'effect/sql'

const program = Effect.gen(function* () {
  const sql = yield* SqlClient.SqlClient

  yield* sql`CREATE EXTENSION IF NOT EXISTS vector`

  const pid = () =>
    sql<{ pid: number }>`
      SELECT
        pg_backend_pid() AS pid
    `.pipe(Effect.map(rows => rows[0]!.pid))
  const before = yield* pid()

  // 1. Most vectors are not valid UTF-8, so the query fails
  const failure = yield* sql`
    SELECT
      '[1,2,3]'::vector AS embedding
  `.pipe(Effect.flip)
  console.log('1.', failure.message, failure.reason.cause)

  // 2. The physical connection was dropped
  console.log(`2. backend pid before: ${before}, after: ${yield* pid()}`)

  // 3. Some vectors happen to be valid UTF-8, so they succeed and return their binary representation as a string
  const garbled = yield* sql`
    SELECT
      '[0]'::vector AS v,
      '[1,2,3]'::halfvec AS h
  `
  console.log('3.', garbled[0])
})

program.pipe(
  Effect.provide(
    PgClient.layer({
      url: Redacted.make('postgres://postgres:password@localhost:55432/postgres'),
      maxConnections: 1,
    }),
  ),
  NodeRuntime.runMain,
)

Output:

1. PgConnection: Failed to decode row { message: 'Invalid UTF-8 in text value', _tag: 'PgTypesCodecError' }
2. backend pid before: 70, after: 71
3. {
  v: '\x00\x01\x00\x00\x00\x00\x00\x00',
  h: '\x00\x03\x00\x00<\x00@\x00B\x00'
}

Expected behavior

Would have appreciated a more actionable message mentioning:

  • the table/column causing the issue
  • column not having registered codec
  • that text decoding was a fallback

Accidentally valid UTF-8 succeeding here feels like a weird edge case. Failing early from a lack of codec would mean less surprises.

Workaround

Workaround we used:

SELECT embedding::text AS embedding FROM ...

Claude also came up with this:

Registering a codec after looking up the OID at runtime.

const types = PgTypes.makeRegistry() // passed as PgClient.layer({ types })

const [{ oid }] =
  yield *
  sql<{ oid: number }>`
    SELECT
      oid::int AS oid
    FROM
      pg_type
    WHERE
      typname = 'vector'
  `
types.register(oid, {
  encode: () => Result.fail(new PgTypes.CodecError({ message: 'Not implemented' })),
  decode: bytes => {
    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
    const dim = view.getUint16(0)
    return Result.succeed(Array.from({ length: dim }, (_, i) => view.getFloat32(4 + i * 4)))
  },
})
// SELECT '[1,2,3]'::vector → [1, 2, 3]

Related: #8237, #8240, #8304.

Lenguaje dominante
TypeScript
Estrellas
16.7k
Forks
808
Merge medio
12 h 16 min
PR fusionados (30 d)
809

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Effect-TS/effect

Todos los issues de Effect-TS/effect

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.