xataio/pgstream

Make number of target connections configurable

Fechada

#1.047 aberto em 28 de jul. de 2026

 (0 comentário) (1 reação) (0 responsável)Go (66 forks)auto 404
enhancementgood first issue

Métricas do repositório

Stars
 (1.151 estrelas)
Métricas de merge de PR
 (Métricas PR pendentes)

Description

Every Postgres pool in pgstream is capped at a compile-time constant of 50, assigned in the shared constructor:

// internal/postgres/pg_conn_pool.go
const MaxConns = 50            // :22

pgCfg, err := pgxpool.ParseConfig(escapedURL)
...
pgCfg.MaxConns = MaxConns      // :33 — after ParseConfig, before the opts loop

WithMaxConnections exists (added in e773a55, Nov 2025) but has exactly one caller: the snapshot data generator (pkg/snapshot/generator/postgres/data/pg_snapshot_generator.go:77). The target writer calls NewConnPool(ctx, config.URL) bare (pkg/wal/processor/postgres/postgres_writer.go:61,65), and no YAML key or environment variable reaches it.

Three consequences:

  1. Target write concurrency is not tunable. The bulk-ingest writer derives its global COPY budget from the constant:

    // pkg/wal/processor/postgres/postgres_bulk_ingest_writer.go:59
    copyBudget: synclib.NewWeightedSemaphore(int64(pglib.MaxConns - copyBudgetReserve)),
    

    That is a hard ceiling of 45 concurrent COPYs across all tables, regardless of how large the target is. It binds at the shipped defaults, not just at exotic settings: copy_workers defaults to 8 (cmd/config/config.go:25), so six tables in flight already saturate it.

  2. pool_max_conns in the connection URL is silently discarded. pgx documents and parses it (pgxpool/pool.go:345,370); line 33 then overwrites the result. No error, no warning — the setting simply has no effect.

  3. The read and write sides are asymmetric. Source read parallelism is configurable (snapshot.data.max_connections / PGSTREAM_POSTGRES_SNAPSHOT_MAX_CONNECTIONS, documented in docs/configuration.md:230). The write side has no equivalent.

Guia do colaborador