xataio/pgstream

Make number of target connections configurable

Open

#1,047 opened on Jul 28, 2026

 (0 comments) (1 reaction) (0 assignees)Go (66 forks)auto 404
enhancementgood first issue

Repository metrics

Stars
 (1,151 stars)
PR merge metrics
 (PR metrics pending)

Description

Every Postgres pool in pgstream is capped at a compile-time constant of 50, assigned in the shared constructor:

// internal/postgres/pg_conn_pool.go
const MaxConns = 50            // :22

pgCfg, err := pgxpool.ParseConfig(escapedURL)
...
pgCfg.MaxConns = MaxConns      // :33 — after ParseConfig, before the opts loop

WithMaxConnections exists (added in e773a55, Nov 2025) but has exactly one caller: the snapshot data generator (pkg/snapshot/generator/postgres/data/pg_snapshot_generator.go:77). The target writer calls NewConnPool(ctx, config.URL) bare (pkg/wal/processor/postgres/postgres_writer.go:61,65), and no YAML key or environment variable reaches it.

Three consequences:

  1. Target write concurrency is not tunable. The bulk-ingest writer derives its global COPY budget from the constant:

    // pkg/wal/processor/postgres/postgres_bulk_ingest_writer.go:59
    copyBudget: synclib.NewWeightedSemaphore(int64(pglib.MaxConns - copyBudgetReserve)),
    

    That is a hard ceiling of 45 concurrent COPYs across all tables, regardless of how large the target is. It binds at the shipped defaults, not just at exotic settings: copy_workers defaults to 8 (cmd/config/config.go:25), so six tables in flight already saturate it.

  2. pool_max_conns in the connection URL is silently discarded. pgx documents and parses it (pgxpool/pool.go:345,370); line 33 then overwrites the result. No error, no warning — the setting simply has no effect.

  3. The read and write sides are asymmetric. Source read parallelism is configurable (snapshot.data.max_connections / PGSTREAM_POSTGRES_SNAPSHOT_MAX_CONNECTIONS, documented in docs/configuration.md:230). The write side has no equivalent.

Contributor guide