xataio/pgstream

Make number of target connections configurable

Geschlossen

#1.047 geöffnet am 28.07.2026

 (0 Kommentare) (1 Reaktion) (0 zugewiesene Personen)Go (66 Forks)auto 404
enhancementgood first issue

Repository-Metriken

Stars
 (1.151 Sterne)
PR-Merge-Metriken
 (PR-Metriken ausstehend)

Beschreibung

Every Postgres pool in pgstream is capped at a compile-time constant of 50, assigned in the shared constructor:

// internal/postgres/pg_conn_pool.go
const MaxConns = 50            // :22

pgCfg, err := pgxpool.ParseConfig(escapedURL)
...
pgCfg.MaxConns = MaxConns      // :33 — after ParseConfig, before the opts loop

WithMaxConnections exists (added in e773a55, Nov 2025) but has exactly one caller: the snapshot data generator (pkg/snapshot/generator/postgres/data/pg_snapshot_generator.go:77). The target writer calls NewConnPool(ctx, config.URL) bare (pkg/wal/processor/postgres/postgres_writer.go:61,65), and no YAML key or environment variable reaches it.

Three consequences:

  1. Target write concurrency is not tunable. The bulk-ingest writer derives its global COPY budget from the constant:

    // pkg/wal/processor/postgres/postgres_bulk_ingest_writer.go:59
    copyBudget: synclib.NewWeightedSemaphore(int64(pglib.MaxConns - copyBudgetReserve)),
    

    That is a hard ceiling of 45 concurrent COPYs across all tables, regardless of how large the target is. It binds at the shipped defaults, not just at exotic settings: copy_workers defaults to 8 (cmd/config/config.go:25), so six tables in flight already saturate it.

  2. pool_max_conns in the connection URL is silently discarded. pgx documents and parses it (pgxpool/pool.go:345,370); line 33 then overwrites the result. No error, no warning — the setting simply has no effect.

  3. The read and write sides are asymmetric. Source read parallelism is configurable (snapshot.data.max_connections / PGSTREAM_POSTGRES_SNAPSHOT_MAX_CONNECTIONS, documented in docs/configuration.md:230). The write side has no equivalent.

Contributor Guide