xataio/pgstream

Make number of target connections configurable

已關閉

#1,047 建立於 2026年7月28日

 (0 則留言) (1 個反應) (0 位負責人)Go (66 個分叉)auto 404
enhancementgood first issue

倉庫指標

星標
 (1,151 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Every Postgres pool in pgstream is capped at a compile-time constant of 50, assigned in the shared constructor:

// internal/postgres/pg_conn_pool.go
const MaxConns = 50            // :22

pgCfg, err := pgxpool.ParseConfig(escapedURL)
...
pgCfg.MaxConns = MaxConns      // :33 — after ParseConfig, before the opts loop

WithMaxConnections exists (added in e773a55, Nov 2025) but has exactly one caller: the snapshot data generator (pkg/snapshot/generator/postgres/data/pg_snapshot_generator.go:77). The target writer calls NewConnPool(ctx, config.URL) bare (pkg/wal/processor/postgres/postgres_writer.go:61,65), and no YAML key or environment variable reaches it.

Three consequences:

  1. Target write concurrency is not tunable. The bulk-ingest writer derives its global COPY budget from the constant:

    // pkg/wal/processor/postgres/postgres_bulk_ingest_writer.go:59
    copyBudget: synclib.NewWeightedSemaphore(int64(pglib.MaxConns - copyBudgetReserve)),
    

    That is a hard ceiling of 45 concurrent COPYs across all tables, regardless of how large the target is. It binds at the shipped defaults, not just at exotic settings: copy_workers defaults to 8 (cmd/config/config.go:25), so six tables in flight already saturate it.

  2. pool_max_conns in the connection URL is silently discarded. pgx documents and parses it (pgxpool/pool.go:345,370); line 33 then overwrites the result. No error, no warning — the setting simply has no effect.

  3. The read and write sides are asymmetric. Source read parallelism is configurable (snapshot.data.max_connections / PGSTREAM_POSTGRES_SNAPSHOT_MAX_CONNECTIONS, documented in docs/configuration.md:230). The write side has no equivalent.

貢獻者指南