BReg: retry a refused first database connection in one-shot operations

Open Beginner friendly
#2,765 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
65/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
postgresql, rust
Domain
databases

Research direction

Start at connect_dedicated in crates/registry-breg/src/postgres/interlock.rs and read how the one-shot commands (status, doctor, plan, apply, instance-claim) call it, plus how the sqlx connect error exposes a refused/unreachable connection. Pick a bounded window (the issue suggests 10 s) and only retry errors raised before any statement is sent. A test that a refused first connect is retried while a mid-statement failure is not is what "done" looks like; note the issue leaves the chart-based Option 2 open, so confirm the approach before writing code.

Written by the indexing model from the issue text.

Description

area:breg enhancement

Operation Jobs in the Helm chart run one attempt (backoffLimit: 0), so the operator's reconcile-first refusal decides what happens next rather than a Kubernetes retry. With one attempt, a failed first database connection fails the whole operation, and the operator has to rerun it.

On a single-server K3s install, a newly started Pod labelled registrystack.org/database-client: "true" got Connection refused from PostgreSQL for about 600 ms before its first successful TCP connection. The K3s network policy controller admits a new Pod's address shortly after the container starts. No BReg Job failed in that run, probably because the stage-operation-inputs init container delays the bregctl container. That margin is a matter of timing, not something the design guarantees.

status and the maintenance interlock connect through connect_dedicated in crates/registry-breg/src/postgres/interlock.rs, which makes one attempt and has no retry.

Options:

  1. Give the one-shot maintenance commands (status, doctor, plan, apply, instance-claim) a short bounded retry, for example up to 10 s, that covers only connection refused or unreachable before any statement is sent. Nothing has changed in the database at that point, so the retry cannot repeat work.
  2. Add a wait for the database port to the operation Job's init container. This stays in the chart and covers only Kubernetes installs.

Option 1 also helps any other orchestrator that starts a fresh process next to a fresh network policy.

Dominant language
Rust
Stars
2
Forks
0
Avg merge
9h 48m
Merged PRs (30d)
151

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from registrystack/registry-stack

All issues in registrystack/registry-stack

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.