Sidecar metrics collection over plugin gRPC has no deadline — one stuck operation hangs the whole instance /metrics
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- go, grpc
- Domain
- backend, observability
Research direction
Trace the instance sidecar's plugin gRPC metrics collect path and reproduce the issue with a stuck plugin operation, then inspect how the /metrics endpoint handles collector errors. Verify that a per-collect timeout lets the endpoint return the remaining metrics and records an error when the plugin call stalls, while healthy plugin metrics continue to work.
Written by the indexing model from the issue text.
Description
Environment
CloudNativePG 1.30.0, plugin-barman-cloud v0.14.0, PostgreSQL 17/18 clusters, VersityGW (posix) S3 endpoint.
What happens
The instance sidecar collects plugin metrics over the plugin gRPC with no deadline on the collect path. Consequence: if any plugin operation gets stuck (in my case a retention/catalog-maintenance run that never completes against a posix-backend S3 gateway — filed separately), the next scrape blocks on the collector and the entire /metrics endpoint of that instance hangs indefinitely. Prometheus marks the target down and every metric of that instance disappears, not just the plugin's own collectors.
Observed in production: the primary instance's exporter dark for 6.7 hours while the database itself was healthy — WAL-archiver and backup-staleness alerting was blind precisely on the instance where it matters.
Expected
- A per-collect deadline (a few seconds) on the plugin metrics gRPC call.
- On timeout/error: return the rest of the metrics plus an error counter (e.g.
..._collector_errors_total), instead of hanging the whole endpoint. A misbehaving plugin should degrade its own metrics, never the instance's.
Reproduction sketch
- ObjectStore against VersityGW (posix backend) with any
retentionPolicyset. - Wait for a retention run to start (~30 min cadence) — it never completes against this backend.
- Scrape the instance sidecar's metrics port: the request hangs until the client gives up;
upgoes to 0 for the instance.
Companion issue describes the retention hang itself; this one is about the missing deadline that turns any such hang into a full metrics outage.
- Dominant language
- Go
- Stars
- 192
- Forks
- 75
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 17
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudnative-pg/plugin-barman-cloud
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
cloudnative-pg/plugin-barman-cloud#1104 · 4 reactions ·
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
cloudnative-pg/plugin-barman-cloud#1113 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
All issues in cloudnative-pg/plugin-barman-cloud
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Bob Shell support Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
santhosh-tekuri/jsonschema#276 ·