Alert on backend errors, peer fallbacks, ring size, readiness and disk-tier damage
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- helm, kubernetes, prometheus, rust
- Domain
- cloud, devops, documentation, observability
Research direction
Start with deploy/helm/pacer/templates/prometheusrule.yaml and compare the existing alerts with the listed daemon metrics. Use docs/runbooks/daemon-oom.md as the runbook pattern, establish thresholds from a healthy run, and account for the optional kube-state-metrics dependency. Done means the proposed failure-mode alerts have runbooks and docs/helm/monitoring.md lists their expressions.
Written by the indexing model from the issue text.
Description
The chart's PrometheusRule (deploy/helm/pacer/templates/prometheusrule.yaml) has five alerts, all on memory and OOM. The daemon already exports metrics for other failure modes that no alert covers.
Proposed alerts, each with a runbook entry in docs/runbooks/ like daemon-oom.md:
- backend reads failing or retrying:
pacer_backend_read_failures_total,pacer_backend_read_retries_total; - peers unavailable: a sustained rate of
pacer_peer_fallbacks_total; - ring smaller than the fleet:
pacer_ring_membersbelow the DaemonSet's desired count (needs kube-state-metrics, which none of the current alerts depend on, so it should be optional); - a daemon not ready for longer than a set time, and a DaemonSet rollout that has stopped progressing;
- disk-tier damage:
pacer_chunk_store_corrupt_slots_total,pacer_chunk_store_io_errors_total; - RDMA errors:
pacer_rdma_cq_errors_total,pacer_delivery_write_failures_total.
Thresholds need a baseline from a healthy run before they are set, and docs/helm/monitoring.md should list the new alerts with their expressions.
- Dominant language
- Rust
- Stars
- 16
- Forks
- 1
- Avg merge
- 7h 19m
- Merged PRs (30d)
- 29
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws-samples/sample-pacer
-
Measure requester-mode hit rate with diskTier=store — the disk slot may drop every ETag witnessOpen
Difficulty 3/5 1-2 days Newbie friendliness 68/100
aws-samples/sample-pacer#73 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 42/100
aws-samples/sample-pacer#72 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
aws-samples/sample-pacer#71 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 68/100
aws-samples/sample-pacer#68 ·
Maintainers usually reply within 1 day
-
enhancement rust
Difficulty 5/5 Over a week Newbie friendliness 45/100
aws-samples/sample-pacer#67 ·
Maintainers usually reply within 1 day
All issues in aws-samples/sample-pacer
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
DayuanJiang/PecoFence#139 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
nearai/compose-manager#73 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
akitaonrails/ai-memory#1664 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day