[Remote-write sink] 83,560 samples dropped 43 seconds after start when 5,000 poll: always devices all come due at once

Open
  • Is something wrong?
#900 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java
Domain
backend, observability, performance

Research direction

Start with InterfaceSnapshotPoller.Registration and its nextWalkNanos initialization, then trace how the first sweep feeds the remote-write queue and produces metrics_sink_droppedSamples. Reproduce the devices-5000-pw128 benchmark using the steps and compare the journal, /metrics, and bench/RUNLOG.md evidence. Done means the 5,000-device startup sweep completes without dropped samples while steady-state behavior remains intact.

Written by the indexing model from the issue text.

Description

bug priority: medium
  • Build / Version: ec65137f on branch spec/snmp-metrics (unreleased SNMP metrics phase 1)
  • Device & OS: libvirt guest riptide-snmp-bench, 4 vCPU, 16 GiB, Debian 13, Temurin 25.0.4.1
  • Browser / Client: n/a, server process; agents are nl6 v0.32.0 cisco_catalyst_9500 (48 interfaces, SNMPv2c); store is VictoriaMetrics v1.152.0 on the same LAN
  • Severity: Major, about 2.3 percent of the first sweep's samples are lost on every start or restart; workaround exists
  • Priority: P2, bounded to start-up, but it fires on every restart at the spec's target fleet size

Preconditions

  • 5,000 devices in the inventory, each a /32 agent entry with a v2c credential set and an exporters entry with poll: always
  • Polling profile refresh-interval=PT60S, collect[0]=if-mib-interfaces; riptide.snmp.poll.pool-width=128
  • Remote-write sink at defaults: batch.max-samples=5000, batch.max-latency=2s, queue-capacity=2000000; the store accepts every batch (204) and ingests 60,000 rows per second in steady state

Steps to reproduce

  1. Restart riptide with the inventory above.
  2. Watch metrics_sink_droppedSamples on /metrics for the first two minutes.
  3. Read the riptide journal for the sink's warnings.

Expected result

No samples dropped: the first sweep of 5,000 devices produces 3.6 million samples that fit an interval of 60 seconds, and the store is far from saturated.

Actual result

metrics_sink_droppedSamples jumps to 83,560 at 12:25:01Z, 43 seconds after the 12:24:18Z restart, with one WARN Dropping samples (queue full because ...) from thread snmp-walk-io-94, and then stays flat for the rest of the run (0 further drops in 30 minutes; steady state 61,000 samples per second sent, 0 failed). snmp_poller_deferred stays at 0 per minute afterwards. The same restart with queue-capacity=4000000 (and a 30 second interval, 256 permits) dropped nothing.

Evidence

  • /metrics snapshots, the journal line and the store's vm_rows_inserted_total rate recorded in the benchmark run log (session scratchpad bench/RUNLOG.md, step devices-5000-pw128), and the benchmark-capture artifact devices-5000-pw128
  • Code observed in the same build: InterfaceSnapshotPoller.Registration sets nextWalkNanos = nowNanos on construction, so every inventory-registered device is due on the first tick; only re-walks are spread by the address-derived offset (nextWalkAt), so the first sweep runs as fast as the permits allow, about 40 seconds for 5,000 devices at 128 permits, offering about 90,000 samples per second to a queue that drains at the sink's flush rate

Workaround

Set riptide.metrics.remote-write.queue-capacity to at least one full sweep, devices times 720 samples (3,600,000 for 5,000 devices); 4,000,000 absorbed the burst in the same rig. The loss is confined to the first sweep after a start.

Dominant language
Java
Stars
2
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Riptide-Labs/riptide

All issues in Riptide-Labs/riptide

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.