[SNMP poller] Collects cap at pool-width per second and about a quarter of the fleet is never polled when more devices fall due per second than pool-width

Open
  • Is something wrong?
#899 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java, spring-boot
Domain
backend, networking, observability

Research direction

Start in InterfaceSnapshotPoller: tick is scheduled with scheduleWithFixedDelay every 1 second and iterates registrations.values() in map order. When budget.tryAcquire() fails a due registration is only marked deferred and is not retried until a later tick, so collects cap at pool-width per second. Confirm with the devices-2500-pw32 metrics and store query in the issue. Done when a 2500-device 60s fleet is fully polled at default or documented pool-width, deferred does not grow unbounded, and the reference sizing rule matches the real constraint.

Written by the indexing model from the issue text.

Description

bug priority: high
  • Build / Version: ec65137f on branch spec/snmp-metrics (unreleased SNMP metrics phase 1)
  • Device & OS: libvirt guest riptide-snmp-bench, 4 vCPU, 16 GiB, Debian 13, Temurin 25.0.4.1
  • Browser / Client: n/a, server process; agents are nl6 v0.32.0 cisco_catalyst_9500 (48 interfaces, SNMPv2c)
  • Severity: Critical, a subset of devices silently produces no series at all while every health counter reads zero failures; workaround exists
  • Priority: P1, ships silent data absence at the spec's own target fleet size
  • User state: n/a

Preconditions

  • 2,500 devices in the inventory, each a /32 agent entry with a v2c credential set and an exporters entry with poll: always
  • Polling profile refresh-interval=PT60S, timeout=500, retries=1, collect[0]=if-mib-interfaces
  • riptide.snmp.poll.pool-width=32 (the default is 4), suspect-pool-width=8, max-exporters=8192
  • Remote-write sink enabled against VictoriaMetrics; agents answer every request within 40 ms

Steps to reproduce

  1. Start riptide with the inventory above and wait 5 minutes for the cold-start burst to settle.
  2. Read snmp_collects from /metrics twice, 60 seconds apart.
  3. Read snmp_poller_deferred at the same two instants.
  4. In the store, run count(count by (exporter) (ifHCInOctets{tenant="bench"})).

Expected result

About 2,500 collects per minute, snmp_poller_deferred flat once the burst has drained, and 2,500 exporters with series.

Actual result

1,920 collects per minute, exactly 32 per second, with snmp_poller_inFlight reading 0 at every sample, walk p99 38 ms and load average 0.0; snmp_poller_deferred grows by about 615 per second indefinitely (240,902 to 277,799 in one minute; 1,219,343 after 27 minutes). Only 1,917 of 2,500 exporters ever had a series, and the same 1,917 stay fresh minute after minute, so about 580 devices are never polled. snmp_collects_failed, snmp_poller_collectsFailed, metrics_sink_droppedSamples and metrics_sink_failedSamples all read 0 throughout. At 1,000 devices (16.7 due per second) the same configuration polls every device.

Evidence

  • Poller /metrics snapshots and the store query above, recorded in the benchmark run log (session scratchpad bench/RUNLOG.md, step devices-2500-pw32), and the benchmark-capture artifact devices-2500-pw32 (JFR, procfs, pcap, Prometheus dump)
  • Code observed in the same build: InterfaceSnapshotPoller schedules tick with scheduleWithFixedDelay(..., 0, 1, SECONDS); in tick, a due registration whose budget.tryAcquire() fails marks deferred and continues, and nothing retries it before the next tick; the tick iterates registrations.values() in map order every second
  • Same fleet with pool-width=128: 2,541 collects per minute, deferred flat, 2,500 of 2,500 exporters fresh

Workaround

Set riptide.snmp.poll.pool-width to at least the fleet's walks per second (devices divided by the interval in seconds), for example 128 for 5,000 devices at 60 seconds. The reference page's sizing rule (permits about walks per second times walk latency) understates this and needs correcting together with the fix.

Dominant language
Java
Stars
2
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Riptide-Labs/riptide

All issues in Riptide-Labs/riptide

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.