opensearch-project/OpenSearch

WLM: monitor-mode workload group still rejects requests with 429 on resource breach

Ouverte

#22 616 ouverte le 31 juil. 2026

 (5 commentaires) (0 réaction) (0 personne assignée)Java (1 505 forks)batch import
Search:Resiliencygood first issue

Métriques du dépôt

Stars
 (8 123 étoiles)
Métriques de merge PR
 (Merge moyen 5j 9h) (266 PRs mergées en 30 j)

Description

Describe the bug

A workload group with resiliency_mode: monitor still rejects search requests with HTTP 429 when its resource limit is breached. monitor is documented as observe-only ("it will not cause any cancellation but just log the eligible task cancellations"), so it should not reject live traffic.

The cause is in WorkloadGroupService.rejectIfNeeded: the only resiliency-mode exemption is for SOFT (skipped unless the node is in duress). MONITOR falls through to the same rejection path as ENFORCED, throwing OpenSearchRejectedExecutionException and incrementing total_rejections.

This makes monitor self-contradictory: it is correctly dormant on the cancellation path (only ENFORCED groups are passed to cancelTasks), but fully enforcing on the rejection path.

Related component

Search:Resiliency

To Reproduce

  1. Enable WLM: wlm.workload_group.mode: enabled, and lower wlm.workload_group.node.cpu_rejection_threshold (e.g. 0.1) to make a breach easy to trigger.
  2. Create a workload group with resiliency_mode: monitor and a small cpu resource limit (e.g. 0.01).
  3. Add an auto-tagging rule routing an index to that group.
  4. Drive sustained search load against the index so the recorded CPU usage exceeds the (limit × threshold) rejection point.

Expected behavior

A monitor group observes/logs only — searches are not rejected regardless of resource usage. (Only enforced should reject/cancel; soft rejects only under node duress.)

Actual behavior

Searches are rejected with 429 and total_rejections increments. The shard-level cause reason is, verbatim:

rejected_execution_exception: WorkloadGroup <id> is already contended. CPU limit is breaching
for workload group <id>, 0.001 < 0.062..., wlm mode is MONITOR.

(Note the message itself prints wlm mode is MONITOR.)

Additional context

Reproduced locally on a single-node cluster built from the 3.7-wlm-throttling line. This is in the pre-existing node-level resource-rejection path (rejectIfNeeded), independent of the cluster-level throttling work.

Likely fix: extend the exemption so MONITOR bypasses rejection, e.g.

if (mode == ResiliencyMode.MONITOR) return;                 // observe-only: never reject
if (mode == ResiliencyMode.SOFT && !nodeDuressTrackers.isNodeInDuress()) return;

Guide contributeur