opensearch-project/OpenSearch

WLM: monitor-mode workload group still rejects requests with 429 on resource breach

Aberta

#22.616 aberto em 31 de jul. de 2026

 (5 comentários) (0 reação) (0 responsável)Java (1.505 forks)batch import
Search:Resiliencygood first issue

Métricas do repositório

Stars
 (8.123 estrelas)
Métricas de merge de PR
 (Mesclagem média 5d 9h) (266 fundiu PRs em 30d)

Description

Describe the bug

A workload group with resiliency_mode: monitor still rejects search requests with HTTP 429 when its resource limit is breached. monitor is documented as observe-only ("it will not cause any cancellation but just log the eligible task cancellations"), so it should not reject live traffic.

The cause is in WorkloadGroupService.rejectIfNeeded: the only resiliency-mode exemption is for SOFT (skipped unless the node is in duress). MONITOR falls through to the same rejection path as ENFORCED, throwing OpenSearchRejectedExecutionException and incrementing total_rejections.

This makes monitor self-contradictory: it is correctly dormant on the cancellation path (only ENFORCED groups are passed to cancelTasks), but fully enforcing on the rejection path.

Related component

Search:Resiliency

To Reproduce

  1. Enable WLM: wlm.workload_group.mode: enabled, and lower wlm.workload_group.node.cpu_rejection_threshold (e.g. 0.1) to make a breach easy to trigger.
  2. Create a workload group with resiliency_mode: monitor and a small cpu resource limit (e.g. 0.01).
  3. Add an auto-tagging rule routing an index to that group.
  4. Drive sustained search load against the index so the recorded CPU usage exceeds the (limit × threshold) rejection point.

Expected behavior

A monitor group observes/logs only — searches are not rejected regardless of resource usage. (Only enforced should reject/cancel; soft rejects only under node duress.)

Actual behavior

Searches are rejected with 429 and total_rejections increments. The shard-level cause reason is, verbatim:

rejected_execution_exception: WorkloadGroup <id> is already contended. CPU limit is breaching
for workload group <id>, 0.001 < 0.062..., wlm mode is MONITOR.

(Note the message itself prints wlm mode is MONITOR.)

Additional context

Reproduced locally on a single-node cluster built from the 3.7-wlm-throttling line. This is in the pre-existing node-level resource-rejection path (rejectIfNeeded), independent of the cluster-level throttling work.

Likely fix: extend the exemption so MONITOR bypasses rejection, e.g.

if (mode == ResiliencyMode.MONITOR) return;                 // observe-only: never reject
if (mode == ResiliencyMode.SOFT && !nodeDuressTrackers.isNodeInDuress()) return;

Guia do colaborador