WLM: monitor-mode workload group still rejects requests with 429 on resource breach
#22.616 aberto em 31 de jul. de 2026
Métricas do repositório
- Stars
- (8.123 estrelas)
- Métricas de merge de PR
- (Mesclagem média 5d 9h) (266 fundiu PRs em 30d)
Description
Describe the bug
A workload group with resiliency_mode: monitor still rejects search requests with HTTP 429 when its resource limit is breached. monitor is documented as observe-only ("it will not cause any cancellation but just log the eligible task cancellations"), so it should not reject live traffic.
The cause is in WorkloadGroupService.rejectIfNeeded: the only resiliency-mode exemption is for SOFT (skipped unless the node is in duress). MONITOR falls through to the same rejection path as ENFORCED, throwing OpenSearchRejectedExecutionException and incrementing total_rejections.
This makes monitor self-contradictory: it is correctly dormant on the cancellation path (only ENFORCED groups are passed to cancelTasks), but fully enforcing on the rejection path.
Related component
Search:Resiliency
To Reproduce
- Enable WLM:
wlm.workload_group.mode: enabled, and lowerwlm.workload_group.node.cpu_rejection_threshold(e.g.0.1) to make a breach easy to trigger. - Create a workload group with
resiliency_mode: monitorand a smallcpuresource limit (e.g.0.01). - Add an auto-tagging rule routing an index to that group.
- Drive sustained search load against the index so the recorded CPU usage exceeds the (limit × threshold) rejection point.
Expected behavior
A monitor group observes/logs only — searches are not rejected regardless of resource usage. (Only enforced should reject/cancel; soft rejects only under node duress.)
Actual behavior
Searches are rejected with 429 and total_rejections increments. The shard-level cause reason is, verbatim:
rejected_execution_exception: WorkloadGroup <id> is already contended. CPU limit is breaching
for workload group <id>, 0.001 < 0.062..., wlm mode is MONITOR.
(Note the message itself prints wlm mode is MONITOR.)
Additional context
Reproduced locally on a single-node cluster built from the 3.7-wlm-throttling line. This is in the pre-existing node-level resource-rejection path (rejectIfNeeded), independent of the cluster-level throttling work.
Likely fix: extend the exemption so MONITOR bypasses rejection, e.g.
if (mode == ResiliencyMode.MONITOR) return; // observe-only: never reject
if (mode == ResiliencyMode.SOFT && !nodeDuressTrackers.isNodeInDuress()) return;