[Bug]: Upstream target weight 0 still receives traffic, unset weights count as 1, and the spec says weights are not applied
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 58/100
Research direction
Start with gateway/gateway-controller/pkg/xds/translator.go at createWeightedCluster and compare its weight handling with the zero-weight branch in kubernetes/gateway-operator/internal/controller/httproute_compile.go. Check the validator and the upstream-definition cluster path in gateway/gateway-controller/pkg/transform/restapi.go; the issue also names gateway/gateway-controller/api/management-openapi.yaml and generated REST docs. Done means zero-weight targets receive no traffic, unset-weight behavior is documented or validated, and the spec matches runtime behavior.
Written by the indexing model from the issue text.
Description
Please select the area the issue is related to
Gateway
Please select the aspect the issue is related to
Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)
Description
Weights on upstreamDefinitions[].upstreams[] don't behave the way the API accepts and documents them. There are three related problems.
1. A target with weight: 0 still receives traffic.
The validator accepts 0 (gateway/gateway-controller/pkg/config/api_validator.go:376, range 0–100). createWeightedCluster sets Envoy's load_balancing_weight only when the weight is above 0 (gateway/gateway-controller/pkg/xds/translator.go:676). A target with weight 0 is therefore emitted with no weight, and Envoy treats a missing weight as 1.
- Targets weighted 100 and 0: the "0" target gets about 1% of requests.
- Targets weighted 0 and 0: they split 50/50.
An operator who sets a target to 0 to drain it during maintenance keeps sending it traffic. The Kubernetes operator already handles this by dropping weight-0 backends before it builds the definition (kubernetes/gateway-operator/internal/controller/httproute_compile.go, the if w == 0 { continue } branch, whose comment describes this Envoy behaviour). The REST API path has no equivalent.
2. A target without a weight silently counts as 1.
In a definition that mixes weighted and unweighted targets, each unweighted target is emitted with no weight, which Envoy treats as 1. Targets of 80 and unset split 80:1, not anything close to even. This isn't documented.
3. The API spec says weights aren't applied, but they are.
gateway/gateway-controller/api/management-openapi.yaml:4437 describes weight as "Reserved for future multi-target load balancing; not applied yet (only the first target is currently used)". The same text appears in the generated docs under docs/rest-apis/gateway/. That's only true when upstream.main or upstream.sandbox references the definition with ref. Every upstream definition also gets its own cluster that keeps all targets and their weights with round-robin balancing (gateway/gateway-controller/pkg/transform/restapi.go:295-347, translator.go:649). A dynamic-endpoint policy that routes to the definition by name uses that cluster, so the weights do apply.
Suggested fix
- Exclude weight-0 targets from the generated cluster, as the operator already does, so that 0 means "no traffic". If every target in a definition has weight 0, either reject the definition at deploy time or return the same 503 the operator path produces.
- Document how an unset weight behaves (it counts as 1), or require all targets in a definition to be either weighted or unweighted.
- Correct the
weightdescription inmanagement-openapi.yamland regenerate the docs.
Steps to Reproduce
- Deploy two echo backends,
backend-a:5000andbackend-b:5000. - Deploy this API to the gateway:
apiVersion: gateway.api-platform.wso2.com/v1
kind: RestApi
metadata:
name: weight-zero-test
spec:
displayName: Weight Zero Test
version: v1.0
context: /weight-test/v1.0
upstreamDefinitions:
- name: pool
upstreams:
- url: http://backend-a:5000
weight: 100
- url: http://backend-b:5000
weight: 0
upstream:
main:
url: http://backend-a:5000
policies:
- name: dynamic-endpoint
version: v1
params:
targetUpstream: pool
operations:
- method: GET
path: /echo
- Send around 1,000 requests to
/weight-test/v1.0/echoand count which backend answered each one.
Expected: every request reaches backend-a.
Actual: about 1% of requests reach backend-b.
Setting both weights to 0 gives a 50/50 split, and leaving backend-b's weight out gives the same result as setting it to 0. The generated cluster can also be inspected through the Envoy admin /config_dump: the backend-b endpoint has no load_balancing_weight.
Severity Level of the Issue
Severity/Major (Important functionality is broken. Should be prioritized. Doesn't need immediate attention)
Environment Details (with versions)
Reproduced on a running gateway built from main at 9ceea25b8. The gateway-controller, gateway-runtime and gateway-builder images were built locally with make build and run with the repository's docker-compose.yaml, with two hashicorp/http-echo backends that reply backend-a and backend-b.
Each case is a separate API deployed with the YAML from the steps above, with only the weights changed. Every request returned 200.
| Weights (a / b) | Requests | backend-a | backend-b | Envoy endpoint weights in /config_dump |
|---|---|---|---|---|
| 100 / 0 | 1000 | 990 | 10 | a=100, b=unset |
| 0 / 0 | 400 | 200 | 200 | a=unset, b=unset |
| 80 / unset | 400 | 395 | 5 | a=80, b=unset |
| 50 / 50 (control) | 400 | 200 | 200 | a=50, b=50 |
- A weight-0 target still gets traffic: about 1% at 100/0, and an even split at 0/0. The weight-0 endpoint has no
load_balancing_weightin/config_dump, so Envoy uses its default of 1. - An unset weight behaves as 1: 80/unset splits about 80:1.
- The 50/50 control shows that weights are applied on upstream definition clusters, which contradicts the
weightdescription inmanagement-openapi.yaml.
- Dominant language
- Go
- Stars
- 85
- Forks
- 137
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 247
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from wso2/api-platform
-
Area/DeveloperPortal Aspect/UI Severity/Minor Type/Bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
wso2/api-platform#4369 ·
Maintainers usually reply within 1 day
-
[Bug]: host-rewrite policy documentation lists fewer allowed host characters than the policy acceptsOpenArea/Policies Aspect/Other Severity/Trivial Type/Bug
Difficulty 1/5 Under an hour Newbie friendliness 91/100
wso2/api-platform#4355 ·
Maintainers usually reply within 1 day
-
Area/Gateway Aspect/API Severity/Minor Type/Bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
wso2/api-platform#4348 ·
Maintainers usually reply within 1 day
-
Aspect/AI Aspect/API. Severity/Major
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
wso2/api-platform#4343 ·
Maintainers usually reply within 1 day
-
Area/Policies Aspect/Configuration Severity/Minor Type/Bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
wso2/api-platform#4330 ·
Maintainers usually reply within 1 day
All issues in wso2/api-platform
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Inconsistency between golang and cue in specPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Tencent/LoopForge#1087 · 1 comment ·
Maintainers usually reply within 1 day