[Bug]: Upstream target weight 0 still receives traffic, unset weights count as 1, and the spec says weights are not applied

Open
#4,176 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
api, backend, devops

Research direction

Start with gateway/gateway-controller/pkg/xds/translator.go at createWeightedCluster and compare its weight handling with the zero-weight branch in kubernetes/gateway-operator/internal/controller/httproute_compile.go. Check the validator and the upstream-definition cluster path in gateway/gateway-controller/pkg/transform/restapi.go; the issue also names gateway/gateway-controller/api/management-openapi.yaml and generated REST docs. Done means zero-weight targets receive no traffic, unset-weight behavior is documented or validated, and the spec matches runtime behavior.

Written by the indexing model from the issue text.

Description

Aspect/API Severity/Major Type/Bug
Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)

Description

Weights on upstreamDefinitions[].upstreams[] don't behave the way the API accepts and documents them. There are three related problems.

1. A target with weight: 0 still receives traffic.
The validator accepts 0 (gateway/gateway-controller/pkg/config/api_validator.go:376, range 0–100). createWeightedCluster sets Envoy's load_balancing_weight only when the weight is above 0 (gateway/gateway-controller/pkg/xds/translator.go:676). A target with weight 0 is therefore emitted with no weight, and Envoy treats a missing weight as 1.

  • Targets weighted 100 and 0: the "0" target gets about 1% of requests.
  • Targets weighted 0 and 0: they split 50/50.

An operator who sets a target to 0 to drain it during maintenance keeps sending it traffic. The Kubernetes operator already handles this by dropping weight-0 backends before it builds the definition (kubernetes/gateway-operator/internal/controller/httproute_compile.go, the if w == 0 { continue } branch, whose comment describes this Envoy behaviour). The REST API path has no equivalent.

2. A target without a weight silently counts as 1.
In a definition that mixes weighted and unweighted targets, each unweighted target is emitted with no weight, which Envoy treats as 1. Targets of 80 and unset split 80:1, not anything close to even. This isn't documented.

3. The API spec says weights aren't applied, but they are.
gateway/gateway-controller/api/management-openapi.yaml:4437 describes weight as "Reserved for future multi-target load balancing; not applied yet (only the first target is currently used)". The same text appears in the generated docs under docs/rest-apis/gateway/. That's only true when upstream.main or upstream.sandbox references the definition with ref. Every upstream definition also gets its own cluster that keeps all targets and their weights with round-robin balancing (gateway/gateway-controller/pkg/transform/restapi.go:295-347, translator.go:649). A dynamic-endpoint policy that routes to the definition by name uses that cluster, so the weights do apply.

Suggested fix

  • Exclude weight-0 targets from the generated cluster, as the operator already does, so that 0 means "no traffic". If every target in a definition has weight 0, either reject the definition at deploy time or return the same 503 the operator path produces.
  • Document how an unset weight behaves (it counts as 1), or require all targets in a definition to be either weighted or unweighted.
  • Correct the weight description in management-openapi.yaml and regenerate the docs.
Steps to Reproduce
  1. Deploy two echo backends, backend-a:5000 and backend-b:5000.
  2. Deploy this API to the gateway:
apiVersion: gateway.api-platform.wso2.com/v1
kind: RestApi
metadata:
  name: weight-zero-test
spec:
  displayName: Weight Zero Test
  version: v1.0
  context: /weight-test/v1.0
  upstreamDefinitions:
    - name: pool
      upstreams:
        - url: http://backend-a:5000
          weight: 100
        - url: http://backend-b:5000
          weight: 0
  upstream:
    main:
      url: http://backend-a:5000
  policies:
    - name: dynamic-endpoint
      version: v1
      params:
        targetUpstream: pool
  operations:
    - method: GET
      path: /echo
  1. Send around 1,000 requests to /weight-test/v1.0/echo and count which backend answered each one.

Expected: every request reaches backend-a.
Actual: about 1% of requests reach backend-b.

Setting both weights to 0 gives a 50/50 split, and leaving backend-b's weight out gives the same result as setting it to 0. The generated cluster can also be inspected through the Envoy admin /config_dump: the backend-b endpoint has no load_balancing_weight.

Severity Level of the Issue

Severity/Major (Important functionality is broken. Should be prioritized. Doesn't need immediate attention)

Environment Details (with versions)

Reproduced on a running gateway built from main at 9ceea25b8. The gateway-controller, gateway-runtime and gateway-builder images were built locally with make build and run with the repository's docker-compose.yaml, with two hashicorp/http-echo backends that reply backend-a and backend-b.

Each case is a separate API deployed with the YAML from the steps above, with only the weights changed. Every request returned 200.

Weights (a / b) Requests backend-a backend-b Envoy endpoint weights in /config_dump
100 / 0 1000 990 10 a=100, b=unset
0 / 0 400 200 200 a=unset, b=unset
80 / unset 400 395 5 a=80, b=unset
50 / 50 (control) 400 200 200 a=50, b=50
  • A weight-0 target still gets traffic: about 1% at 100/0, and an even split at 0/0. The weight-0 endpoint has no load_balancing_weight in /config_dump, so Envoy uses its default of 1.
  • An unset weight behaves as 1: 80/unset splits about 80:1.
  • The 50/50 control shows that weights are applied on upstream definition clusters, which contradicts the weight description in management-openapi.yaml.
Dominant language
Go
Stars
85
Forks
137
Avg merge
1d 12h
Merged PRs (30d)
247

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from wso2/api-platform

All issues in wso2/api-platform

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.