[Bug]: Upstream target weight 0 still receives traffic, unset weights count as 1, and the spec says weights are not applied

Aperta
#4,176 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
58/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
api, backend, devops

Direzione di ricerca

Start with gateway/gateway-controller/pkg/xds/translator.go at createWeightedCluster and compare its weight handling with the zero-weight branch in kubernetes/gateway-operator/internal/controller/httproute_compile.go. Check the validator and the upstream-definition cluster path in gateway/gateway-controller/pkg/transform/restapi.go; the issue also names gateway/gateway-controller/api/management-openapi.yaml and generated REST docs. Done means zero-weight targets receive no traffic, unset-weight behavior is documented or validated, and the spec matches runtime behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Aspect/API Severity/Major Type/Bug
Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)

Description

Weights on upstreamDefinitions[].upstreams[] don't behave the way the API accepts and documents them. There are three related problems.

1. A target with weight: 0 still receives traffic.
The validator accepts 0 (gateway/gateway-controller/pkg/config/api_validator.go:376, range 0–100). createWeightedCluster sets Envoy's load_balancing_weight only when the weight is above 0 (gateway/gateway-controller/pkg/xds/translator.go:676). A target with weight 0 is therefore emitted with no weight, and Envoy treats a missing weight as 1.

  • Targets weighted 100 and 0: the "0" target gets about 1% of requests.
  • Targets weighted 0 and 0: they split 50/50.

An operator who sets a target to 0 to drain it during maintenance keeps sending it traffic. The Kubernetes operator already handles this by dropping weight-0 backends before it builds the definition (kubernetes/gateway-operator/internal/controller/httproute_compile.go, the if w == 0 { continue } branch, whose comment describes this Envoy behaviour). The REST API path has no equivalent.

2. A target without a weight silently counts as 1.
In a definition that mixes weighted and unweighted targets, each unweighted target is emitted with no weight, which Envoy treats as 1. Targets of 80 and unset split 80:1, not anything close to even. This isn't documented.

3. The API spec says weights aren't applied, but they are.
gateway/gateway-controller/api/management-openapi.yaml:4437 describes weight as "Reserved for future multi-target load balancing; not applied yet (only the first target is currently used)". The same text appears in the generated docs under docs/rest-apis/gateway/. That's only true when upstream.main or upstream.sandbox references the definition with ref. Every upstream definition also gets its own cluster that keeps all targets and their weights with round-robin balancing (gateway/gateway-controller/pkg/transform/restapi.go:295-347, translator.go:649). A dynamic-endpoint policy that routes to the definition by name uses that cluster, so the weights do apply.

Suggested fix

  • Exclude weight-0 targets from the generated cluster, as the operator already does, so that 0 means "no traffic". If every target in a definition has weight 0, either reject the definition at deploy time or return the same 503 the operator path produces.
  • Document how an unset weight behaves (it counts as 1), or require all targets in a definition to be either weighted or unweighted.
  • Correct the weight description in management-openapi.yaml and regenerate the docs.
Steps to Reproduce
  1. Deploy two echo backends, backend-a:5000 and backend-b:5000.
  2. Deploy this API to the gateway:
apiVersion: gateway.api-platform.wso2.com/v1
kind: RestApi
metadata:
  name: weight-zero-test
spec:
  displayName: Weight Zero Test
  version: v1.0
  context: /weight-test/v1.0
  upstreamDefinitions:
    - name: pool
      upstreams:
        - url: http://backend-a:5000
          weight: 100
        - url: http://backend-b:5000
          weight: 0
  upstream:
    main:
      url: http://backend-a:5000
  policies:
    - name: dynamic-endpoint
      version: v1
      params:
        targetUpstream: pool
  operations:
    - method: GET
      path: /echo
  1. Send around 1,000 requests to /weight-test/v1.0/echo and count which backend answered each one.

Expected: every request reaches backend-a.
Actual: about 1% of requests reach backend-b.

Setting both weights to 0 gives a 50/50 split, and leaving backend-b's weight out gives the same result as setting it to 0. The generated cluster can also be inspected through the Envoy admin /config_dump: the backend-b endpoint has no load_balancing_weight.

Severity Level of the Issue

Severity/Major (Important functionality is broken. Should be prioritized. Doesn't need immediate attention)

Environment Details (with versions)

Reproduced on a running gateway built from main at 9ceea25b8. The gateway-controller, gateway-runtime and gateway-builder images were built locally with make build and run with the repository's docker-compose.yaml, with two hashicorp/http-echo backends that reply backend-a and backend-b.

Each case is a separate API deployed with the YAML from the steps above, with only the weights changed. Every request returned 200.

Weights (a / b) Requests backend-a backend-b Envoy endpoint weights in /config_dump
100 / 0 1000 990 10 a=100, b=unset
0 / 0 400 200 200 a=unset, b=unset
80 / unset 400 395 5 a=80, b=unset
50 / 50 (control) 400 200 200 a=50, b=50
  • A weight-0 target still gets traffic: about 1% at 100/0, and an even split at 0/0. The weight-0 endpoint has no load_balancing_weight in /config_dump, so Envoy uses its default of 1.
  • An unset weight behaves as 1: 80/unset splits about 80:1.
  • The 50/50 control shows that weights are applied on upstream definition clusters, which contradicts the weight description in management-openapi.yaml.
Lingua principale
Go
Stelle
85
Fork
137
Merge medio
1g 12h
PR unite (30g)
247

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di wso2/api-platform

Tutte le issue di wso2/api-platform

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.