Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[watchdog] Persist 2Gi memory limit in deployment configuration to mitigate GC OOM

Aperta Adatta ai principianti
  • Qualcosa non va?
#205 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

@hippogr ci sta già lavorando.

Dal 23/10/2026.

  • #211 di @hippogr — aperta

Valutazione

Difficoltà
1/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
84/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kubernetes, yaml
Ambito
devops

Direzione di ricerca

Leggi src/watchdog/deploy/watchdog.yaml.template e individua resources.limits.memory nella condizione qos-switch esistente. Cambia il valore della condizione abilitata da "1Gi" a "2Gi", mantenendo la condizione e le altre impostazioni. Esegui il rendering del template con la condizione abilitata e disabilitata; conferma che l'output con la condizione abilitata usa 2Gi e che il comportamento con la condizione disabilitata rimane invariato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Persist the watchdog memory-limit increase from 1Gi to 2Gi in the deployment template. A temporary live mitigation used the higher limit, but a subsequent deployment from the current template would restore 1Gi and risk recurring OOM kills.

Configuration to update

  • File: src/watchdog/deploy/watchdog.yaml.template
  • Under the existing qos-switch condition, change resources.limits.memory from "1Gi" to "2Gi".
  • Preserve the existing conditional behavior and unrelated deployment settings.

Observed behavior

  • With the 1Gi limit, watchdog experienced repeated OOM kills.
  • One failure occurred shortly after scheduled garbage collection started. Normal metric collection used substantially less memory.
  • Metadata-only checks found a large backlog of Secrets and PriorityClasses relative to the small number of Frameworks.
  • Garbage collection currently lists full objects without pagination. Bulk-list memory pressure is the leading explanation; the exact failing API request was not instrumented.

Temporary mitigation and observation

  • After a temporary increase to 2Gi, the observed process remained Ready without additional restarts or cgroup OOM events during a multi-hour observation window.
  • Observed container memory remained above the original limit and decreased over the observation window.
  • The process RSS high-water mark also exceeded the original 1Gi limit. This is process RSS, not a measured container-memory peak.
  • Garbage collection progressed through orphan PriorityClasses into orphan Secrets, but had not completed the full pass. This observation supports mitigation, not a claim of complete or repeated successful GC cycles.

Acceptance criteria

  • Deployment template renders a 2Gi watchdog memory limit when the existing QoS condition is enabled.
  • Behavior when that condition is disabled remains unchanged.
  • A subsequent deployment no longer reverts watchdog to 1Gi.
  • After deployment, verify the effective container limit and observe garbage collection for OOM/restart recurrence.

Follow-up (not required for this configuration change)

Consider paginated/metadata-only listing and bounded processing to reduce garbage-collection memory usage. Raising the limit is a mitigation, not a root-cause code fix.

No repository code or deployment-template changes have been made as part of creating this issue.

Deployment safety

The watchdog uses host networking and a fixed listener port. A default surge-based rolling update can overlap old and new instances on the same node: the existing instance remains available while the replacement repeatedly exits and the rollout stalls. During a controlled memory-limit update, temporarily allowing the old instance to stop before starting the replacement resolved this overlap; the previous rollout policy was restored afterward.

  • When implementing/deploying the memory-limit change, explicitly account for the single-node fixed-port constraint and verify that the replacement becomes Ready without a stalled rollout.
  • Treat online resource changes as temporary mitigation; the deployment template still needs the tracked persistent change.
Lingua principale
Python
Stelle
11
Fork
5
Merge medio
5h 46m
PR unite (30g)
4

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/ltp-platform

Tutte le issue di microsoft/ltp-platform

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.