Feature Request: PostSync Monitoring on all Config Sync managed resources

Aperta
  • Qualcosa non va?
#1,886 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
gcp, go, google-cloud, kubernetes
Ambito
cloud, devops, observability-sre

Direzione di ricerca

Inizia con il flusso sperimentale PostSync in examples/post-sync e con la documentazione esistente sul monitoraggio di pipeline_error_observed. Confronta l'ambito attuale di RootSync e RepoSync con gli stati di errore per risorsa mostrati nell'issue, incluse le condizioni transitorie DependencyNotFound. Il lavoro è completato quando i log strutturati degli errori a livello di risorsa possono supportare il flusso proposto di Pub/Sub e workload senza trattare gli errori temporanei DependencyNotFound come permanenti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement
Checklist
  • I did not find a related open enhancement request.
  • I understand that enhancement requests filed in the GitHub repository are by default low priority.
  • If this request is time-sensitive, I have submitted a corresponding issue with GCP support.
Describe the feature

The current experimental post sync feature is scoped to only monitor RootSyncs and RepoSyncs. However, most issues that can be caused to those resources have been shifted left and are caught earlier in our CI pipelines using kubectl apply --dry-run=server. What would be really powerful is PostSync monitoring for the resources that our RootSyncs control.

We are consuming config sync as a part of config-controller, so our RootSyncs are managing KCC/GCP resources. Those resources are however a lot harder to dry run because there is a strong dependency on the GCP APIs willingness to accept a given change. We often have resources reaching UpdateFailed or DependencyNotFound states. Currently, the lowest granularity we can monitor/alert that at is a RootSync level using pipeline_error_observed metric. This is a huge downside because a RootSync can have > 1000 resources and we have to alert a centralized team rather than the owner of the breaking change. The ideal is to measure, monitor, and alert on this at the per resource level.

The feature request is that PostSync would also log a structured log for these resource failures, and then we can follow the postsync flow of logs -> pubsub -> workload (which would file a bug to the relevant assignee) on a per resource rather than per RootSync basis. There is some intricacy to work around here because a resource might be in DependencyNotFound state for a few minutes, then resolve when the dependency is created.

An example of those failures on the GKE resources

> kc get LoggingLink -n REDACTED
NAME                            AGE   READY   STATUS         STATUS AGE
fooservicefranc-stg-stg1-link   9d    False   UpdateFailed   9d
insightsprocess-stg-stg1-link   9d    False   UpdateFailed   9d

And we can actually see those failures from config sync/nomos

> nomos status --name REDACTED | grep logginglink
     REDACTED   logginglink.logging.cnrm.cloud.google.com/fooservicefranc-stg-stg1-link                                        InProgress   ac89d2a
     REDACTED   logginglink.logging.cnrm.cloud.google.com/insightsprocess-stg-stg1-link   InProgress   ac89d2a
Importance

This is a blocker for our adoption of PostSync. Monitoring only our RootSyncs doesnt provide us much value. If we can't get per resource monitoring from the system, we are reaching the scale where we would need to implement this feature ourselves.

Lingua principale
Go
Stelle
279
Fork
59
Merge medio
1g 23h
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di GoogleContainerTools/config-sync

Tutte le issue di GoogleContainerTools/config-sync

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.