Feature Request: PostSync Monitoring on all Config Sync managed resources

未关闭
  • 哪里不对？
#1,886 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
功能
描述清晰度
需要澄清
活跃度
停滞
技术栈
gcp, go, google-cloud, kubernetes
领域
cloud, devops, observability-sre

调研方向

从 examples/post-sync 中的实验性 PostSync 流程以及现有的 pipeline_error_observed 监控文档开始。将其当前的 RootSync 和 RepoSync 范围与 issue 中所示的资源级失败状态进行比较，包括暂时性的 DependencyNotFound 条件。完成的标准是：资源级结构化失败日志能够支持所提议的 Pub/Sub 和 workload 流程，同时不会将临时的 DependencyNotFound 失败视为永久性失败。

由索引模型根据 Issue 内容生成。

描述

enhancement
Checklist
  • I did not find a related open enhancement request.
  • I understand that enhancement requests filed in the GitHub repository are by default low priority.
  • If this request is time-sensitive, I have submitted a corresponding issue with GCP support.
Describe the feature

The current experimental post sync feature is scoped to only monitor RootSyncs and RepoSyncs. However, most issues that can be caused to those resources have been shifted left and are caught earlier in our CI pipelines using kubectl apply --dry-run=server. What would be really powerful is PostSync monitoring for the resources that our RootSyncs control.

We are consuming config sync as a part of config-controller, so our RootSyncs are managing KCC/GCP resources. Those resources are however a lot harder to dry run because there is a strong dependency on the GCP APIs willingness to accept a given change. We often have resources reaching UpdateFailed or DependencyNotFound states. Currently, the lowest granularity we can monitor/alert that at is a RootSync level using pipeline_error_observed metric. This is a huge downside because a RootSync can have > 1000 resources and we have to alert a centralized team rather than the owner of the breaking change. The ideal is to measure, monitor, and alert on this at the per resource level.

The feature request is that PostSync would also log a structured log for these resource failures, and then we can follow the postsync flow of logs -> pubsub -> workload (which would file a bug to the relevant assignee) on a per resource rather than per RootSync basis. There is some intricacy to work around here because a resource might be in DependencyNotFound state for a few minutes, then resolve when the dependency is created.

An example of those failures on the GKE resources

> kc get LoggingLink -n REDACTED
NAME                            AGE   READY   STATUS         STATUS AGE
fooservicefranc-stg-stg1-link   9d    False   UpdateFailed   9d
insightsprocess-stg-stg1-link   9d    False   UpdateFailed   9d

And we can actually see those failures from config sync/nomos

> nomos status --name REDACTED | grep logginglink
     REDACTED   logginglink.logging.cnrm.cloud.google.com/fooservicefranc-stg-stg1-link                                        InProgress   ac89d2a
     REDACTED   logginglink.logging.cnrm.cloud.google.com/insightsprocess-stg-stg1-link   InProgress   ac89d2a
Importance

This is a blocker for our adoption of PostSync. Monitoring only our RootSyncs doesnt provide us much value. If we can't get per resource monitoring from the system, we are reaching the scale where we would need to implement this feature ourselves.

主要语言
Go
星标
279
派生
59
平均合并
1 天 23 小时
30 天内合并 PR
12

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

GoogleContainerTools/config-sync 的其他 Issue

查看 GoogleContainerTools/config-sync 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。