reconciling of multiple trino clusters results in clusterwide coordinator downtime
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
- 技术栈
- kubernetes, rust
调研方向
首先跟踪多个 TrinoCluster 资源或 catalog match labels 同时发生变化时 operator 的 reconciliation 行为，然后检查与所提供的 PodDisruptionBudget 相关的 coordinator 删除行为。复现同时发生的更改，并确定完成是否需要串行化 reconciliation、修正 disruption 处理，或两者都需要；完成的标准是在并发更新期间避免 coordinator 停机。
由索引模型根据 Issue 内容生成。
描述
we're dealing with the issue of concurrent reconcilations when trinocluster resources change. this issue occurs e.g. when a catalog is applied to the cluster matching more than one catalog-matchlabel or when all trino cluster resources are changed at the same time because they are configured in custom helm wrappers.
since we use argo for continous deployments we are not able to change clusters / upsert catalogs subsequently in a manual way.
we did not make progress with trino-lb (https://github.com/stackabletech/trino-operator/issues/490) yet but I'm sure even with trino-lb running this would cause outages everytime the trinocluster resources are (re-)configured or catalogs are upserted. unfortunately running trino in a high available way is mission critical for our production scenario
possible solution: subsequent reconcilation
introducing a flag for the operator (maybe other product operators might be affecated as well) which enables subsequent reconcilations in a queue style instead of parallelized reconcilations which lead to all clusters going offline at the same time.
disadvantage might be that a malicious cluster kills the whole reconcilation process until the resource is fixed manually.
possible solution: pdb
we already defined following pdb to make sure one coordinator per kubernetes cluster is available. unfortunately the pdb is ignored and all coordinators get killed concurrently. @maltesander @sbernauer already told about delete operations instead of evictions which would take care of the pdb. feel free to edit / add some further details
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: trino-highavailiability-coordinator
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/component: coordinator
Seems like somebody is feeling similar pain with elasticsearch https://github.com/kubernetes/kubernetes/issues/91808#issuecomment-1244596481
- 主要语言
- Rust
- 星标
- 64
- 派生
- 14
- 平均合并
- 1 天 8 小时
- 30 天内合并 PR
- 10
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
stackabletech/trino-operator 的其他 Issue
-
customer-request
难度 2/5 1-3 小时 新手友好度 60/100
stackabletech/trino-operator#499 ·
维护者通常 1 天内回复
-
type/bug
难度 4/5 3-5 天 新手友好度 45/100
stackabletech/trino-operator#936 · 3 条评论 ·
维护者通常 1 天内回复
-
bug: pod spec doesn't match sts template spec可能已有人在做 @razvan 于 222 天前认领。 未关闭release-note
stackabletech/trino-operator#854 · 3 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Add Iceberg REST catalog support to TrinoCatalog可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 3/5 1-2 天 新手友好度 45/100
stackabletech/trino-operator#849 ·
维护者通常 1 天内回复
-
customer-request type/feature-improvement
难度 3/5 1-2 天 新手友好度 35/100
stackabletech/trino-operator#813 · 1 条评论 ·
维护者通常 1 天内回复
查看 stackabletech/trino-operator 的全部 Issue
相似的 Issue
-
llm translation
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
area: backend type: enhancement
难度 2/5 1-3 小时 新手友好度 61/100
armadavalor/WinKnife#6 ·
-
难度 2/5 1-3 小时 新手友好度 68/100
-
难度 2/5 1-3 小时 新手友好度 78/100
5omeOtherGuy/phaseone#870 ·
维护者通常 1 天内回复
-
难度 1/5 1-3 小时 新手友好度 72/100
MattA-Official/vwmcp#20 ·