Refuse replica counts that break a cluster

Bắt đầu bằng cách đọc Scope và hành vi hiện có của Scaler, sau đó kiểm tra cách giới hạn bản sao beat của superset-operator hiện được áp dụng. Xác định mối liên hệ giữa khai báo ở cấp framework, việc báo cáo từ chối, phát hiện mục tiêu và các mức sàn tĩnh được mô tả trong issue; được xem là hoàn tất khi các cơ chế framework này được định nghĩa mà không triển khai các ràng buộc dành riêng cho sản phẩm.

Part of #872

Problem

Currently nothing stops anyone from setting a replica count that does break a cluster.
Right now this is mostly a manual thing but when we add auto scaling this could also more easily done by a machine.

These are the roles that I know are problematic but input is welcome for the tools where I don't know enough:

• Quorum members: ZooKeeper server, Kafka controller, HDFS journalnode, and any OpenSearch role group carrying cluster_manager (TODO: Is that correct?). These can be resized it just might break some guarantees.
• HA pairs and leader-elected roles: HDFS namenode, HBase master, Airflow scheduler. In theory most of these can live alone but then we lose HA... not sure about these.
• Roles sized by something other than load: Airflow dagProcessor tracks the number of DAG files.
• Hard limits: Superset beat and Trino coordinator. On beat: I don't know this but Claude tells me that running two instances would schedule the same tasks twice. This already seems to be enforced in the operator. Trino does not support multiple coordinators and nothing enforces that today. Those are the two roles I'm aware of with hard limits.

In summary: Quorum members and the HA pairs are usually sized not based on the load (which auto scaling would often "attack") but by design.
Example: No matter how large my cluster is I'll always want a ZooKeeper quorum of 3 and don't let an auto-scaler muck with that because it thinks they are idle.
Superset beat and the Trino coordinator are the only true hard limits.

Scope

Framework only.
Declaring each product's actual constraints happens alongside wiring that product up to the Scaler, in the same change, because whoever does that already has the product context.

1. A role declares whether it may be autoscaled. For roles that may not: no Scaler is created. This is at the framework level.

2. Hard limits where they exist. superset-operator already does this for beat.
   "Replicas greater than one are ignored and set to 1", so we have implemented it once but probably worth it to pull that out into the operator framework? I'll leave it to the implementer.

3. Detech HPA or KEDA objects targeting one of ours. I'm not sure if this is possible or how easy it is.
   We have two use-cases where it'll be useful to know if this is the case and warn the user.
   They are detection wise the same but two different "messages": First if we find a thing targeting one of the roles that have no Scaler by design (see point 1) and secondly finding something that targets the StatefulSet directly which DOES have a Scaler (especially as that is what the docs currently suggest).

4. A floor, where the product has one we can know statically.

   • NiFi: 1. It refuses to disconnect its last connected node, so it cannot reach 0 however it is configured.
   • HBase regionserver: 1, for availability. The data is in HDFS so nothing is lost by going low but it needs a RS for the meta region.
   • HDFS: dfs_replication gives us a cheap pre-check, and we already reason this way for the PDB (max_unavailable = dfs_replication.saturating_sub(2)). It is not authoritative though, because HDFS replication is per-file and dfs.replication is only the default for new files. So... if you disagree with HDFS then I'm happy to leave it out.

Kafka is left out deliberately because topic replication is...per-topic.
We already have a comment on that in our code.
Support for this will be added in another issue.
0 should be allowed for roles whose drain only waits for work in flight: Celery works and Trino workers I'd say.
If someone wants it stopped entirely they can use clusterOperation.stopped.

[!NOTE]
With Kubernetes 1.37 the HPA can now scale to 0 which would be a problem for NiFi. KEDA could already do it.

Any time we "refuse" to honor something we should complain loudly. I see multiple options and I don't know what's best. Left for discussion.

• Status
• Logs / Metrics / ...
• An admission controller

Out of scope

• Declaring the actual constraints per product. See the note under Scope. This is only about the framewrok "machinery"
• User architectural preferences. "We always run three to five ZooKeeper nodes" does make sense but if they don't run auto-scalers then we already support it and if they do then it's the job of the scaler to clamp this.
• Parity. ZooKeeper ensembles and KRaft quorums want odd counts. We can't express that with min/max. Ignore.
• Any drain, and the hooks framework (#880). Refusing a change is synchronous and needs no state machine.
• Anything that decides when to scale.

Open questions

• Where does enforcement live? (see above)
• Claude flagged this, please verify: OpenSearch does not fit a per-role lookup. Its nodeRoles are per-role-group configuration rather than a role, so whether a group may be autoscaled depends on how it is configured: a group carrying cluster_manager must not be, a coordinating_only group is fine. Everywhere else this can be a simple per-role property.

Expected effort

5 days, and if it takes longer tell me.

My reasoning: the declaration and the check are small, and the per-product sweep is explicitly not here.
This could grow if we decide for an admission controller because I believe we have none today.