Epic: Safe and automatic scaling

Inizia con la issue #873 del framework e con la Scaler CRD esistente e la sottorisorsa /scale in operator-rs. Poi esamina #880 e #881 per il lavoro su hook e guard e per l’attività di documentazione concepts/operations; l’epic è completato quando il framework, le integrazioni dei prodotti applicabili e le indicazioni per uno scaling sicuro sono completati.

[!IMPORTANT]
This is under active refinement. The contents is not final.

Problem

We currently suggest users to use a HorizontalPodAutoscaler (HPA) when they need to scale one of our products.
That is fine in theory but can be dangerous in practice for stateful apps like NiFi: The HPA changes the StatefulSet directly, so the operator never sees it and nothing safely drains the Pod before it is deleted.

There is a second problem which seems kinda obvious but which I clearned from reading a blog on the experience Netflix had: Scalin gon CPU or memory doesn't necessarily make sense as e.g. NiFi might be totally blocked on something else. So for that different signals might be more useful. That is out of scope in this issue though (the signal work).

We need a way to safely scale our products, ideally automatically.

Scope

Deliver a path where a replica change, whoever makes it, goes through the operator so product-specific work can run before Pods are removed.

Two capabilities both using the /scale subresource on the Scaler CRD we already added to operator-rs (see https://github.com/stackabletech/issues/issues/667):

• Safe scaling Refuse or delay a change that would lose data, and drain first where the product needs it.
• Autoscaling for products where load-responsive scaling makes sense.

This epic is about framework/generic work as well as the product-specific bits to achieve those two goals.

How it fits together

[!NOTE]
I needed this section to help me understand all that's going on. It represents my current understanding. Let me know if you think anything here is wrong.

The Scaler is the place for the replica number for most (see below) of our product's roles to land where the operator can see it before anything happens to Pods.

an HPA, or a user, writes Scaler.spec.replicas
   |
   v
1. Guard        Is this change safe? If not: refuse or revert, and stop here. 
   v
2. Pre-scale hook (Scaler::PreScaling)      Drain whatever is about to leave. #880 
   | 
   v
3. Propagate (Scaler::Scaling)              Write replicas to the StatefulSet and wait for Pods to converge.  #873 
   |            
   v
4. Post-scale hook (Scaler::PostScaling)   Rebalance onto whatever just arrived. #880 
   |
   v
done

When looking at our tools I tried to put them in different buckets to see which of our tools need which of these steps.

Two reasons this scaling stuff has to live in the operator and not in the pods in a preStop hook for almost everything:

1. Thanks @soenkeliebau for the heads-up about this: Whatever a pod does is subject to a termination grace period SIGKILL and that would be bad for anything coordinated.

2. From all I can tell there is no way that a Pod that is being terminated can actually tell why it is being terminated.
   It'd be good to know if I'm wrong here. But we need to distinguish between "scaling down forever...for now" or "just a restart".
   Because in the first case we'd like to move data from NiFi, HDFS, Kafka but not in the latter
   The initial idea was to use a preStop hook to "drain" the pod but due to that problem that'd e.g. move all Kafka partitions around on every restart. That sucks.

Product	Scaler	Guard	Hooks
HDFS, Kafka, NiFi	yes	yes	yes (moves data)
Trino, Airflow and Superset workers	yes	no	no (just waiting for work to finish)
HBase	yes	probably	I'm not sure yet... in theory moving regions is cheap-ish, but in practice it can cause trouble. Needs more thought
OpenSearch	yes	TODO	TODO don't know enough about OpenSearch
Stateless roles	yes, for consistency	no	no
ZooKeeper, Spark, Phoenix, Omid, OPA	n/a	n/a	n/a, none of these have anything we scale. See below.

Stateless roles don't strictly need a Scaler to be safe but I think it's best if we introduce it anyway so the configuration is the same for every product.

Out of scope

• Any per-product work on selecting or exposing "signals" to scale on.
• Anything related to KEDA
• Evictions: handling anything that drains a node. Such a drain bypasses the Scaler entirely, so the same problem applies, but it is not solved here.
• Evictions. I think/I hope we do what we can with PDBs and everything else should be outside of "scaling" as this does not change any replica counts
• Autoscaling HDFS (and maybe Kafka)

Tasks

Added as they are refined.

Framework

• https://github.com/stackabletech/issues/issues/873 - This should go first
• https://github.com/stackabletech/issues/issues/880
• https://github.com/stackabletech/issues/issues/881

Per product

Product	Scalable role(s)	How scale-down is made safe
Airflow	webserver, worker (Celery), maybe triggerer	worker waits for running tasks, in-Pod. With the Kubernetes executor there is no worker role group at all. On triggerer: Airflow supports running several of them for HA, so it might be scalable, but I don't know whether load ever justifies it. Input welcome.
HBase	regionserver, restserver	Regions are reassigned; the data stays in HDFS. Who drives it is to be determined.
HDFS	datanode	Decommissioning is issued at the NameNode, so the departing Pod cannot do it.
Hive	metastore	Nothing needed. State lives in the RDBMS.
Kafka	broker	Partitions must be reassigned off the broker first, and that is driven externally.
NiFi	node	Operator-driven in the pre-scale hook. NiFi's ClusterDecommissionTask does the actual work. Not yet sure how the operator reaches it. Maybe a question for #865
OPA	none today, it is a DaemonSet	Running it as a Deployment instead is tracked in opa-operator#525.
OpenSearch	coordinating_only, ingest, and probably data, warm, search	Coordinating and ingest need nothing. Data nodes need shard reallocation. TODO: I do not know enough about OpenSearch to fully assess this
Phoenix, Omid	none	Shipped as libraries alongside HBase. Nothing to scale.
Spark	none	Spark manages its own executors through dynamic allocation. Not our problem to solve.
Superset	node, worker (Celery)	worker waits for running tasks, in-Pod.
Trino	worker	In-Pod preStop, already implemented today.
ZooKeeper	none	Ensemble size is a design decision, not a response to load.

Roles that must never be autoscaled

See #881 for details on this rather than me repeating it here.

Documentation

• Fix concepts/operations pages telling people to point an HPA at the StatefulSet. That's a no-no now.
• Add some best practices:
  • Scaling down should be slower than scaling up because a) it's expensive to rebalance/drain etc. and often after that happens it takes a while for things to settle so it briefly looks worse than it is (e.g. HBase). HPA seems to use behavior.scaleDown. Netflix runs their Flink autoscaler at a target utilization of 0.45 (default 0.7) for this reason, "deliberately trading a little efficiency for stability". We should also document this as a best practice.
  • And there seems to be a downscale stabilisation setting which is at 5min. That's not enough.
  • Basically go through all the settings (behavior.scaleDown.policies is also mentioned) and check what makes sense for us