Fix the processing of environment variables

Problem description

Environment variables are handled inconsistently in the operators.

Some operators apply the overrides at the end (https://github.com/stackabletech/opensearch-operator/blob/de6bff9911d9a9d1f55266ace24d05034f00120e/rust/operator-binary/src/controller/build/node_config.rs#L402) and others apply them in between (https://github.com/stackabletech/airflow-operator/blob/e5ef43c4b3d98b85b1fcb853beed1bfc72a7366a/rust/operator-binary/src/controller/build/properties/env_vars.rs#L259-L271). It is unclear if an override is effective or not.

Some operators use a set to assemble all environment variables (https://github.com/stackabletech/opensearch-operator/blob/de6bff9911d9a9d1f55266ace24d05034f00120e/rust/operator-binary/src/controller/build/node_config.rs#L349) and others add additional environment variables to the list (https://github.com/stackabletech/hbase-operator/blob/c06e9a4fe10d090233dc348a389d24f6c960686e/rust/operator-binary/src/controller/build/resource/statefulset.rs#L187-L192). It is unclear if an environment variable is duplicated by an override. If the override is removed, then also an environment variable provided by the operator could vanish (see kubernetes/kubernetes#58477).

The environment variables which are assembled in a set, are sorted alphabetically and the ones which are appended to the container directly, are ordered as they appear in the code. If dependent environment variables are used, then it is a game of chance if they appear in the right order. The OpenSearch operator uses dependent environment variables internally (https://github.com/stackabletech/opensearch-operator/blob/de6bff9911d9a9d1f55266ace24d05034f00120e/rust/operator-binary/src/controller/build/node_config.rs#L345) and prefixes POD_NAME with _ to force the correct order. This does not work reliably for overrides where dependent environment variables are also actually used.

Solution

Provide one consistent solution which is implemented (and documented) in all operators.

All environment variables must be assembled in one EnvVarSet. Duplicates are not possible. The overrides must be applied last. The iterator of the EnvVarSet ensures that dependent environment variables are ordered correctly and deterministically (i.e. alphabetically or directly after the referenced variables).

part of stackabletech/internal-issues#167

Tasks

• stackabletech/operator-rs#1249