tracking: Add new integration test tooling

tracking: Add new integration test tooling

This issue tracks the design and implementation of an improved integration testing procedure of operators and products on Kubernetes clusters. The past showed that existing tools like kuttl and chainsaw are not sufficient for our use-cases. As such, it was decided that the best way forward is the design and development of a new tool to run automated integration tests against Kubernetes clusters. This tool will be called bbq (bebeku, better better kuttl).

The tool is developed with internal requirements in mind, but should obviously also be useful for the open-source community as a whole. We might want to position it as a kuttl and chainsaw replacement in the future.

Design tasks

This section lists many of the required design tasks which need to be done before initial implementation can start. It should however also be noted that not everything can or should be designed before starting to work on the implementation. Also, some design decisions might change along the way.

The design of individual aspects is split up into separate sections which list detailed design decisions for core requirements/goals.

At the end, the final design decisions must be noted down in a DESIGN.md document.

File and directory structure

(repository root)
|  # Configure global settings, test suites and dimensions
|- bbq.yaml
|
|  # The test directory
|- tests
|  |
|  |  # Test cases are just directories
|  |- my-test-case
|  |  |
|  |  |  # This defines the test case via a DAG. The exact format is
|  |  |  # discussed in the next section.
|  |  |- bbq.yaml
|  |  |
|  |  |  # Any number and depth of directories can be used to group
|  |  |  # manifests and assertions. The test case config references
|  |  |  # these directories.
|  |  |- keycloak
|  |  |  |- install.yaml
|  |  |  |- secrets.yaml
|  |  |- opa
|  |  |  |- install.yaml
|  |  |  |- rules.yaml
|  |  |- my-product.yaml
|  |- another-test-case
|  |  |- ...
|  |- ...
|- ...

There are a few open questions/design decisions to be done:

• Which language should be used for the various config files? YAML and TOML are the most obvious contenders - we are currently favouring YAML.
• Do we really want to have multiple bbq.yaml files? Having the whole definition in one file is convenient, but could get very large.
• Decide where the global config file should be located. Directly under the repository root or inside the tests directory.

Global configuration

[!NOTE]
There are still some decisions to be made about dimensions and the $bbq hints (and defaults).

dimensions:
  version:
  - v1.0
  - v1.2
  - v1.3
  tls_enabled: $bbq/bool # select list of valid strings for primitives
  openshift: $bbq/bool(false) # do we need the $bbq hint? We will need to specify defaults for opt-in things like openshift

# The idea is that these aren't dimensions to test against, but they can be used in interpolations.
# These won't end up in the test name.
# Idea: These _could_ be overridden via ENV/CLI?
consts:
  vector_address: vector-aggregator.vector-aggregator.svc

# Instead of scripting the namespace patching, we could do what
# kustomize allows with configuring metadata on generated objects.
namespaceGenerator:
  options:
    # How to do some conditional labels
    labels:
      - name: pod-security.kubernetes.io/enforce
        value: privileged
        if: openshift
      - name: foo
        value: bar

preHooks: ...
postHooks: ...

suites:
  # Want to run the latest version with tls enabled as a smoke test.
  smoke:
    # Note: There should be an option to select all cases. I would like to
    # avoid making it implicit by omitting the `cases` key for example.
    # Instead, I want this to be explicit.
    cases:
    - folder_1
    - folder_2
    dimensions:
      tls_enabled: true
      # Question: Which syntax do we wanna use for the special selectors?
      # I would also suggest to namespace them like `bbq/last` to avoid any
      # collisions with other tools.
      version: $bbq/last

Test case definition and configuration

With bbq we want to introduce a new and better way to define tests and the dependencies in these tests. To more efficiently utilize concurrency (which reduces test run time and setup time), we aim to use a DAG (with optional hooks). Also see the petgraph crate which provides a directed acyclic graph implementation.

• Each test-case is defined as a DAG. The current plan is to design this file with inspiration from GitHub Action workflows (namely, the job and step definitions, the needs keyword to define the dependency graph).
• Define a well-known set of special namespaced values, eg. $bbq/last or $bbq/bool which can be used to set various fields in the definition files.
• Think about hooks (pre-run and post-run). On which level are these supported? How can they be used to install common stuff only once cluster-wide to avoid re-installing common stuff over and over again for different tests.

Here's an initial idea around how we can support as much concurency as possible, while allowing sequences of steps (especially considering scripts which could have side-effects).
For now, assume hooks would work similarly - we might consider reusable hooks.

# would look much like the tasks. They just all run before tests start
# We chose to separate preHooks and postHooks (instead of having a single `hooks`
# key) so that the file can be laid out chronologically.
# We might want to consider reusable hooks/tasks (eg: install vector), and/or...
# We might even move this out to the global config (to run before all test cases).
# Implementation note: We can ignore hooks until we get tasks running - this
# defers some decisions without blocking the initial development.
preHooks: []

tasks:
# Note: this task would likely be moved into preHooks.
- id: minio
  steps:
    # Assuming parallelism of 5, and containing 10 documents including manifests
    # to apply and assertions the first 5 will be scheduled, while the rest are
    # queued. The next step (script) will not begin until all 10 items have
    # successfully finished.
    - manifest: ../shared/minio/secret.yaml # 5 manifests, 
    # But these steps run in sequence after the manifests are applied (and asserted)
    - script: |
        helm install minio
          --namespace $NAMESPACE
          --version 12.6.4
          -f ../shared/minio/helm-values.yaml
          --repo https://charts.bitnami.com/bitnami minio
    - script: |
        POD=$(kubectl -n $NAMESPACE get pod -l app.kubernetes.io/instance=minio -o name | head -n 1 | sed -e 's#pod/##')
        kubectl cp -n $NAMESPACE yellow_tripdata_2021-07.csv $POD:/tmp
        kubectl -n $NAMESPACE exec $POD -- mc cp /tmp/yellow_tripdata_2021-07.csv local/trino/taxi-data/

- id: postgres
  steps:
   - manifests: postgres-secret.yaml
   # This script could run at the same time as one of the scripts defined above.
   # The only way to
   - script: helm install postgres ...

- id: trino
  needs: [hive, hdfs, minio]
  steps:
    # These will be done in sequence, with each being applied in parallel
    - manifest: trino/cluster.yaml
    - manifest: trino/catalogs.yaml
- id: hive
  needs: [postgres]
  steps: []
- id: hdfs
  needs: [zookeeper]
  steps: []
- id: zookeeper
  steps: []
  
# Any special cleanup tasks
# Manifests applied in above steps would be unapplied at the end.
postHooks: []

Manifest definition

• There should be no need to number the files to define the order of execution. This should instead be handled by the test case DAG definition.
• Support multiple levels of directories (directories have not special meaning)
• Defined as plain YAML files (we have chose YAML by default, but we could explore other config languages)

Assertions

There are multiple ideas on how to declare assertions.

A common idea which can be applied to all approaches below is to allow developers/users to add descriptions and error notes/links to assertions. This could look something like this:

# A summary of what the assertion is for
description: Assert that there is no more than one replica

# Additional help for the tester that is displayed when the 
# assertion failed
error-notes: |
  This can occastionally fail on AKS clusters due to the 
  storage-provisioner.
  
  Try re-running the test.

# Templated links to display on error
# Think about common top-level/global links we could provide an a test suite
# or test case level.
error-links:
  - https://logs.example.com/search?q=namespace=%namespace&cluster_name=${env:KUBERNETES_CLUSTER_NAME}

Assert by using labels

Be able to place assertions in the same file that created them. Assertions are then marked using annotations/labels. This mechanism could look something like this:

This has the advantage of being able to be copy/pasted from a real object with minimal adjustments to make it an assertion (just a label added).

# Install a NiFi cluster with 2 nodes
---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
image:
  productVersion: 2.6.0
nodes:
  roleGroups:
    default:
      replicas: 2
# Assertions indicated by an annotation/label or in a different way?
# Assert that there is a service and is internal to the cluster
---
apiVersion: v1
kind: Service
metadata:
  name: simple-nifi-node-default
labels:
  bbq.stackable.tech/assertion: "true"
spec:
  type: ClusterIP
# Assert that the StatefuleSet has ready pods
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: simple-nifi-node-default
labels:
  bbq.stackable.tech/assertion: "true"
status:
  readyReplicas: 2
  replicas: 2

Leverage CEL

Another idea was to integrate with CEL with the additional idea to provide custom functions to the evaluation environment. A rough example might look like this:

• https://kubernetes.io/docs/reference/using-api/cel/
• https://github.com/google/cel-spec/blob/master/doc/langdef.md
• https://codelabs.developers.google.com/codelabs/cel-go#0
• https://github.com/cel-rust/cel-rust

apiVersion: bbq.stackable.tech/v1alpha1 # or bbq.kore.rs/v1alpha1
kind: CELAssertions # or TestAssertion or Assertion
metadata:
  name: pod-is-named-foo-and-has-3-containers
assertions:
# This doesn't interact with k8s, only looks the same.
# Consider what type we need to construct in order to lookup the _thang_,
# and conversion methods.
- selector:
    apiVersion: v1 # Required
    kind: Pod # Required
    # select on `name` or `labels`, not both, and not anything else.
    metadata:
      name: foo # Option<String>
      # or (if name is not set)
      labels:
        foo: bar
  assertion: |
    self.containers.size() == 3
- selector:
    apiVersion: v1
    kind: Service
    metadata:
      name: foo
  assertion: |
    self.spec.type == 'ClusterIP'
- selector:
    apiVersion: v1
    kind: StatefulSet
    metadata:
      name: simple-nifi-node-default
  assertion: |
    self.status.readyReplicas == 2
    self.status.replicas == 2
# Challenge: Can we assert an env var for a particular container in a Pod?
# Yes we can!
- selector: | ...
  assertion: |
    self.spec.containers
      // We could make our own custom function to combine filter() and map() into filter_map()
      .filter(c, c.name == 'nifi')
      .map(c, c.env)
      .filter(e, e.name == 'MY_ENV') == 'foo'
# Provide an error message and additional links to help fix/investigate the error
- selector: | ...
  assertion: |
    self.status.replicas == 2
  message: Replica count was less than 2
  links:
    - https://example.org

Leverage scripts

This would allow local scripts cases where other methods are not flexible/powerful enough.

[!IMPORTANT]
We should consider whether we really want to allow local scripts to be run.
One idea could be to deploy a job to run in-cluster (local scripts can still be applied in ConfigMaps and mounted to the job).

This would keep this to being focused on assertions and not about setting up for assertions.

apiVersion: bbq.stackable.tech/v1alpha1 # or bbq.kore.rs/v1alpha1
kind: ScriptAssertions
metadata:
  name: We are free to make this whatever (we aren't applying it to k8s)
assertions:
# We could have defaults for shells, eg:
# bash: -euo pipefail -c <script body>
# sh: -eu -c <script body>
# python: -c <script body>
- script: |
    echo "This is the same as below..."
- shell: bash
  script: |
    echo "Running in bash with bbq default shell flags"
- shell: bash
  script: |
    set +u
    echo "Deviate from the default shell flags"
- shell: python
  script: |
    def do_things:
      pass
      
    print("about to do things")
    do_things
- script: |
    echo "oops, looks like we're taking the L"
    exit 69

---

We could also allow a combination of two or more approaches.

Diff rendering

• More compact so you only see what you need to see.

• Possible idea: use jmespath (or similar) to only show scalars? See also json_diff!

  at spec/container/[name=app]/env/[name=FOO]/value expected `BAR` got `BAZ`
  

• Enough context to see where the diff lies (eg: name of container is not omitted)

   apiVersion: v1
   kind: Pod
   metadata:
     name: app
   spec:
     containers:
     - env:
       - name: FOO
  -      value: BAR
  +      value: BAZ
       image: blah:latest
       name: app
  

Test suite definition and configuration

• A test suite is a collection of tests that have something in common. It also contains setup and teardown mechanims to for example install operators or databases.
  • Example test suites: Smoke tests for all versions, ...
  • Multiple test suites can be run using the same underlying cluster by calling bbq multiple times.
• A test case is a set of steps with optional assertions, eg:
  • Install this
  • Install that
  • Assert this
  • Assert that
• ...

A note about runtime

bbq won't store any external state. Instead, if it encounters objects that already exist, it will error out.
For example: The auto-generated namespace exists, or a cluster-scoped resource exists (each of these objects - if deployed by bbq - will have labels, so maybe we can print out extra information before exiting or moving on to the next test if fail-fast is not enabled).

Once bbq supports pausing and continuation (for developing test), we still don't need to store state because it will be done in the same invokation (requiring key-press to continue, rather than a new invokation). See the CLI design section below for more details.

We probably should consider how to manage stateless invokations when developing tests (eg: when a cluster scoped resource that we would apply already exists - but we expect it in this case. Maybe a flag to force?).

Templating

We want to provide native templating mechanisms instead of bolting it onto an existing tool (that's exactly how beku worked). We should aim to basically make every piece of dynamic data available during templating.

• Define a well-known set of provided templating variables with a clear hierarchy/namespacing. Provide a list with name, type, default and example values
• Define a well-known set of helpers
• Ability to export the rendered manifests

Traceability of tests

In order to improve the inspection and debugging of (failed) tests, we need to improve the traceability of our tests tremendously. As such, we want to move to an OTel-first approach with bbq. We can already collect traces and logs for our operators, a bunch of the products, the Kubernetes cluster/apiserver and in the future bbq as well.

• Define well-known label prefix, eg. bbq.kore.rs or bbq.stackable.tech.
• Add label for namespace.
• Add labels for test case, suite, selected dimensions and their values
• Add labels for run-conditions, like the Kubernetes version and distribution, test and job parallelism.
• Utilize OpenTelemetry mechanisms for logging, tracing, and metrics.
• Collect Kubernetes traces, logs and metrics alongside the test telemetry data to easily provide a complete (and temporally aligned) overview of test runs without tediously looking through multiple places where data is stored/collected.
• Attach attributes/fields to traces, logs, and metrics for easier filtering and drill-downs.
• Define semantic conventions.
• Grafana dashboards (high level overviews of all tests, and drill down dashboard for troubleshooting).

Progress reporting and log output

The current progress and log output in kuttl is basically useless. In successful cases, way to much noise is emitted. In failure cases, the errors are hard to spot and sometimes not really visible. The progress of test runs is basically impossible to gauge currently.

• Make use of progress reporting mechanisms, like spinners, progress bars, counters, etc...
• Provide de-cluttered log output which focuses on important messages.
• Auto-detect CI (non-interactive/no tty) environments and streamline the output for these situations in addition to the rich (interactive) output. A rough example might look like this:

  Starting test suite {name: foo, parallel: 2}
  Running pre-run hooks
  Queuing test 3 cases: ...
  Starting test case {name: bar, namespace: bar-abc, parallel: 8}
  Starting test case {name: baz, namespace: baz-xyz, parallel: 8}
  bar (bar-abc): Applying manifest my-manifest.yaml
  ...
  Finished test case (1/2) {name: bar}
  Finished test suite {name: foo}
  Done!
  

• Optional/down the line: Provide a TUI for a better DX (that means most code will be developed as a library, and initially we implement a CLI).

Persistence of test results

The CRA (and possibly other regulations???) mandate the long-term storage of test results. As such, the results including traces, logs and metrics must be persisted. The retrieval and inspection of these test results must be as easy as running the tests themself. The searchability of these past tests should be fairly high because we attach a whole bunch of metadata through OTel attributes/fields.

• As OpenTelemetry is used, the test results (traces, logs, metrics) can be stored and viewed in any supported OTel stack, like Grafana + Loki + Tempo.
• The tool itself can provide means to retrieve historical data which can be viewed via the terminal or possibly as files (explore the use of HAR (or something inspired by it) and JSON files).
• Optional: Produce TAP (or other common outputs) output to be consumed by other tooling.

Automatic retry mechanisms

It is a well-known fact that a bunch of tests are flaky, because they depend on external resources or sometimes the Kubernetes apiserver is overloaded on our in-situ test clusters. As such, there should be automatic retries of failed tests if the user selected to run the tests with retries enabled.

• Define on which levels a retry mechanism should be available.
• Define how the mechanism works (when to retry, how often to retry, when are retries scheduled when other tests are still in-flight).
• Think about an option to permanently enable retries for a known flaky test/step/hook.
• Explore incorporating Kubernetes-specific diagnostics into failure/retry detection.

CLI design

As this is the main way (before a TUI might be implemented) with which users interact with the integration test tooling, this needs to be designed exceptionally well. The clap crate provides powerful mechanisms to build these CLIs. We already use clap across many of our apps (operators and tools) and as such it is strongly recommended to be used for bbq as well.

• Consistent argument naming.
• Provide short arguments for commonly used arguments. Less frequently used ones should not "waste" short arguments.
• Provide clear and extensive help texts for all (sub)commands and arguments.
• Provide value hints for as many arguments as possible.
• Explore the option to add commands which ease the creation of new test suites, test cases, etc. Also could snapshotting be used for this? Basically create a snapshot of some manifest(s) and store them as test manifests.
• If no tests are selected, the tool must exit with a non-0 exit code instead of silently running nothing and reporting it as a success.
• Add an argument to enable "fail fast" mode (exit on first test failure instead of running all selected tests until they are finished), eg:

  bbq run --fail-fast
  

• Add an argument to skip cleanup tasks when the test (suite or case) finishes (and optionally fails), eg:

  bbq run --skip-cleanup # Shorthand (for success and failures)
  bbq run --skip-cleanup=failure # Explicitely say to skip cleanup on failures
  

• Add an argument to pause between tests (and jobs) for human checks during development/debugging, eg:

  bbq run --pause-tests
  bbq run --pause-jobs
  bbq run --continue # Or not exit and wait for input
  bbq run --abort
  

  Consider what happens when files are edited during pauses (should bbq re-read the next files?)
• Easy selection of test case, test suite, eg:

  bbq run --case <MY_CASE>
  bbq run --suite <MY_SUITE>
  

• Add argument to override dimensions, eg:

  bbq run --dimension tls_enabled=true
  # or
  bbq run --set tls_enabled=true
  

• Add argument to filter based on dimensions, eg:

  bbq run --where tls_enabled=false
  # or
  bbq run --filter tls_enabled=false
  

• Add a command for rendering test cases/suites (consider a better command name):

  bbq template --case <MY_CASE> > rendered.yaml
  bbq template --suite <MY_SUITE> > rendered.yaml
  # or
  bbq render
  

• Add argument to specify the test directory path and the config file path, eg:

  bbq run --test-directory my-tests # Defaults to 'tests'
  bbq run --config boo.{toml,yaml} # Defaults to '<TEST_DIRECTORY>/bbq.{toml,yaml}'
  

• Add command (with filtering support) to list test suites, cases and dimensions, eg:

  # Will list all defined dimensions with all (?) their possible values
  $ bbq list dimensions
  version: [...]
  tls_enabled: [true, false]
  timeout: [30s]
  ...
  
  # Will list all test suites or ones filtered by the provided dimensions
  $ bbq list suites --where tls_enabled=true
  smoke
  
  # Will list all test cases, which are just folders. Can be filtered by suites.
  # This could be `bbq describe cases` instead of `--output=brief`
  $ bbq list cases --suite smoke --output=brief
  smoke {version=[3.0.6,3.0.7], tls_enabled=bool, load_dags=bool}
  smoke_latest {version=3.0.7, tls_enabled=bool, load_dags=bool}
  
  # List out all combinations of tests for the suite.
  # If using `bbq describe cases` like mentioned above, then we don't
  # need `--output=verbose/expanded`.
  # This output should be copy+pastable for bbq run --case <CASE>
  $ bbq list cases --suite smoke --output=verbose/expanded
  smoke {version=3.0.6, tls_enabled=true, load_dags=true}
  smoke {version=3.0.6, tls_enabled=false, load_dags=true}
  smoke {version=3.0.6, tls_enabled=true, load_dags=false}
  smoke {version=3.0.6, tls_enabled=false, load_dags=false}
  ...
  
  # Showing different output in case of too many dimensions making output
  # hard to read.
  $ bbq list cases --suite smoke --output=yaml
  smoke:
    version: [3.0.6,3.0.7]
    tls_enabled: bool(ean)
    load_dags: true
    # or
    # flags/bool(eans): [tls_enabled,load_dags]
  

• Add argument to control test and worker parallelism, eg:

  # Or just --paralellism, because this is probably used more often than
  # --worker-parallelism
  $ bbq run --test-parallelism
  
  $ bbq run --worker-parallelism
  

Implementation tasks

Stage 1

Get something working

[!NOTE]
There is existing prototyping code. See what can be reused from that.

• Reading multi-doc YAMLs, and extracting manifests to apply and assertions
  • Track which assertions were already run and which ones still need to be run
  • Template manifests (context: namespace, k8s stuff, dimensions), snapshot testing?
• DAG Executor with instrumentation, stages and hooks (with concurrency settings).
  • This should be testable without k8s (maybe the executor fires events to channels, and a Reactor does the work).
• Simple Console logging for now (we should start to see more attributes appearing)
  • Just so we see something
• Define semantic conventions and label prefixes, and begin using them.
  • Consider upstreaming generic testing semantic conventions.
• Clap (sub)commands/args as needed
• Model config files (naming consistent with Clap commands/args)
  • represent a DAG and stages

At this point, tests can be run (minus scripts), but the UX is not much better than existing tooling.

Stage 2

Make it look nicer and be on-par with beku/kuttl

• Tidy diffing (on assertion failures)
  • Less noise in diff output
• Buffered output (de-noising)
  • Print complete output on failure only
• Look into indicatif for fancy status output (spinners, counters, bars, etc...)
• Support scripting (a manifest with a known Kind)
  • This might get moved to Stage 1 if necessary.

At this point, UX is much nicer than existing tooling and we can replace buku/kuttl.

Stage 3

Polishing

• OTLP exporter (for easier troubleshooting on the observability stack)
  • Should work in conjunction with the buffered output, but is realtime.
• Grafana dashboards
• Ability to pause between stages/test - allow reloading of files (this might get moved up to stage 2)

Documentation tasks

• Create a complete (Markdown-based) documentation page, similar to what kuttl and chainsaw have.
• Create a DESIGN.md file detailing all technical design decisions.
• Create an auto-generated changelog.

Acceptance criteria

• Provide pre-compiled binaries for at least Linux (x86_64 and aarch64) and Mac (aarch64). Use immutable releases on GitHub.
• Able to convert and successfully run all Stackable tests.
• A whole bunch more...