Initial failed deployment can not be fixed

Open
#147 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
kubernetes, postgresql, redis, rust

Research direction

Use airflow.yml and the AirflowCluster reconciliation path as the entry point; reproduce the failed PostgreSQL initialization, delete and reapply the resource, and inspect how the operator records terminal failure after its retries. Done means correcting the connection and reapplying causes database initialization and cluster deployment to retry and succeed without reinstalling the operator or its CRD.

Written by the indexing model from the issue text.

Description

Affected version

0.5.0-nightly

Current and expected behavior
  • stackablectl --namespace=stackable operator install airflow
    
  • helm repo add bitnami https://charts.bitnami.com/bitnami
    
  • helm install airflow-postgresql bitnami/postgresql --version 11.0.0 \
    --set auth.username=airflow \
    --set auth.password=airflow \
    --set auth.database=airflow
    
  • helm install redis bitnami/redis \
    --set auth.password=redis
    
  • Wait for all ressources are deployed

  • create airflow file

  • # airflow.yml
    apiVersion: v1
    kind: Secret
    metadata:
      name: simple-airflow-credentials
    type: Opaque
    stringData:
      adminUser.username: airflow
      adminUser.firstname: Airflow
      adminUser.lastname: Admin
      adminUser.email: [email protected]
      adminUser.password: airflow
      connections.secretKey: thisISaSECRET_1234
      connections.sqlalchemyDatabaseUri: postgresql+psycopg2://airflow:[email protected]/airflow
      connections.celeryResultBackend: db+postgresql://airflow:[email protected]/airflow
      connections.celeryBrokerUrl: redis://:redis@redis-master:6379/0
    
    ---
    apiVersion: airflow.stackable.tech/v1alpha1
    kind: AirflowCluster
    metadata:
      name: airflow
    spec:
      version: 2.2.4-python3.9-stackable0.3.0
      statsdExporterVersion: v0.22.4
      executor: CeleryExecutor
      loadExamples: true
      exposeConfig: false
      credentialsSecret: simple-airflow-credentials
      webservers:
        roleGroups:
          default:
            replicas: 1
      workers:
        roleGroups:
          default:
            replicas: 2
      schedulers:
        roleGroups:
          default:
            replicas: 1
    
  •   kubectl apply -f airflow.yml
    

This config has a bug (wrong database url)

The operator now tries to deploy airflow but the database init fails.
After seven retries all the retries are done and we are in a failure state.
Now I deleted the deployment with
kubectl delete -f airflow.yml

After that i fixed the bug and run kubectl apply -f airflow.yml. This does not try again the database init and just goes to error state.

My expected behavior would be that the database init would tried again and succeeds this time.

Possible solution

Delete the airflow operator and the Custom ressource definition and reinstall the operator. Then apply the correct yaml file. This time the database init succeeds.

Additional context

No response

Environment

Client Version: v1.22.4
Server Version: v1.23.9

Would you like to work on fixing this bug?

No response

Dominant language
Rust
Stars
32
Forks
6
Avg merge
1d 6h
Merged PRs (30d)
12

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stackabletech/airflow-operator

All issues in stackabletech/airflow-operator

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.