projectcalico/calico

Migration from Manifest to Operator install fails on microk8s; `missing expected volume 'var-run-calico' with hostPath '/var/run/calico'.`

Open

#10,733 opened on Jul 27, 2025

View on GitHub
 (7 comments) (1 reaction) (0 assignees)Go (1,594 forks)auto 404
area/operatorgood first issuehelp wantedkind/enhancement

Repository metrics

Stars
 (7,292 stars)
PR merge metrics
 (PR metrics pending)

Description

Expected Behavior

I would expect that the migration works.

Current Behavior

The current behavior is a failure:

❯ kubectl describe tigerastatus calico
Name:         calico
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  operator.tigera.io/v1
Kind:         TigeraStatus
Metadata:
  Creation Timestamp:  2025-07-27T22:23:45Z
  Generation:          1
  Resource Version:    203984155
  UID:                 a94d79df-1377-4f55-80d0-c2fb7c5fccb2
Spec:
Status:
  Conditions:
    Last Transition Time:  2025-07-27T22:23:50Z
    Message:               Existing Calico installation can not be managed by Tigera Operator as it is configured in a way that Operator does not currently support. Please update your existing Calico install config: missing expected volume 'var-run-calico' with hostPath '/var/run/calico'. To fix it, add the expected volume to daemonset/calico-node on daemonset/calico-node
    Observed Generation:   1
    Reason:                MigrationError
    Status:                True
    Type:                  Degraded
Events:                    <none>

Context

I am running a small microk8s (v1.33.0) cluster; 3 nodes. All are amd64 running latest ubuntu 24.04

root@node03:/var/snap/microk8s/current/args/cni-network# microk8s kubectl version
Client Version: v1.33.0
Kustomize Version: v5.6.0
Server Version: v1.33.0
root@node03:/var/snap/microk8s/current/args/cni-network# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble
root@node03:/var/snap/microk8s/current/args/cni-network# uname -a
Linux node03 6.8.0-55-generic #57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

For background, the microk8s project bundles calico and installs it via manifests and then never touches it again. Any updates / configuration tweaks are done via changes to the manifests and then a re-apply.

I have been doing updates from time to time by patching the image: ... to the latest version of calico and that's been fine. Just a few hours ago I migrated to 3.30.2 from 3.28.1.

I am currently trying to migrate to the Operator based pattern so I have additional CRDs at my disposal.

Unfortunately, the migration has failed with the above error.

How I got here:

❯ kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.2/manifests/operator-crds.yaml
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io serverside-applied
<...>
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org serverside-applied
<...>
customresourcedefinition.apiextensions.k8s.io/baselineadminnetworkpolicies.policy.networking.k8s.io serverside-applied

❯ kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.2/manifests/tigera-operator.yaml
namespace/tigera-operator serverside-applied
serviceaccount/tigera-operator serverside-applied
clusterrole.rbac.authorization.k8s.io/tigera-operator-secrets serverside-applied
clusterrole.rbac.authorization.k8s.io/tigera-operator serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator serverside-applied
rolebinding.rbac.authorization.k8s.io/tigera-operator-secrets serverside-applied
deployment.apps/tigera-operator serverside-applied

❯ kubectl create -f - <<EOF
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec: {}
EOF
installation.operator.tigera.io/default created

And then the very next command in shell history is describe tigerastatus calico which gives me the error about /var/run/calico not being as expected.

But here's the kicker, the existing CNI config does have a var-run-calico volume defined:

root@node03:/var/snap/microk8s/current/args/cni-network# cat cni.yaml | grep -b5 var-run-calico
359397-            #              name: sys-fs
359437-            #              # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host
359567-            #              # so that it outlives the init container.
359636-            #              mountPropagation: Bidirectional
359695-            #            - mountPath: /var/run/calico
359749:            #              name: var-run-calico
359797-            #              # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host
359934-            #              # so that it outlives the init container.
360003-            #              mountPropagation: Bidirectional
360062-            #            # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,
360190-            #            # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.
--
366666-                        readOnly: true
366705-                      - mountPath: /run/xtables.lock
366758-                        name: xtables-lock
366801-                        readOnly: false
366841-                      - mountPath: /var/run/calico
366892:                        name: var-run-calico
366937-                        readOnly: false
366977-                      - mountPath: /var/lib/calico
367028-                        name: var-lib-calico
367073-                        readOnly: false
367113-                      - name: policysync
--
367593-            volumes:
367614-                # Used by calico-node.
367653-                - name: lib-modules
367689-                  hostPath:
367717-                      path: /lib/modules
367758:                - name: var-run-calico
367797-                  hostPath:
367825-                      path: /var/snap/microk8s/current/var/run/calico
367895-                - name: var-lib-calico
367934-                  hostPath:
367962-                      path: /var/snap/microk8s/current/var/lib/calico

And both /var/run/calico and /var/snap/microk8s/current/var/run/calico exist:

root@node03:/var/snap/microk8s/current/args/cni-network# ls -lah /var/snap/microk8s/current/var/run/calico
total 16K
drwxr-xr-x 4 root root 4.0K Jul 27 13:36 .
drwxr-xr-x 4 root root 4.0K Mar  4  2024 ..
drwx------ 2 root root 4.0K Mar  4  2024 cgroup
drwxr-xr-x 2 root root 4.0K Jul 27 15:10 endpoint-status
root@node03:/var/snap/microk8s/current/args/cni-network# ls -lah /var/run/calico/
total 0
drwxr-xr-x  2 root root   60 Jul 27 12:59 .
drwxr-xr-x 33 root root 1.1K Jul 27 12:59 ..
-rw-------  1 root root    0 Jul 27 12:59 ipam.lock

It's not clear to me what the issue is or if there's a setting in the kind: Installation that would let me adjust where the operator thinks var-run-calico should be?

Contributor guide