Microk8s - Error: container <container-name> is waiting to start: <image with missing tag> can't be pulled

   $ skaffold dev
...
    - pod/server-deployment-5f496cf9c7-2mn6x: container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled

Expected behavior

Image should be pulled/available and everything should work

Actual behavior

deployment fails -- is this user error?

https://github.com/aliwatters/dkc-mutli-k8s/tree/feature-skaffold-minimal (minimal branch)

Information

I'm hitting this error on a unbuntu 20.04 installation, with microk8s and skaffold.

container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled

I'm logged into docker.

Full error is

exiting dev mode because first deploy failed: 1/1 deployment(s) failed
ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold dev
Listing files to watch...
 - aliwatters/dkc-multi-server
Generating tags...
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:90d1283
Checking cache...
 - aliwatters/dkc-multi-server: Found Locally
Tags used in deployment:
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909
Starting deploy...
 - service/server-cluster-ip-service created
 - deployment.apps/server-deployment created
Waiting for deployments to stabilize...
 - deployment/server-deployment: creating container server
    - pod/server-deployment-5f496cf9c7-8dfp8: creating container server
    - pod/server-deployment-5f496cf9c7-hzc2l: creating container server
    - pod/server-deployment-5f496cf9c7-2mn6x: container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled
 - deployment/server-deployment failed. Error: creating container server.
Cleaning up...
 - service "server-cluster-ip-service" deleted
 - deployment.apps "server-deployment" deleted
exiting dev mode because first deploy failed: 1/1 deployment(s) failed

All the relevant files;

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat skaffold.yaml 
apiVersion: skaffold/v2beta11
kind: Config
build:
  artifacts:
    - image: aliwatters/dkc-multi-server
      context: server
      docker:
        dockerfile: Dockerfile.dev
      sync:
        manual:
        - src: '**/*.js'
          dest: .
  local:
    push: false
deploy:
  kubectl:
    manifests:
      - k8s/server-cluster-ip-service.yaml
      - k8s/server-deployment.yaml

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: aliwatters/dkc-multi-server
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
          env:
            - name: REDIS_HOST
              value: redis-cluster-ip-service
            - name: REDIS_PORT
              value: '6379'
            - name: PGUSER
              value: postgres
            - name: PGHOST
              value: postgres-cluster-ip-service
            - name: PGPORT
              value: '5432'
            - name: PGDATABASE
              value: postgres
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: pgpasswd
                  key: PGPASSWORD

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-cluster-ip-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 5000
      targetPort: 5000

And the microk8s config

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ more ~/.kube/config 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBVENDQWVtZ0F3SUJBZ0lKQUx
aR3B0NDZ... snip ..EhqYThFZjBsMzVFdmxFa1pJeGpwOHh5NEVLUHMKTFpCbDB3TT0KLS0tLS1FTkQgQ0VSV
ElGSUNBVEUtLS0tLQo=
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: VDlqa3BnWlR4Q3UzaHhpQnFvamlkbDdBRk9aMzdBcDkvMFhZT21ad1VqND0K

Versions

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ snap list
Name                  Version                     Rev    Tracking         Publisher    Notes
kubectl               1.20.2                      1811   latest/stable    canonical✓   classic
microk8s              v1.20.1                     1910   1.20/stable      canonical✓   classic

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:48:42Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:50:46Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold version
v1.18.0

Steps to reproduce the behavior

1. https://github.com/aliwatters/dkc-mutli-k8s/tree/feature-skaffold-minimal
2. skaffold dev

Additonal info

I posted on stackoverflow -- and received a comment suggest I run through the private image steps https://stackoverflow.com/questions/65964535/skaffold-and-microk8s-container-server-is-waiting-to-start-image-cant-be?noredirect=1#comment116673535_65964535

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret -- seems like a red herring as the image is not in a private registry.

$ kubectl get secret regcred --output=yaml
apiVersion: v1
data:
  .dockerconfigjson: ewoJ<snip>KfQ==
kind: Secret
metadata:
  creationTimestamp: "2021-02-01T15:36:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:.dockerconfigjson: {}
      f:type: {}
    manager: kubectl-create
    operation: Update
    time: "2021-02-01T15:36:02Z"
  name: regcred
  namespace: default
  resourceVersion: "2368742"
  selfLink: /api/v1/namespaces/default/secrets/regcred
  uid: 7f292deb-6d6a-4ea8-97c4-b3c2966d7abc
type: kubernetes.io/dockerconfigjson

Added

      imagePullSecrets:
        - name: regcred

to the end of my k8s/server-deployment.yaml -- made no difference.

Thanks for looking!

Added: https://github.com/GoogleContainerTools/skaffold/issues/5334 - related?