GoogleContainerTools/skaffold

Support configuring remote cluster for pulling images

Open

#4,132 opened on May 8, 2020

View on GitHub
 (6 comments) (2 reactions) (0 assignees)Go (12,822 stars) (1,416 forks)batch import
area/confighelp wantedkind/feature-requestkind/frictionpriority/p3

Description

I'm trying to build a container on GCB, and then deploy it on my local Docker for Desktop Kubernetes cluster, but I seem to be hitting auth issues. The container builds fine, but the Kubernetes pod gets stuck in an error state. I went through https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper and set gcloud as the credential helper via gcloud auth configure-docker, but the problem still seems to persist.

Using this example: https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/java/java-hello-world

$ pwd
/Users/michihara/Code/cloud-code-samples/java/java-hello-world

$ skaffold version
v1.8.0

$ gcloud version
Google Cloud SDK 289.0.0
alpha 2020.02.25
app-engine-java 1.9.79
app-engine-python 1.9.90
beta 2020.02.25
bq 2.0.56
cloud-datastore-emulator 2.1.0
core 2020.04.10
gsutil 4.49
minikube
skaffold

$ gcloud auth list
   Credentialed Accounts
ACTIVE  ACCOUNT
*       michihara@google.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

$ gcloud auth configure-docker
WARNING: Your config file at [/Users/michihara/.docker/config.json] contains these credential helper entries:

{
  "credHelpers": {
    "us.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud",
    "gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud"
  }
}
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
gcloud credential helpers already registered correctly.

$ skaffold run -p cloudbuild --default-repo gcr.io/chelseamarket/p
Generating tags...
 - java-hello-world -> gcr.io/chelseamarket/p/java-hello-world:latest
Checking cache...
 - java-hello-world: Found Remotely
Tags used in deployment:
 - java-hello-world -> gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997
Starting deploy...
 - deployment.apps/java-hello-world configured
 - service/java-hello-world-external configured
Waiting for deployments to stabilize...
 - deployment/java-hello-world: waiting for rollout to finish: 1 old replicas are pending termination...
 - deployment/java-hello-world failed. Error: could not stabilize within 2m0s: context deadline exceeded.
FATA[0124] 1/1 deployment(s) failed

$ kubectl config current-context
docker-desktop

$ kubectl get all
NAME                                    READY   STATUS             RESTARTS   AGE
pod/java-hello-world-6d47c78bf9-xpm7f   0/1     ImagePullBackOff   0          36s

NAME                                TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/java-hello-world-external   LoadBalancer   10.101.0.92   localhost     80:30743/TCP   37s
service/kubernetes                  ClusterIP      10.96.0.1     <none>        443/TCP        45s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/java-hello-world   0/1     1            0           37s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/java-hello-world-6d47c78bf9   1         1         0       37s

$ kubectl describe pod/java-hello-world-6d47c78bf9-xpm7f
Name:           java-hello-world-6d47c78bf9-xpm7f
Namespace:      default
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Fri, 08 May 2020 11:27:08 -0400
Labels:         app=java-hello-world
                app.kubernetes.io/managed-by=skaffold-v1.8.0
                pod-template-hash=6d47c78bf9
                skaffold.dev/builder=google-cloud-build
                skaffold.dev/cleanup=true
                skaffold.dev/deployer=kubectl
                skaffold.dev/profile.0=cloudbuild
                skaffold.dev/run-id=3cffaa33-5716-4371-92ed-43be9f3a1d70
                skaffold.dev/tag-policy=sha256
                skaffold.dev/tail=true
Annotations:    <none>
Status:         Pending
IP:             10.1.1.133
IPs:            <none>
Controlled By:  ReplicaSet/java-hello-world-6d47c78bf9
Containers:
  server:
    Container ID:
    Image:          gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      PORT:               8080
      JAVA_TOOL_OPTIONS:  -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-q4ngt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-q4ngt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-q4ngt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                     Message
  ----     ------     ----               ----                     -------
  Normal   Scheduled  49s                default-scheduler        Successfully assigned default/java-hello-world-6d47c78bf9-xpm7f to docker-desktop
  Normal   BackOff    24s (x3 over 47s)  kubelet, docker-desktop  Back-off pulling image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997"
  Warning  Failed     24s (x3 over 47s)  kubelet, docker-desktop  Error: ImagePullBackOff
  Normal   Pulling    13s (x3 over 48s)  kubelet, docker-desktop  Pulling image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997"
  Warning  Failed     12s (x3 over 47s)  kubelet, docker-desktop  Failed to pull image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
  Warning  Failed     12s (x3 over 47s)  kubelet, docker-desktop  Error: ErrImagePull

Contributor guide