nfs: Failed to resolve server nfs-server.default.svc.cluster.local: Name or service not known

BUG REPORT

Environment:

Minikube version: v0.30.0

• OS: Fedora 29
• VM Driver: virtualbox, kvm2
• ISO version: v0.30.0
• Others:
  • kubernetes version: tested on v1.10.0, v1.13.0
  • tested with coredns and kube-dns minikube addons

What happened: NFS volume fails to mount due to DNS error (Failed to resolve server nfs-server.default.svc.cluster.local: Name or service not known). This problem does not occur when deployed on GKE.

What you expected to happen: NFS volume is mounted without an error.

How to reproduce it (as minimally and precisely as possible):

1. Start nfs-server:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-server
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: gcr.io/google_containers/volume-nfs:0.8
        ports:
        - name: nfs
          containerPort: 2049
        - name: mountd
          containerPort: 20048
        - name: rpcbind
          containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /exports
          name: exports
      volumes:
      - name: exports
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: nfs-server
spec:
  ports:
  - name: nfs
    port: 2049
  - name: mountd
    port: 20048
  - name: rpcbind
    port: 111
  selector:
    role: nfs-server

2. Start service consuming the nfs volume (e.g. busybox):

apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-busybox
spec:
  replicas: 1
  selector:
    name: nfs-busybox
  template:
    metadata:
      labels:
        name: nfs-busybox
    spec:
      containers:
      - image: busybox
        command:
          - sh
          - -c
          - 'while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $(($RANDOM % 5 + 5)); done'
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
          - name: nfs
            mountPath: "/mnt"
      volumes:
      - name: nfs
        nfs:
          server: nfs-server.default.svc.cluster.local
          path: "/"

Output of minikube logs (if applicable): In kubectl describe pod nfs-busybox-... is this error:

  Warning  FailedMount  4m    kubelet, minikube  MountVolume.SetUp failed for volume "nfs" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/ab2e9ad4-f88b-11e8-8a56-4004c9e1505b/volumes/kubernetes.io~nfs/nfs --scope -- mount -t nfs nfs-server.default.svc.cluster.local:/ /var/lib/kubelet/pods/ab2e9ad4-f88b-11e8-8a56-4004c9e1505b/volumes/kubernetes.io~nfs/nfs
Output: Running scope as unit: run-r23cae2998bf349df8046ac3c61bfe4e9.scope
mount.nfs: Failed to resolve server nfs-server.default.svc.cluster.local: Name or service not known

Which indicates problem with DNS resolution for nfs-server.default.svc.cluster.local.

Note: The NFS is mounted successfully when specified by ClusterIP instead of domain name.

Anything else do we need to know: The same problem was reported already for previous version #2218, but it is closed due to inactivity of the author and no-one seems to really looked into it. There is a workaround for this, but it is required to do it every time a minikube VM is created.

When running kubectl exec -ti nfs-busybox-... -- nslookup nfs-server.default.svc.cluster.local:

Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   nfs-server.default.svc.cluster.local
Address: 10.105.22.251

*** Can't find nfs-server.default.svc.cluster.local: No answer

Where strangely the service ClusterIP is present (when using kube-dns the service ClusterIP part is missing completely).