kubernetes/minikube

Minikube tunnel is not working on Windows (endessly trying to add a route)

Open

#11,645 opened on Jun 13, 2021

View on GitHub
 (14 comments) (4 reactions) (0 assignees)Go (31,799 stars) (5,222 forks)batch import
area/tunnelgood first issuehelp wantedkind/buglifecycle/frozenpriority/important-longterm

Description

Minikube tunnel is not working on Windows because of minikube can not recognize the correct answer (or status code) of the route command.

Steps to reproduce the issue:

  1. minikube start --driver vmware
  2. minikube tunnel --cleanup --alsologtostderr

Full output of failed command:

PS C:\Windows\system32> minikube tunnel --cleanup --alsologtostderr
I0614 02:41:08.169600    3528 out.go:291] Setting OutFile to fd 84 ...
I0614 02:41:08.178573    3528 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0614 02:41:08.178573    3528 out.go:304] Setting ErrFile to fd 88...
I0614 02:41:08.178573    3528 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0614 02:41:08.185616    3528 root.go:291] Error reading config file at C:\Users\User\.minikube\config\config.json: open C:\Users\User\.minikube\config\config.json: The system cannot find the file specified.
I0614 02:41:08.186129    3528 mustload.go:65] Loading cluster: minikube
I0614 02:41:08.191755    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.191755    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.204499    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60699
I0614 02:41:08.205496    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.206051    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.206051    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.206564    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.206564    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:08.207075    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:08.417717    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:08.417717    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:08.426165    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:08.431377    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.431377    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.444381    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60702
I0614 02:41:08.444381    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.444891    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.444891    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.445405    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.445405    3528 main.go:128] libmachine: (minikube) Calling .DriverName
I0614 02:41:08.445917    3528 api_server.go:148] Checking apiserver status ...
I0614 02:41:08.452561    3528 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0614 02:41:08.453071    3528 main.go:128] libmachine: (minikube) Calling .GetSSHHostname
I0614 02:41:08.454096    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:08.663265    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:08.663265    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:08.671568    3528 main.go:128] libmachine: (minikube) DBG | MAC address in VMX: 00:0c:29:ca:93:41
I0614 02:41:08.671568    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in configuration file: C:\ProgramData\VMware\vmnetdhcp.conf
I0614 02:41:08.672079    3528 main.go:128] libmachine: (minikube) DBG | Following IPs found map[00:50:56:c0:00:01:192.168.106.1 00:50:56:c0:00:08:192.168.198.1]
I0614 02:41:08.672591    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in leases file: C:\ProgramData\VMware\vmnetdhcp.leases
I0614 02:41:08.673102    3528 main.go:128] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.198.128
I0614 02:41:08.673102    3528 main.go:128] libmachine: (minikube) Calling .GetSSHPort
I0614 02:41:08.673615    3528 main.go:128] libmachine: (minikube) Calling .GetSSHKeyPath
I0614 02:41:08.673615    3528 main.go:128] libmachine: (minikube) Calling .GetSSHUsername
I0614 02:41:08.673615    3528 sshutil.go:53] new ssh client: &{IP:192.168.198.128 Port:22 SSHKeyPath:C:\Users\User\.minikube\machines\minikube\id_rsa Username:docker}
I0614 02:41:08.768157    3528 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/6269/cgroup
I0614 02:41:08.771472    3528 api_server.go:164] apiserver freezer: "6:freezer:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b6effc84e76606921f1b221ad131c.slice/docker-5149535924cb43a882b3a4bcd18074fb7844c7e0f76c4234fbb9e9f0dd958119.scope"
I0614 02:41:08.778203    3528 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b6effc84e76606921f1b221ad131c.slice/docker-5149535924cb43a882b3a4bcd18074fb7844c7e0f76c4234fbb9e9f0dd958119.scope/freezer.state
I0614 02:41:08.781719    3528 api_server.go:186] freezer state: "THAWED"
I0614 02:41:08.781719    3528 api_server.go:223] Checking apiserver healthz at https://192.168.198.128:8443/healthz ...
I0614 02:41:08.786036    3528 api_server.go:249] https://192.168.198.128:8443/healthz returned 200:
ok
I0614 02:41:08.786551    3528 tunnel.go:57] Checking for tunnels to cleanup...
I0614 02:41:08.794762    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:08.800968    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.800968    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.813239    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60706
I0614 02:41:08.813749    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.813749    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.813749    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.814742    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.814742    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:08.815302    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:09.021094    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:09.021094    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:09.029606    3528 main.go:128] libmachine: (minikube) Calling .GetIP
I0614 02:41:09.030118    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:09.237544    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:09.238580    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:09.246815    3528 main.go:128] libmachine: (minikube) DBG | MAC address in VMX: 00:0c:29:ca:93:41
I0614 02:41:09.246815    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in configuration file: C:\ProgramData\VMware\vmnetdhcp.conf
I0614 02:41:09.247328    3528 main.go:128] libmachine: (minikube) DBG | Following IPs found map[00:50:56:c0:00:01:192.168.106.1 00:50:56:c0:00:08:192.168.198.1]
I0614 02:41:09.247328    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in leases file: C:\ProgramData\VMware\vmnetdhcp.leases
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.198.128
I0614 02:41:09.248352    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.248352    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.248868    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.248868    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.249384    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.249384    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.249384    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.249897    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.249897    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.250411    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.250411    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.250411    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.250942    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.251455    3528 tunnel_manager.go:71] Setting up tunnel...
I0614 02:41:09.251455    3528 tunnel_manager.go:81] Started minikube tunnel.
I0614 02:41:14.251829    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:14.256747    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:14.257256    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:14.270187    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60719
I0614 02:41:14.270700    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:14.270700    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:14.270700    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:14.271210    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:14.271210    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:14.272235    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:14.479222    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:14.479222    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:14.602249    3528 route_windows.go:47] Adding route for CIDR 10.96.0.0/12 to gateway 192.168.198.128
I0614 02:41:14.602249    3528 route_windows.go:49] About to run command: [route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128]
Status:
        machine: minikube
        pid: 3528
        route: 10.96.0.0/12 -> 192.168.198.128
        minikube: Running
        services: []
    errors:
                minikube: no errors
                router: error adding route:  ��
, 2
                loadbalancer emulator: no errors
I0614 02:41:14.618056    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:14.618056    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:14.618569    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:14.618569    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:14.619082    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:19.619835    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:19.625308    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:19.625371    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:19.638659    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60739
I0614 02:41:19.638659    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:19.639172    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:19.639172    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:19.639683    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:19.639683    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:19.640708    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:19.847709    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:19.847709    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:19.976476    3528 route_windows.go:47] Adding route for CIDR 10.96.0.0/12 to gateway 192.168.198.128
I0614 02:41:19.976476    3528 route_windows.go:49] About to run command: [route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128]
Status:
        machine: minikube
        pid: 3528
        route: 10.96.0.0/12 -> 192.168.198.128
        minikube: Running
        services: []
    errors:
                minikube: no errors
                router: error adding route: ���� ���������� ��������: ���� ��ꥪ� 㦥 ����������.

, 3
                loadbalancer emulator: no errors
I0614 02:41:19.992076    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:19.992076    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:19.992588    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:19.992588    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:19.993098    3528 main.go:128] libmachine: Making call to close connection to plugin binary

The first time the route is successfully created, command route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128 returns OK (on russian). All next tries to add a route are ended with error The route exists (on russian too). So, in spite of route is created minikube thinks that is not and endlessly tries to create it again and again.

Contributor guide

Tech stack
go
Domain
devops
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
GoWindows networkingminikube tunnel code
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the route windows.go file, specifically the route addition logic. The current parsing of the route command output appears to fail on Windows. Look at how the command output is checked (e.g., the return code or string matching). Consider using a proper Windows API or improving the condition that checks for success. Also verify if there are any existing tests for this functionality.