Cannot make traffic from host on Mac to guest IP with vpnkit

まず2つの hyperkit 構成を再現し、vpnkit のコマンドラインに表示されるゲートウェイアドレスとホストアドレスを含めて、virtio-net と virtio-vpnkit の経路を比較します。ポート 8080 のサービスについて、vpnkit の診断情報と転送動作を調査します。完了条件は、到達可能なアドレス、またはホストからの受信トラフィックに対する文書化された転送構成を特定することです。

I could not find any e-mailing list or another forum I could post this question to so I am creating an issue which I will close eventually.

So I am trying to make inbound connect to the OSv unikernel guest running on hyperkit over vpnkit and it does not seem to work or I simply do not know which host IP address to use.

I am using Docker Desktop community version 2.0.0.4. By default I have 2 processes related to Docker/hyperkit/vpnkit running like so:

ps -ef | grep hyperkit
  501  3258  3256   0 Thu09AM ??         0:25.76 com.docker.vpnkit --ethernet fd:3 --port vpnkit.port.sock --port hyperkit://:62373/./vms/0 --diagnostics fd:4 --pcap fd:5 --vsock-path vms/0/connect --host-names host.docker.internal,docker.for.mac.host.internal,docker.for.mac.localhost --gateway-names gateway.docker.internal,docker.for.mac.gateway.internal,docker.for.mac.http.internal --vm-names docker-for-desktop --listen-backlog 32 --mtu 1500 --allowed-bind-addresses 0.0.0.0 --http /Users/****/Library/Group Containers/group.com.docker/http_proxy.json --dhcp /Users/****/Library/Group Containers/group.com.docker/dhcp.json --port-max-idle-time 300 --max-connections 2000 --gateway-ip 192.168.65.1 --host-ip 192.168.65.2 --lowest-ip 192.168.65.3 --highest-ip 192.168.65.254 --log-destination asl --udpv4-forwards 123:127.0.0.1:53859 --gc-compact-interval 1800
  501  3265  3260   0 Thu09AM ??        22:45.92 com.docker.hyperkit -A -u -F vms/0/hyperkit.pid -c 4 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=vpnkit.eth.sock,uuid=fb0a6ebd-8e87-42c2-ae06-83c1027f0c55 -U 8354e93d-6601-4091-923f-0428d28d1c0c -s 2:0,ahci-hd,/Users/****/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.raw -s 3,virtio-sock,guest_cid=3,path=vms/0,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,vms/0/config.iso -s 6,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker.iso -s 7,virtio-rnd -l com1,autopty=vms/0/tty,asl -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,

When I start OSv on hyperkit executing to virtio (instead of vpnkit), the OSv starts and grabs 192.168.64.16 (using dhcp) like so:

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-net
OSv v0.53.0-61-g513e776b
eth0: 192.168.64.16
....

and I can make traffic over port 8080 from the host, more specifically the guest responds on HTTP at http://192.168.64.16:8080/

The extra hyperkit process looks like this:

0 19871 19869   0  6:08PM ttys007    0:39.01 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/****/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/****/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-net

Now when I use vpnkit instead I can see guest grabbing an IP but I cannot connect to it in any way and it does not work.

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/wkozaczuk/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-vpnkit,path=/Users/****/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
  -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState
OSv v0.53.0-61-g513e776b
eth0: 192.168.65.24
...

And I see this extra process:

501 19935 19933   0  6:14PM ttys007    0:30.10 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-vpnkit,path=/Users/***/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState

I have tried obvious 192.168.65.24 as the guest reports grabbing, I have tried host IP 192.168.65.2 and couple of other ones? I event have tried to add ,guest_forwards=8080 to vsock path and also does not work with any IPs.

What am I doing wrong? Please advise.

Waldek