Network policies break daskhub

まず、ネットワークポリシーを有効にした Kubernetes 上で、ドキュメントに記載された helm upgrade --install によるデプロイメントを再現し、その後、DaskHub および JupyterHub chart からレンダリングされた NetworkPolicy リソースを調査します。報告された Hubble drops を使用して、hub、single-user、dask-gateway、kube-apiserver 間に必要な通信を検証します。デフォルトのデプロイメントで、allow-all ポリシーなしに pod を起動し、dask-gateway と通信できれば完了です。

Describe the issue:
It appears the default network policies from the jupyterhub helm chart breaks communication with dask-gateway and the kube-apiserver.

I deployed daskhub with default values onto a vanilla K8s cluster with a CNI that supports network policies (cilium).
helm upgrade --install --create-namespace --namespace jhub01 jhub01 dask/daskhub

With this deployment, the jupyterhub pod will not spawn and I receive this output:

Using hubble, I am able to see the packets are being dropped via network policy:
hubble observe -n jhub01 -t drop -f

Feb 16 18:05:30.839: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:31.862: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:33.910: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:35.043: jhub01/hub-fc455bdb8-2n7ct:38918 (ID:133898) <> XXX.XXX.XXX.148:6443 (kube-apiserver) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:36.086: jhub01/hub-fc455bdb8-2n7ct:38918 (ID:133898) <> XXX.XXX.XXX.148:6443 (kube-apiserver) Policy denied DROPPED (TCP Flags: SYN)

If I allow access to the kube-apiserver (reference ticket below), the pod will then spawn but I still get drops for dask-gateway communication:

Feb 16 19:40:58.002: jhub01/jupyter-test:34604 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:00.498: jhub01/jupyter-test:53158 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:06.130: jhub01/jupyter-test:34604 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:07.936: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:08.950: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:10.998: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:14.914: jhub01/jupyter-test:43114 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:15.030: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)

Here is the list of network policies defined for the whole cluster:
kubectl get networkpolicies.networking.k8s.io -A

NAMESPACE NAME POD-SELECTOR AGE
jhub01 hub app=jupyterhub,component=hub,release=jhub01 22h
jhub01 proxy app=jupyterhub,component=proxy,release=jhub01 22h
jhub01 singleuser app=jupyterhub,component=singleuser-server,release=jhub01 22h

Everything works when I deploy this network policy into the namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress-egress
spec:
  podSelector: {}
  egress:
  - {}
  ingress:
  - {}
  policyTypes:
  - Egress
  - Ingress

Anything else we need to know?:
Bare Metal - K8s Server Version: v1.29.1
CRI-O Version: v1.29.1
Cilium Version: v1.15.1

This issue is related and is why we are seeing drops for the kube-apiserver:

• https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/3202

If I allow access to the kube-apiserver I then hit this issue:

• https://github.com/dask/helm-chart/issues/431

I think that the daskhub chart should deploy network policies to allow the jupyterhub pod to communicate with dask-gateway. Or perhaps something about the correct network policies should be documented since the default values don't allow dask-gateway communication?

Environment:

• Dask version: daskhub-2024.1.1
• Python version:
• Operating System: AlmaLinux 9
• Install method (conda, pip, source): helm