[rbac] Envoy doesn't break an existing connection when rbac rules no longer allow traffic

Title: Break existing connections when new RBAC rules stop allowing them

This was initially reported as a security vulnerability. Eventually, it was classified as a feature request by @mattklein123, so opening an issue for it.

Problem Statement

Suppose that client is connecting to server. Initially, Istio RBAC allows the connection to happen. After some time, Istio RBAC is changed to not allow traffic from the client to the server. However, Envoy doesn't close the TCP connection and the client can still send data.

Reproducer

0. Prepare a K8s cluster with Istio (e.g., MiniKF), disable EnableAutoMtls in the Istio ConfigMap (not sure why this is needed but it is) and enable Istio RBAC (on by default in MiniKF).

1. Apply the attached YAML, which creates a sidecar-enabled namespace with a client and server deployment, as well as necessary Istio RBAC: istio-reproducer-rbac.yaml (2).log istio-reproducer.yaml (2).log

2. Exec into the server container and start a TCP server on the declared port 8081:

   # server-container
   nc -v -l -p 8081
   

3. Exec into the client container and try to connect to the server:

   # Is not denied by Istio RBAC
   nc -v server.test-istio-rbac.svc.cluster.local 8081
   

4. Send messages from client to server. They are allowed by Istio RBAC.

5. Delete the Istio RBAC that permits the connection: istio-reproducer-rbac.yaml (2).log

6. Continue sending messages from the client and see that they are received.

7. Drop client connection, try to connect again. After connecting, send some data. Istio RBAC closes the connection because it's now enforced (on new connections).

Note: You can enable Istio RBAC debug log in the server's Istio sidecar, in order to see the enforcement messages:

kubectl exec -it -n test-istio-rbac <server_pod> -c istio-proxy -- pilot-agent request POST 'logging?rbac=debug'

Proposed Solution

When RBAC rules are updated, Envoy should drop any connections no longer allowed by the new policy.