Observability configuration for prometheus is not explained at all anywhere

All the observability for envoy is somehow skewed into otel connectors while neglecting how prometheus should be configured. More than that, it looks like the "addon" chart does not solve the problem too, i.e. after installing prometheus there are no metrics from Envoy anywhere to be found. It looks like the following needs to be created:

1. PodMonitor for envoy proxy metrics: https://github.com/envoyproxy/gateway/blob/main/examples/kubernetes/metric/pod-monitor.yaml
2. CustomResourceStateMetrics for gateway controller metrics (although these don't look useful much...)

None of that is explained anywhere...