Private AKS cluster, enabled outbound, metrics pod keeps failing

まず、レポートにリンクされている AKS managed Prometheus の有効化およびトラブルシューティング ガイドを確認し、次に ama-metrics pod のログと UAMI、DCE、DCR の構成を調査します。メトリクス拡張機能の失敗原因となっている構成または接続性の問題を特定し、再現可能な修正または実行可能な診断を文書化できれば完了です。

Hello,
I have private AKS cluster with open outbound connection and UAMI. I have tried to enable managed prometheus for metrics collections but it seems to fail. Can someone help? I followed this guide https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli

││ prometheus-collector Error: configmap section not mounted, using defaults 
││ addon-token-adapter 2025/02/09 00:21:37 handlers.go:57: received token request, handling...                                                                                                                                                    ││ addon-token-adapter 2025/02/09 00:23:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:24:51 handlers.go:57: received token request, handling...                                                                                                                                                    ││ addon-token-adapter 2025/02/09 00:25:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:27:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:29:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ stream closed EOF for kube-system/ama-metrics-5bff7d784d-mccqf (prometheus-collector)                             
││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 0 minutes                                                                                                                                                                                  ││ prometheus-collector 2025-02-09T00:33:05 No configuration present for the AKS resource                                                                                                                                                         ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 0 minutes                                                                                                                                                                                  ││ prometheus-collector 2025-02-09T00:33:20 No configuration present for the AKS resource                                                                                                                                                         ││ addon-token-adapter 2025/02/09 00:33:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 1 minutes                                                                                                                                                                                  ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 1 minutes    
││ prometheus-collector Container has been running for 1 minutes                                                                                                                                                                                  ││ prometheus-collector {"time":1739061266.590147,"filepath":"/opt/microsoft/linuxmonagent/mdsd.err","log":"2025-02-09T00:34:26.5900480Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:318,GetAgentCon ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)   

I tried to look at https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-troubleshoot to find solutions here but it did not help. for example

• Checked if UAMI has metric publish role
• DCE and DCR are created, not sure how they work though