SSL verification option is not applied

まず、homeserver への接続に使用される aiohttp のリクエスト設定に bridge の verify_ssl 設定がどのように到達するかを、traceback の mautrix/api.py のリクエストパスを入口として追跡します。verify_ssl を false に設定した Kubernetes 形式の接続を再現し、その結果のリクエストで証明書検証が行われなくなることを確認します。

I'm running Matrix in Kubernetes with in-cluster TLS encryption enabled, and the bridge is deployed there next to Matrix. When the bridge is configured with the external address of the homeserver (e.g., https://matrix.example.com), the connection is fine as the certificate is valid. However, when I try to use the in-cluster Matrix service (https://matrix:8448), and so turn off the verify_ssl option of the bridge, the bridge is still trying to check the cert's validity as if the verification option wasn't there at all. I also tried adding the service name as another SAN to the cert to circumvent the verification issue but it doesn't help. I'd like to avoid using the external Matrix URL as it means an unnecessary roundtrip for all the packets and depending on external services, so it would be great if the verify_ssl option could be fixed to apply when false.

Bridge startup error log:

 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.                                                                                                                                                                            
 INFO  [alembic.runtime.migration] Will assume transactional DDL.                                                                                                                                                                          
 [2020-12-16 06:47:56,864] [INFO@mau.init[] Initializing mautrix-facebook 0.1.2                                                                                                                                                            
 [2020-12-16 06:47:56,986] [INFO@mau.init[] Initialization complete in 2.1 seconds                                                                                                                                                         
 [2020-12-16 06:47:56,986] [DEBUG@mau.init[] Running startup actions...                                                                                                                                                                    
 [2020-12-16 06:47:56,987] [DEBUG@mau.init[] Starting appservice...                                                                                                                                                                        
 [2020-12-16 06:47:56,987] [DEBUG@mau.as[] Starting appservice web server on 0.0.0.0:29319                                                                                                                                                 
 [2020-12-16 06:47:56,987] [INFO@mau.mx[] Ensuring connectivity to homeserver                                                                                                                                                              
 [2020-12-16 06:47:57,188] [CRITICAL@mau.init[] Unexpected error in main event loop                                                                                                                                                        
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection                                                                                                                                      
     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa                                                                                                                                                    
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection                                                                                                                                                       
     transport, protocol = await self._create_connection_transport(                                                                                                                                                                        
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport                                                                                                                                            
     await waiter                                                                                                                                                                                                                          
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 529, in data_received                                                                                                                                                               
     ssldata, appdata = self._sslpipe.feed_ssldata(data)                                                                                                                                                                                   
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata                                                                                                                                                                
     self._sslobj.do_handshake()                                                                                                                                                                                                           
   File "/usr/lib/python3.8/ssl.py", line 944, in do_handshake                                                                                                                                                                             
     self._sslobj.do_handshake()                                                                                                                                                                                                           
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)                                                                                            
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 233, in request                                                                                                                                                            
     return await self._send(method, full_url, content, query_params, headers or {})                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 149, in _send                                                                                                                                                              
     async with request as response:                                                                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__                                                                                                                                                     
     self._resp = await self._coro                                                                                                                                                                                                         
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request                                                                                                                                                        
     conn = await self._connector.connect(                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect                                                                                                                                                      
     proto = await self._create_connection(req, traces, timeout)                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection                                                                                                                                           
     _, proto = await self._create_direct_connection(                                                                                                                                                                                      
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection                                                                                                                                   
     raise last_exc                                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection                                                                                                                                    
     transp, proto = await self._wrap_create_connection(                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 938, in _wrap_create_connection                                                                                                                                      
     raise ClientConnectorCertificateError(                                                                                                                                                                                                
 aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 
  (_ssl.c:1123)')]                                                                                                                                                                                                                         
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/util/program.py", line 190, in _run                                                                                                                                                      
     self.loop.run_until_complete(self.start())                                                                                                                                                                                            
   File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete                                                                                                                                                       
     return future.result()                                                                                                                                                                                                                
   File "/usr/lib/python3.8/site-packages/mautrix_facebook/__main__.py", line 81, in start                                                                                                                                                 
     await super().start()                                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/bridge.py", line 149, in start                                                                                                                                                    
     await self.matrix.init_encryption()                                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/matrix.py", line 126, in init_encryption                                                                                                                                          
     if not await self.e2ee.check_server_support():                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/e2ee.py", line 174, in check_server_support                                                                                                                                       
     flows = await self.client.get_login_flows()                                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/mautrix/client/api/authentication.py", line 36, in get_login_flows                                                                                                                               
     resp = await self.api.request(Method.GET, Path.login)                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 235, in request                                                                                                                                                            
     raise MatrixConnectionError(str(e)) from e                                                                                                                                                                                            
 mautrix.errors.base.MatrixConnectionError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)') 
 ]                                                                                                                                                                                                                                         

Bridge config:

homeserver:
    address: https://matrix:8448
    verify_ssl: false
    ...

Cert check run from within the Matrix pod:

$ openssl s_client -connect matrix:8448 </dev/null 2>/dev/null | openssl x509 -noout -text | grep DNS:
DNS:matrix.example.com, DNS:matrix