SSL verification option is not applied

Open
#38 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
backend

Research direction

Start by tracing how the bridge's verify_ssl configuration reaches the aiohttp request setup used for homeserver connectivity, using the traceback's mautrix/api.py request path as an entry point. Reproduce the Kubernetes-style connection with verify_ssl set to false and confirm that the resulting request no longer performs certificate verification.

Written by the indexing model from the issue text.

Description

bug

I'm running Matrix in Kubernetes with in-cluster TLS encryption enabled, and the bridge is deployed there next to Matrix. When the bridge is configured with the external address of the homeserver (e.g., https://matrix.example.com), the connection is fine as the certificate is valid. However, when I try to use the in-cluster Matrix service (https://matrix:8448), and so turn off the verify_ssl option of the bridge, the bridge is still trying to check the cert's validity as if the verification option wasn't there at all. I also tried adding the service name as another SAN to the cert to circumvent the verification issue but it doesn't help. I'd like to avoid using the external Matrix URL as it means an unnecessary roundtrip for all the packets and depending on external services, so it would be great if the verify_ssl option could be fixed to apply when false.

Bridge startup error log:

 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.                                                                                                                                                                            
 INFO  [alembic.runtime.migration] Will assume transactional DDL.                                                                                                                                                                          
 [2020-12-16 06:47:56,864] [INFO@mau.init[] Initializing mautrix-facebook 0.1.2                                                                                                                                                            
 [2020-12-16 06:47:56,986] [INFO@mau.init[] Initialization complete in 2.1 seconds                                                                                                                                                         
 [2020-12-16 06:47:56,986] [DEBUG@mau.init[] Running startup actions...                                                                                                                                                                    
 [2020-12-16 06:47:56,987] [DEBUG@mau.init[] Starting appservice...                                                                                                                                                                        
 [2020-12-16 06:47:56,987] [DEBUG@mau.as[] Starting appservice web server on 0.0.0.0:29319                                                                                                                                                 
 [2020-12-16 06:47:56,987] [INFO@mau.mx[] Ensuring connectivity to homeserver                                                                                                                                                              
 [2020-12-16 06:47:57,188] [CRITICAL@mau.init[] Unexpected error in main event loop                                                                                                                                                        
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection                                                                                                                                      
     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa                                                                                                                                                    
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection                                                                                                                                                       
     transport, protocol = await self._create_connection_transport(                                                                                                                                                                        
   File "/usr/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport                                                                                                                                            
     await waiter                                                                                                                                                                                                                          
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 529, in data_received                                                                                                                                                               
     ssldata, appdata = self._sslpipe.feed_ssldata(data)                                                                                                                                                                                   
   File "/usr/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata                                                                                                                                                                
     self._sslobj.do_handshake()                                                                                                                                                                                                           
   File "/usr/lib/python3.8/ssl.py", line 944, in do_handshake                                                                                                                                                                             
     self._sslobj.do_handshake()                                                                                                                                                                                                           
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)                                                                                            
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 233, in request                                                                                                                                                            
     return await self._send(method, full_url, content, query_params, headers or {})                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 149, in _send                                                                                                                                                              
     async with request as response:                                                                                                                                                                                                       
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__                                                                                                                                                     
     self._resp = await self._coro                                                                                                                                                                                                         
   File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request                                                                                                                                                        
     conn = await self._connector.connect(                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect                                                                                                                                                      
     proto = await self._create_connection(req, traces, timeout)                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection                                                                                                                                           
     _, proto = await self._create_direct_connection(                                                                                                                                                                                      
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection                                                                                                                                   
     raise last_exc                                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection                                                                                                                                    
     transp, proto = await self._wrap_create_connection(                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 938, in _wrap_create_connection                                                                                                                                      
     raise ClientConnectorCertificateError(                                                                                                                                                                                                
 aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 
  (_ssl.c:1123)')]                                                                                                                                                                                                                         
 The above exception was the direct cause of the following exception:                                                                                                                                                                      
 Traceback (most recent call last):                                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/util/program.py", line 190, in _run                                                                                                                                                      
     self.loop.run_until_complete(self.start())                                                                                                                                                                                            
   File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete                                                                                                                                                       
     return future.result()                                                                                                                                                                                                                
   File "/usr/lib/python3.8/site-packages/mautrix_facebook/__main__.py", line 81, in start                                                                                                                                                 
     await super().start()                                                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/bridge.py", line 149, in start                                                                                                                                                    
     await self.matrix.init_encryption()                                                                                                                                                                                                   
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/matrix.py", line 126, in init_encryption                                                                                                                                          
     if not await self.e2ee.check_server_support():                                                                                                                                                                                        
   File "/usr/lib/python3.8/site-packages/mautrix/bridge/e2ee.py", line 174, in check_server_support                                                                                                                                       
     flows = await self.client.get_login_flows()                                                                                                                                                                                           
   File "/usr/lib/python3.8/site-packages/mautrix/client/api/authentication.py", line 36, in get_login_flows                                                                                                                               
     resp = await self.api.request(Method.GET, Path.login)                                                                                                                                                                                 
   File "/usr/lib/python3.8/site-packages/mautrix/api.py", line 235, in request                                                                                                                                                            
     raise MatrixConnectionError(str(e)) from e                                                                                                                                                                                            
 mautrix.errors.base.MatrixConnectionError: Cannot connect to host matrix:8448 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)') 
 ]                                                                                                                                                                                                                                         

Bridge config:

homeserver:
    address: https://matrix:8448
    verify_ssl: false
    ...

Cert check run from within the Matrix pod:

$ openssl s_client -connect matrix:8448 </dev/null 2>/dev/null | openssl x509 -noout -text | grep DNS:
DNS:matrix.example.com, DNS:matrix
Dominant language
Python
Stars
249
Forks
84
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mautrix/python

All issues in mautrix/python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.