aiohttp TLS websocket fails for continously sending packages on ESP32

從提供的 ESP32 重現開始，追蹤 aiohttp_ws.py 中的 websocket 操作，尤其是 receive、send_str、close 和 _read_frame，然後檢查 asyncio/stream.py 中的錯誤。比較遠端和本機的故障，包括 TLS 關閉行為和記憶體增長。只有在找出可重現的原因，並為連續的 2–4 kB TLS websocket 訊息定義經過驗證的修正方案後，才算完成。

@Carglglz
I have an application i want to send TLS / SSL packages over websockets using aiohttp on an ESP32. The problem is that the websockets fail after a short while when using packages that have a little bigger size around 2kB to 4kB.

Here is a simple test script:

import aiohttp
import asyncio
import gc

# allocate and prepare fixed block of data to be sent
b = bytearray(10_000)
for k in range(100):
    p = k*100
    b[p:p] = b'X'*96 + f'{k:3}' + '\n'
mv = memoryview(b)

URL = "wss://somewebsocketserver/echo"
sslctx = False
if URL.startswith("wss:"):
    try:
        import ssl
        
        sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
        sslctx.verify_mode = ssl.CERT_NONE
    except Exception:
        pass

async def ws_receive(ws : ClientWebSocketResponse):
    try:
        async for msg in ws:
            if msg.type != aiohttp.WSMsgType.TEXT:
                print('type:', msg.type, repr(msg.data))
    except TypeError as e:
        print('ws_receive', repr(e))
            

async def ws_test_echo(session):
    n = 1
    
    while True:
        ws_receive_task = None
        async with session.ws_connect(URL, ssl=sslctx) as ws:
            ws_receive_task = asyncio.create_task(ws_receive(ws))
            try:
                while True:
                    gc.collect()
                    print('-------------------', n, '------------------------')
                    await ws.send_str(b[0:100*n].decode())
                    n += 1

                    await asyncio.sleep_ms(100)

            except KeyboardInterrupt:
                pass

            finally:
                await ws.close()


async def main():
    async with aiohttp.ClientSession() as session:
        print('session')
        await ws_test_echo(session)


if __name__ == "__main__":
    asyncio.run(main())

remote

I'll get the following exception after a while. Just dealing with the exception is not satisfying since it takes a while and the applicaiton is blocked in that time.

Task exception wasn't retrieved
future: <Task> coro= <generator object 'ws_receive' at 3ffec850>
Traceback (most recent call last):
  File "asyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 29, in ws_receive
  File "/lib/aiohttp/aiohttp_ws.py", line 226, in __anext__
  File "/lib/aiohttp/aiohttp_ws.py", line 171, in receive
  File "/lib/aiohttp/aiohttp_ws.py", line 198, in _read_frame
  File "asyncio/stream.py", line 1, in read
OSError: -113
Traceback (most recent call last):
  File "<stdin>", line 66, in <module>
  File "asyncio/core.py", line 1, in run
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 62, in main
  File "<stdin>", line 56, in ws_test_echo
  File "/lib/aiohttp/aiohttp_ws.py", line 233, in close
  File "/lib/aiohttp/aiohttp_ws.py", line 194, in close
  File "/lib/aiohttp/aiohttp_ws.py", line 189, in send
  File "asyncio/stream.py", line 1, in drain
OSError: -113

(OSError: 113 = ECONNABORTED)

Noteworthy is that there are always some packages that micropyhon thinks are sent already but don't reach the other side.
I also do not receive a websocket close package.
This was tested on a remote Websocket server that i don't control.

local

When i try it on a local server, i see different problems.

Traceback (most recent call last):
  File "<stdin>", line 64, in <module>
  File "asyncio/core.py", line 1, in run
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 60, in main
  File "<stdin>", line 45, in ws_test_echo
  File "/lib/aiohttp/aiohttp_ws.py", line 239, in send_str
  File "/lib/aiohttp/aiohttp_ws.py", line 187, in send
  File "asyncio/stream.py", line 1, in write
OSError: -104

(OSError: 104 = ECONNRESET)

The servers seems to not receive the complete message / only a corrupted message and closes the connection:

...
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 37
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 38
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 39
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
b'\xe9\x82\xc3+\xe9\x82\xbbG\x81\xd0\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3+\xe9\x82\xc3'

Unhandled exception in client_connected_cb
transport: <asyncio.sslproto._SSLProtocolTransport object at 0x7f18b780ecf0>
Traceback (most recent call last):
  File "/home/username/Documents/projects/rfid/websockets_python/microdot/examples/tls/microdot/microdot.py", line 1224, in serve
    await self.handle_request(reader, writer)
  File "/home/username/Documents/projects/rfid/websockets_python/microdot/examples/tls/microdot/microdot.py", line 1338, in handle_request
    await writer.aclose()
  File "/home/username/Documents/projects/rfid/websockets_python/microdot/examples/tls/microdot/microdot.py", line 1218, in aclose
    await self.wait_closed()
  File "/usr/lib/python3.11/asyncio/streams.py", line 364, in wait_closed
    await self._protocol._get_close_waiter(self)
  File "/usr/lib/python3.11/asyncio/sslproto.py", line 648, in _do_shutdown
    self._sslobj.unwrap()
  File "/usr/lib/python3.11/ssl.py", line 983, in unwrap
    return self._sslobj.shutdown()
           ^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2706)

The closing of the websocket is also not handled properly by the micropython application / aiohttp. The close package is received but the connection is not closed automatically.

Another problem is that even if i deal with the exceptions and reconnect automatically, my application will eventually crash because of a run out of memory.
This is really problematic to me since i only send messages of size 2k or so.
I guess this is because of the not so memory economical design of aiohttp. For sending and receiving there is always a lot of new memory allocation involved.
The use of preallocation and memoryview seems reasonable here.

This is the local python websocketserver code:

import ssl
import sys
from microdot import Microdot
from microdot.websocket import with_websocket

app = Microdot()

html = '''<!DOCTYPE html>
<html>
    <head>
        <title>Microdot Example Page</title>
        <meta charset="UTF-8">
    </head>
    <body>
        <div>
            <h1>Microdot Example Page</h1>
            <p>Hello from Microdot!</p>
            <p><a href="/shutdown">Click to shutdown the server</a></p>
        </div>
    </body>
</html>
'''


@app.route('/')
async def hello(request):
    return html, 200, {'Content-Type': 'text/html'}

@app.route('/echo')
@with_websocket
async def echo(request, ws):
    while True:
        data = await ws.receive()
        print(data)
        await ws.send(data)

@app.route('/shutdown')
async def shutdown(request):
    request.app.shutdown()
    return 'The server is shutting down...'


ext = 'der' if sys.implementation.name == 'micropython' else 'pem'
folder_name = "/home/username/Documents/projects/rfid/websockets_python/"
ssl_cert = "cert.pem"
ssl_key = "key.pem"

sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
sslctx.load_cert_chain(ssl_cert, ssl_key)
app.run(port=4443, debug=True, ssl=sslctx)

MicroPython v1.23.0-preview.344.gb1ac266bb.dirty on 2024-04-29; Generic ESP32 module with ESP32