Thin mode asyncio: segfault in data_received after a late FIN from a redirecting listener
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 15/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- python
- Ambito
- database, networking
Direzione di ricerca
Start in src/oracledb/impl/thin/protocol.pyx: read _connect_tcp, where a redirect opens a second transport on the same protocol, then connection_lost and data_received, where self._transport can be None. Check _post_connect in connection.pyx, which clears _in_connect. Done means a late FIN after a redirect no longer nulls the live transport, data_received does not dereference None, and the reproduction script survives with FIN_DELAY = 1.0.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
This looks like the same root cause as #616, but it ends in a crash instead of an error, and the fix in #617 would not stop it. That's why I'm opening it separately. Happy to move it to #616 if you prefer.
1. What versions are you using?
- python-oracledb 26.1.0 and 3.4.2, both crash with the script below. 3.4.1 crashed the same way when I tried it with a fake redirecting listener.
- Oracle AI Database 26ai Free 23.26.3.0.0 (
gvenzl/oracle-free:23-slim-faststart) - Python 3.14.10 on Linux aarch64, in Docker. Same crash on Linux x86_64, where I used 26.1.0 and 3.4.1 with the fake listener and 3.4.2 with the real one.
2. Is it an error or a hang or a crash?
A crash. SIGSEGV, exit code 139, the whole process goes down.
3. What error(s) or behavior you are seeing?
We noticed this in a long-running asyncio service using Thin mode pools. It died with exit 139 a few times a day, always at the same place in thin_impl. Our listeners appear to redirect too. I have no core dump from there, so I can't prove a late FIN is what happened in that service, but the script below crashes at the same Python frame and, on 3.4.2 x86_64, at the same offset in thin_impl.
As far as I can tell this is what happens:
- On a redirect,
_connect_tcpcallscreate_connection(lambda: self, ...)again and the first transport stays open, so two transports share one protocol. connection_lost()doesn't know which transport was lost. It only ignores the loss while_in_connectis set, and_post_connectclears that flag.- If the listener's FIN arrives after that point,
self._transportis set toNone, even though the second transport is healthy and still reading. - The running call fails with DPY-4011. When the server's reply then arrives on the live socket,
data_received()callsself._transport.extract_packet(data)onNoneand the process segfaults. On 3.4.2 x86_64 I mapped the crash address to exactly that call.
Normally the listener's FIN arrives a few milliseconds after the redirect and wins the race, so you need a FIN that shows up late, for example from a stalled listener or a retransmission. In #616 the listener closes about 0.3 s later on a 19c RAC, so this doesn't seem that exotic.
Output from 26.1.0 (trimmed, stdout was lost in the crash):
Fatal Python error: Segmentation fault
Current thread 0x0000eb8a59ee0020 [python] (most recent call first):
File "/usr/local/lib/python3.14/asyncio/selector_events.py", line 1025 in _read_ready__data_received
File "/usr/local/lib/python3.14/asyncio/selector_events.py", line 968 in _read_ready
File "/usr/local/lib/python3.14/asyncio/events.py", line 94 in _run
File "/usr/local/lib/python3.14/asyncio/base_events.py", line 2064 in _run_once
...
Current thread's C stack trace (most recent call first):
Binary file "/usr/local/bin/../lib/libpython3.14.so.1.0", at _Py_DumpStack+0x58 [0xeb8a599c6218]
Binary file "/usr/local/bin/../lib/libpython3.14.so.1.0", at +0xdfcfc [0xeb8a599cfcfc]
Binary file "linux-vdso.so.1", at __kernel_rt_sigreturn+0x0 [0xeb8a59ee88f8]
Binary file "/usr/local/lib/python3.14/site-packages/oracledb/thin_impl.cpython-314-aarch64-linux-gnu.so", at +0x1f80dc [0xeb8a57ec80dc]
Binary file "/usr/local/bin/../lib/libpython3.14.so.1.0", at PyObject_Vectorcall+0x54 [0xeb8a59a06a64]
...
With FIN_DELAY = 0 the same script is fine:
oracledb 26.1.0 python 3.14.10
connected 23.26.3.0.0
waiting for the server's reply on the live socket
survived
Two more things I saw on 3.4.2 from the same cause:
- If no call is in flight when the late FIN arrives, nothing crashes, but the connection reports DPY-4011 although its socket is fine.
close()then does nothing because_transportisNone, so the socket stays ESTABLISHED after close. Pools replace these connections, so you only notice the leaked sockets. - If the FIN arrives while connect is waiting for a packet, the connect fails. That's #616.
4. Does your application call init_oracle_client()?
No, Thin mode.
5. Include a runnable Python script that shows the problem.
It needs a listener that really redirects, so I used shared server with direct handoff off. As SYSDBA:
alter system set shared_servers=1 scope=memory;
alter system set dispatchers='(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=15299))(DISPATCHERS=1)' scope=memory;
alter system register;
Then add this to listener.ora and run lsnrctl reload:
DIRECT_HANDOFF_TTC_LISTENER=OFF
The user is probe / probe in FREEPDB1. Run the script with the client in the database container's network namespace (docker run --network container:<name> ...), so it can reach the dispatcher on 127.0.0.1.
"""Thin mode, asyncio: late FIN from the redirecting listener, then SIGSEGV.
The script puts a small TCP proxy in front of the listener. Data passes through
untouched; the only change is that the listener's FIN reaches the client one
second late. Run it on the database host, because the client must be able to
reach the dispatcher address the listener redirects to.
"""
import asyncio
import faulthandler
import sys
import threading
import time
import oracledb
LISTENER = ("127.0.0.1", 1521)
PROXY_PORT = 15213
FIN_DELAY = 1.0
USER = "probe"
PASSWORD = "probe"
SERVICE = "FREEPDB1"
async def copy(reader, writer):
while data := await reader.read(65536):
writer.write(data)
await writer.drain()
async def handle(client_reader, client_writer):
server_reader, server_writer = await asyncio.open_connection(*LISTENER)
upstream = asyncio.create_task(copy(client_reader, server_writer))
await copy(server_reader, client_writer)
await asyncio.sleep(FIN_DELAY)
client_writer.close()
server_writer.close()
upstream.cancel()
async def proxy():
server = await asyncio.start_server(handle, "127.0.0.1", PROXY_PORT)
await server.serve_forever()
async def client():
conn = await oracledb.connect_async(
user=USER,
password=PASSWORD,
host="127.0.0.1",
port=PROXY_PORT,
service_name=SERVICE,
server_type="shared",
)
print("connected", conn.version)
cursor = conn.cursor()
try:
await cursor.execute("begin dbms_session.sleep(3); end;")
except oracledb.Error as error:
print("query raised:", error)
print("waiting for the server's reply on the live socket")
await asyncio.sleep(6)
print("survived")
if __name__ == "__main__":
faulthandler.enable()
print("oracledb", oracledb.__version__, "python", sys.version.split()[0])
threading.Thread(target=lambda: asyncio.run(proxy()), daemon=True).start()
time.sleep(1)
asyncio.run(client())
26.1.0 and 3.4.2 each crashed 2 out of 2 runs. The control with FIN_DELAY = 0 ran once on 26.1.0 and survived.
Possible fix
#617 won't help here, because its check only runs while _in_connect is set and this happens after that.
I think the cleanest fix is to close the previous transport in _connect_tcp before opening the new one, so a late FIN can never reach the protocol. Guarding both extract_packet calls in data_received against self._transport is None also seems worth doing, since a NULL dispatch shouldn't be able to take the interpreter down.
As a workaround we wrap loop.create_connection so it closes the previous transport when the same protocol comes back. That removed the crash and the DPY-4011 in my tests. Our service hasn't crashed in the day and a half since, but that's early days.
Disclaimer: this issue was generated by AI. The script and the output above come from real runs against a local Oracle Free database.
- Lingua principale
- Python
- Stelle
- 456
- Fork
- 119
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di oracle/python-oracledb
-
bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 22/100
oracle/python-oracledb#628 ·
-
Thin mode: ORA-01483 for string binds of 2731-8191 bytes on a database with the UTF8 character setAperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
oracle/python-oracledb#627 · 1 commento ·
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 75/100
oracle/python-oracledb#626 ·
-
bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
oracle/python-oracledb#625 ·
-
bug
Difficoltà 4/5 1-2 giorni Idoneità per principianti 45/100
oracle/python-oracledb#624 ·
Tutte le issue di oracle/python-oracledb
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
unionai-oss/pandera#2640 ·
I maintainer di solito rispondono entro 2 giorni
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
JdeRobot/RoboticsInfrastructure#843 ·
I maintainer di solito rispondono entro 1 giorno
-
automated endpoint-dead needs-curation
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
almanac-data/climate-almanac#122 ·
I maintainer di solito rispondono entro 1 giorno
-
area:harness enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
evarness-ai/iris-harness#420 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
suitenumerique/find#206 ·
I maintainer di solito rispondono entro 2 giorni