Concurrent close of a CLI TCP connection closes a reused fd: stale poller entry, and HUP delivered to a media socket

The race spans lib/poller.c (poller_del_item_callback, poller_add_item, poller_poll) and daemon/tcp_listener.c (streambuf_stream_closed, streambuf_listener_newconn), with stream_fd_new and stream_fd_closed in daemon/media_socket.c. Reproducing it needs concurrent CLI disconnects under multiple poller threads, which is hard to set up. Done would mean the fd cannot be closed by a thread that does not own it and stale poller slots are cleared, with a regression test.

Summary

On rtpengine mr26.2.1.2 with 16 poller threads sharing one epoll instance, a CLI client that disconnects before reading the whole reply can make rtpengine close a socket it does not own. We saw two effects in production:

1. Read error on media socket: 0 (Success) -- closing call: an unrelated, active call is destroyed.
2. Failed to add stream_fd to poller repeating for hours: one fd number keeps a stale poller entry, and every new media socket that gets this number is never polled.

The relevant code is unchanged on master (checked against 12ecdbc2, 2026-10-08): lib/poller.c and lib/streambuf.c are identical, daemon/tcp_listener.c only has a function rename.

Environment

• mr26.2.1.2, num-threads not set (16 cores, 16 poller threads), no poller-per-thread, kernel forwarding enabled.
• We carry three unrelated patches (recording metadata slot numbers, recording daemon PTS). None of them touches the files below.
• A monitoring client connected to the CLI every 2 s (every 60 s while its browser tab was in the background), sent list numsessions\n, did a single read and closed. In a packet capture the client closed after reading only the first reply line, and its socket answered the rest of the reply with RST.

What we observed

• 58 x Read error on media socket: 0 (Success) -- closing call on two nodes over 7 days. All 58 are within +/-1 s of a CLI request from the monitoring client. Of the daytime seconds in that period, only 19% are within +/-1 s of such a request.
• On one node, Failed to add stream_fd to poller started at 09:23:18 and repeated about 3 times per minute until 16:08:49 (891 times). Among the affected calls that had a recording session, 24% had a leg with zero packets (baseline 0.67%).
• It stopped when a CLI connection got the same fd number (2004): adding it to the poller failed, nothing was logged, and the accepted socket was never closed. It is still in CLOSE-WAIT with the 17-byte command unread.
• We read the poller table of the running process read-only (/proc/PID/mem, offsets taken from the same build with debug info) and compared it with /proc/PID/fdinfo/<epoll fd>. Of 9,546 slots exactly one is inconsistent: pdata[2004] holds an item while fd 2004 is not registered in the kernel epoll. The item belongs to a stream_fd of a call created at 09:23:18.963, the same second as the first Failed to add. That stream_fd had been added successfully and was later released (socket.fd == -1).
• We have since switched our monitoring from the CLI to the NG statistics command.

Analysis (from reading the code)

1. One CLI connection can produce two closing events that are handled by two poller threads at the same time, for example the client's FIN/RST (thread A: streambuf_stream_readable() -> streambuf_stream_closed()) and the HUP that appears once cli_stream_readable() calls shutdown(SHUT_WR) (thread B: item.closed -> streambuf_stream_closed()). The fds are registered with EPOLLET but without EPOLLONESHOT. From epoll(7): "Since even with edge-triggered epoll, multiple events can be generated upon receipt of multiple chunks of data, the caller has the option to specify the EPOLLONESHOT flag ..."
2. streambuf_stream_closed() (daemon/tcp_listener.c:110-127) checks s->sock.fd == -1 without any exclusion, and resets the fd only after rtpe_poller_del_item() has closed it (lines 123-124). Both threads can pass the check holding the same fd number.
3. poller_del_item_callback() (lib/poller.c:142-178) closes the fd by number even when the slot is empty or EPOLL_CTL_DEL fails (lines 157-161, then 171-175; this behaviour came with 89313ed074). The second thread closes whatever socket owns that number at that moment. If a new media socket got the number and was already added to the poller, it is closed without being removed: the kernel drops its epoll registration, but pdata[fd] keeps the item.
4. When the owner of that stale item is released, EPOLL_CTL_DEL fails and the slot is not cleared (lib/poller.c:160-161), so the slot stays occupied for the lifetime of the process. Every later socket that gets this fd number fails in poller_add_item() (line 117), and stream_fd_new() (daemon/media_socket.c:4047-4056) carries on with a socket that is never polled. The deferred release then closes the stored fd number again, which can hit another socket or log Unable to close the socket for port.
5. poller_poll() dispatches by ev->data.fd only (lib/poller.c:212). An event fetched for the CLI connection but dispatched after its fd was closed and reused goes to the new owner. For a media socket, HUP/ERR calls stream_fd_closed() (daemon/media_socket.c:1395-1416), which logs Read error ... 0 (Success) (SO_ERROR is 0, the socket itself has no error) and destroys the call.
6. streambuf_listener_newconn() (daemon/tcp_listener.c:188-203) neither closes the accepted socket nor logs anything when the poller add fails. That is where the CLOSE-WAIT connection comes from.

Why we think rtpengine should handle this

A client may close at any time, and closing with unread data sends a RST (RFC 9293 section 3.6.1, SHLD-3). rtpengine-ctl reads until EOF, but it also gives up after its 3 s receive timeout. A client disconnecting should not be able to close another call's socket or destroy another call.

Possible fixes (suggestions only)

• Let only one thread close a TCP stream (for example, take the fd atomically), and keep the fd open until the stream object is freed, so the number cannot be reused while other threads still hold events or buffers for it.
• In poller_del_item_callback(), do not close by number when the slot does not hold the caller's item. When EPOLL_CTL_DEL fails with ENOENT or EBADF, clear the slot.
• Keep an identity or generation with each registration and check it on dispatch, so stale events are dropped.
• In stream_fd_new(), fail the allocation if adding to the poller fails. In streambuf_listener_newconn(), close the socket and log.

We can share more details if needed.