Windows: Winsock send errors never reach net_error, so the keepalive pool reuses a dead connection ("could not write forward header")

Start with src/flb_io.c around net_io_propagate_critical_error() and the send failure path, then trace connection release and reuse in src/flb_upstream.c. Read the Windows socket error handling in include/fluent-bit/flb_socket.h and event mask update in lib/monkey/mk_core/mk_event_libevent.c; reproduce using the issue's Windows sender and Linux receiver steps. Done when a failed send discards the dead pooled connection and the next flush reconnects.

Bug Report

Describe the bug

On Windows, a failed send() on a plain TCP upstream connection never sets its net_error. With net.keepalive on (the default) the dead connection goes back to the keepalive pool and is handed out again, so after a receiver restart a forwarder busy at that moment keeps writing into it instead of reconnecting.

I run Fluent Bit 5.1.5 (the latest release) on a fleet of 15 Windows machines forwarding to one Linux receiver. After a receiver restart on 2026-11-06 at 14:26 CET (again at 14:28), one machine logged could not write forward header 2,773 times in 47 minutes; the other 14 (same version and output config) resumed within about two minutes. At the next receiver restart (2026-11-07 00:36 CET) the same machine failed 223 times in 4 minutes, until I restarted its service. It was sending a backlog both times; machines whose pooled connection was idle recovered silently. Nothing was lost (filesystem buffer, retry_limit false), but the receiver got nothing from it meanwhile.

The affected machine's log (message texts verbatim; times and counts from my review):

• 14:26:14 to 15:13:14, about once per second, only:

  [error] [output:forward:forward.0] could not write forward header
  [ warn] [engine] failed to flush chunk '...', retry in ... seconds: task_id=..., input=... > output=forward.0 (out_id=0)
  

• No [net] or connect error. The receiver was up and the others were delivering, so a new connection would have worked: the attempts reused the dead pooled one.
• 15:13:14: all buffered chunks (task_id 0 to 332) succeeded at retry 1 to 11 in one burst, consistent with the backoff leaving the connection idle for 30 s and the idle sweep dropping it.

Cause, from the source (master fc966e9, 2026-11-07)

1. net_io_propagate_critical_error() (src/flb_io.c:263-279) switches on errno. Winsock reports through WSAGetLastError() and leaves errno alone, so on Windows nothing matches and net_error stays -1.
2. flb_upstream_conn_release() recycles a keepalive connection whenever net_error == -1 (src/flb_upstream.c:984-987).
3. The handout check in flb_upstream_conn_get() (src/flb_upstream.c:865-879) misses it: on Windows flb_socket_error() is evutil_socket_geterror() (include/fluent-bit/flb_socket.h:31), which returns the thread's WSAGetLastError() and reads SO_ERROR only when that is WSAEWOULDBLOCK.
4. The close watch on idle pooled connections never fires: the libevent backend of the Windows build maps only READ and WRITE (lib/monkey/mk_core/mk_event_libevent.c:228-233), and MK_EVENT_CLOSE (16 | 8 | 8192) has neither bit. Only the idle sweep (src/flb_upstream.c:1178-1187) remains, and a busy forwarder never idles for 30 s: each failed attempt releases the connection and refreshes ts_available.

On Linux errno carries ECONNRESET/EPIPE and the same code works. TLS is not affected: tls_net_read()/tls_net_write() in src/tls/openssl.c set net_error themselves on every fatal error.

A second defect, found by reading only: _mk_event_update() (lib/monkey/mk_core/mk_event_libevent.c:159-201) never updates event->mask. A pooled connection stays registered as MK_EVENT_CLOSE; when its send() would block, net_io_write_async() re-registers it for MK_EVENT_WRITE, then reads the stale mask (src/flb_io.c:516, 530) and returns -1 (553-559) without setting net_error.

917dc321 (2026-11-07, "io: mark blocking send timeouts as critical connection errors") added ETIMEDOUT to the switch and a #ifndef FLB_SYSTEM_WINDOWS branch to fd_io_write() (src/flb_io.c:343-354); Windows is unchanged. v5.1.5 (0039c4f0) has the same code a few lines earlier (src/flb_io.c:260-275, src/flb_upstream.c:966-1006); the 5.1 branch head (537321f) has no change. I found no open PR.

To Reproduce

As posted in #10079 on 2025-04-25:

1. Linux: fluent-bit -i forward -o null.
2. Windows: a dummy input and a forward output to that host, net.keepalive at its default.
3. Restart the receiver while the Windows side is sending.
4. Windows logs could not write forward header on every attempt until 30 s pass without one or the service restarts. With net.keepalive off the next flush succeeds.

My evidence is production on 5.1.5 (logs above) plus the source reading, not a lab run.

Expected behavior

The first failed send() marks the connection broken, flb_upstream_conn_release() destroys it and the next attempt connects anew, as on Linux. A receiver restart costs at most one failed flush per busy connection.

Your Environment

• Version used: 5.1.5 on Windows (MSI) and on the Linux receiver
• Configuration: forwarder output during both incidents; receiver input

  # [SERVICE] flush 5, storage.path set; inputs winevtlog, tail (storage.type filesystem)
  [OUTPUT]
      name                     forward
      match                    *
      host                     <receiver>
      port                     24224
      retry_limit              false
      storage.total_limit_size 300M
      # not set: net.keepalive (default on), Require_ack_response (default off)
  

  [INPUT]
      name              forward
      listen            0.0.0.0
      port              24224
      buffer_chunk_size 1M
      buffer_max_size   8M
      storage.type      filesystem
  

• Environment and OS: physical Windows 11 machine (the affected one); receiver: Linux VM, systemd; no containers
• Filters and plugins: winevtlog, tail → forward over plain TCP (no TLS, no shared_key); receiver forward → file; no filters

Additional context

• Workaround, rolled out to all 15 machines from 2026-11-07: net.keepalive off (each flush opens its own connection, a failed one is always destroyed) plus Require_ack_response On (a chunk counts as sent only after the ack). At a planned receiver restart on 2026-11-08 the 14 machines converted by then logged nothing; all 15 resumed.
• #10079 (Windows 3.1.7 forwarders to Linux, same message, about 70 % data loss with retry_limit 5) was reproduced there on 3.2.0 and 4.0.1 and closed as stale on 2025-07-29.
• Fix sketch: capture the error right after send()/recv() fails (WSAGetLastError() on _WIN32), before net_io_restore_event() runs (src/flb_io.c:565), and pass it to net_io_propagate_critical_error(); there map the WSAE* codes (WSAECONNRESET is 10054, not ECONNRESET) or treat every error except WSAEWOULDBLOCK as critical, as the TLS backend does. _mk_event_update() should set event->mask. A narrower stopgap: out_forward could call flb_upstream_conn_recycle(u_conn, FLB_FALSE) after a failed write (plugins/out_forward/forward.c:1550-1563).
• I can test a patch build on one of my Windows machines.