nginx-mod-ubus: worker processes crash with SIGSEGV (use-after-free) on responses >4KB over HTTPS

Open
#30,648 2 comments 2 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, nginx
Domain
api, backend

Research direction

Start in ngx_http_ubus_module.c at ngx_http_ubus_send_body() and inspect its use of ngx_http_output_filter() and free_output_chain(). Reproduce the HTTPS /ubus request with a response larger than 4KB, then verify the response chain remains valid until nginx finishes sending it and LuCI polling no longer crashes workers.

Written by the indexing model from the issue text.

Description

Summary

nginx-mod-ubus (ngx_http_ubus_module) reliably crashes nginx worker
processes with SIGSEGV when a ubus RPC response is larger than ~4KB and the
vhost uses SSL/TLS. On an OpenWrt box running LuCI behind nginx, this causes
a worker crash roughly every 5 seconds while any LuCI overview page is open
in a browser (LuCI's default poll interval), because the periodic
network.interface dump / luci-rpc getHostHints calls return payloads
comfortably over 4KB.

Environment

  • OpenWrt: 25.12.5 (r33051-f5dae5ece4), target rockchip/armv8, aarch64_generic
  • nginx-ssl: 1.26.3-r4
  • nginx-mod-ubus: 1.26.3-r4
  • nginx-mod-luci: 1.26.3-r4
  • Access via: LuCI web UI, served by nginx as a UCI-managed vhost with ssl_certificate set, /ubus location using ubus_interpreter.

Observed behavior

logread shows worker processes dying on SIGSEGV repeatedly, at intervals
matching LuCI's poll interval (5s):

daemon.err nginx[852]: [alert] 852#0: worker process 27653 exited on signal 11
daemon.err nginx[852]: [alert] 852#0: worker process 27655 exited on signal 11
daemon.err nginx[852]: [alert] 852#0: worker process 27657 exited on signal 11
daemon.err nginx[852]: [alert] 852#0: worker process 27654 exited on signal 11
daemon.err nginx[852]: [alert] 852#0: worker process 14497 exited on signal 11
daemon.err nginx[852]: [alert] 852#0: worker process 14535 exited on signal 11

From the browser side, LuCI's frontend (luci.js) sees the aborted TCP
connection (kernel sends RST when the worker dies) and surfaces this as:

XHR request aborted by browser

A kernel crash dump for one of the crashed workers shows the fault inside
memcpy while nginx is writing/encrypting the response body:

nginx[14535]: unhandled exception: DABT (lower EL), ESR 0x0000000092000007, level 3 translation fault
pc : 0000ffff96af96b0 (memcpy in libc.so)
lr : 0000000000439e08 (nginx output / SSL filter)
x1 : 0000ffff96298260 (pointer into an unmapped page)
x2 : 0000000000001b14 (length: 6932 bytes — the size of the ubus
     network.interface dump response)

Root cause (as far as I can tell from the crash dump)

In ngx_http_ubus_module.c, the response buffer chain is passed to
ngx_http_output_filter() and then immediately freed via
free_output_chain(), e.g. (paraphrased from the send-body path):

static ngx_int_t ngx_http_ubus_send_body(request_ctx_t *request) {
    ...
    rc = ngx_http_output_filter(request->r, request->out_chain_start);
    free_output_chain(request->r, request->out_chain_start);
    return rc;
}

For a plain HTTP response, ngx_http_output_filter() normally consumes the
buffer synchronously, so freeing it right afterward is safe. But for an
HTTPS vhost, the SSL/TLS output filter buffers data before encrypting and
writing it to the socket, and can retain a reference to the buffer beyond
the return of ngx_http_output_filter(). On musl libc (as used by OpenWrt),
ngx_pfree() on a buffer over ~4KB goes straight to free()/munmap()
instead of returning it to a pool, so the page backing the buffer is
unmapped immediately.

When the SSL filter subsequently tries to memcpy() from that buffer to
encrypt/send it, it dereferences a pointer into an already-unmapped page,
producing a SIGSEGV (classic use-after-free). This matches the crash dump:
x2 = 0x1b14 (6932 bytes) is exactly the size of a network.interface dump response, well above the 4KB threshold where musl's free() unmaps
the page directly instead of pooling it.

This would explain why the bug is highly reproducible with LuCI (which
polls /ubus every 5 seconds and regularly gets multi-KB responses) but
much less visible with small ubus responses, and specifically on
HTTPS-terminating vhosts rather than plain HTTP.

Reproduction

  1. Set up an OpenWrt nginx vhost with SSL that proxies LuCI and includes an
    /ubus location using ubus_interpreter (nginx-mod-ubus), e.g.:
    location /ubus {
        ubus_interpreter;
        ubus_socket_path /var/run/ubus/ubus.sock;
        ubus_parallel_req 2;
    }
    
  2. Open LuCI's Status > Overview (or Network) page in a browser over HTTPS
    and leave it open so it polls /ubus every 5 seconds.
  3. Watch logread -f: within seconds, worker processes start exiting with
    signal 11, repeating roughly every poll interval.

Workaround

We worked around this by not using nginx-mod-ubus for /ubus at all, and
instead reverse-proxying /ubus from nginx to the stable
uhttpd-mod-ubus handler running on a loopback-only HTTP listener:

/etc/config/uhttpd:

config uhttpd 'main'
    ...
    list listen_http '127.0.0.1:8000'

nginx /ubus location:

location /ubus {
    proxy_pass http://127.0.0.1:8000/ubus;
    proxy_http_version 1.1;
}

After this change, repeated /ubus calls over HTTPS (including
multi-KB network.interface dump responses) no longer crash any worker
process, and LuCI's overview page no longer shows "XHR request aborted by
browser".

Suggested fix

In ngx_http_ubus_send_body() (and any other call site with the same
pattern in ngx_http_ubus_module.c), don't free the output chain buffers
until nginx has actually finished with them. Either:

  • Check the return code of ngx_http_output_filter() and only free
    buffers that were fully consumed (e.g. skip freeing when rc == NGX_AGAIN and let it be freed by the request pool / connection
    cleanup), or
  • Allocate the response buffer from r->pool and let it be freed as part
    of normal request cleanup, rather than freeing it explicitly right after
    the output filter call.

Happy to help test a patch against 1.26.3-r4 on OpenWrt 25.12 (rockchip/armv8) if useful.

Dominant language
Makefile
Stars
4.7k
Forks
4k
Avg merge
2d 2h
Merged PRs (30d)
251

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openwrt/packages

All issues in openwrt/packages

Similar issues

More Backend & API Design issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.