[BUG] Signed integer overflow in IffInput::readimg() row-flip pass -> OOB read/write on images larger than INT_MAX bytes

Start with src/iff.imageio/iffinput.cpp, especially IffInput::readimg() around the row-flip pass at lines 1050–1061; the report includes a reproducer and ASAN trace. Review the existing IFF reader hardening and tests, then reproduce with pixel decoding enabled. Done means the oversized-image offsets are safe and the regression is covered by a test.

Summary

While auditing the IFF reader after the #5458 zbuf chunk-size fix, we found
that the row-flip pass at the end of IffInput::readimg() computes buffer
offsets with 32-bit signed arithmetic, while check_open() happily admits
images whose byte size exceeds INT_MAX (dims up to 65536 x 65536 are allowed
by #5284's guard). For such an image the index (height - y - 1) * width * bytespp overflows, wraps negative, and m_buf[negative] turns into a wild
pointer that is then passed to memcpy 鈥?an out-of-bounds read and write
well outside the ~2 GiB pixel buffer.

The decode path itself stays fully in bounds (every RLE/span guard added over
the last year holds; we verified both by reading and by construction), so the
flip pass is the only site where the oversized image becomes memory-unsafe.

Still reproduces on current main (121c8c0cc31c5e2fd62db5e2d5d1f6007df154bf,
2026-10-17) at the same line.

This is an incomplete fix of the recent IFF reader hardening series; see the lineage under Incomplete fix below.

Incomplete fix

This report is an incomplete fix finding against the recent IFF reader
hardening series:

• #5284 ("reject implausible image dimensions before allocating tile buffer")
  raised the accepted dimensions to 65536 x 65536 without auditing the
  downstream arithmetic. An image of 65536 x 10924 RGB8
  (2,147,745,792 bytes > INT_MAX) passes every check it added -- that is
  exactly the shape this PoC uses.
• #5458 ("Guard nonsensical zbuf chunk size") then patched a single
  chunk-size case; its commit message itself records the point-patch pattern
  ("We had the analogous check for RGBA chunks, but missed ZBUF chunks").
• The same series also includes #5466 (CVE-2026-59956, tile staging buffer
  heap reveal).

None of these touched the 32-bit index arithmetic in the row-flip pass of
IffInput::readimg(). The raised dimension limits are what allow
check_open() to admit images whose pixel byte count exceeds INT_MAX,
turning the pre-existing int overflow into an out-of-bounds read+write. The
decode path holds every guard the series added -- the flip pass is the
residual gap of this hardening round.

Reproducer

IFF file, 65536 x 10924, RGB 8-bit, RLE, one 65535x10924 tile (32 MB, mostly
the RLE payload the decoder requires):
flip_int_overflow.iff distributed as OIIO-01_poc.zip (SHA-256 below).

$ oiiotool --info -v -a --hash flip_int_overflow.iff

Notes for reproducing:

• --hash (or any operation that reads pixels) is required 鈥?the bug is in
  the pixel-decode path, not the header parse.
• On hosts with more than ~4 GiB of free RAM, add
  ASAN_OPTIONS=max_allocation_size_mb=2560 (or run in a constrained
  container). Without an allocation cap the ~2 GiB m_buf allocation merely
  succeeds and the crash still happens at the flip, but oversized-allocation
  classes blur together on big-memory machines.
• Any tool that decodes the image hits it (oiiotool, iconvert, maketx,
  iinfo); the crash is in the reader.

The image is deliberately just over the boundary: 10924 * 65536 * 3 = 2,147,745,792 > INT_MAX, so the first flip iteration's source index
(10924-1) * 65536 * 3 = 2,147,549,184 overflows by 65,537 and wraps to
-2,147,418,112.

Output

Build: clang 16, -g -O1 -fno-omit-frame-pointer, AFL++/PCGUARD
instrumentation + ASAN, contract assertions at default (ENFORCE),
ASAN_OPTIONS=...:max_allocation_size_mb=2560.

==866883==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa60ea19800 ...
    #2 memcpy ...
    #3 OpenImageIO::v3_3_0::IffInput::readimg() src/iff.imageio/iffinput.cpp:1058:9
    #4 OpenImageIO::v3_3_0::IffInput::read_native_tile(...) src/iff.imageio/iffinput.cpp:556:18
    ...
    #9 OpenImageIO::v3_3_0::pvt::compute_sha1(...) src/libOpenImageIO/printinfo.cpp:59:25
SUMMARY: AddressSanitizer: SEGV in memcpy

src/iff.imageio/iffinput.cpp:1058 is memcpy(src, dst, ...) with src
derived from the wrapped negative index. Identical SEGV, same line, on
main at 121c8c0cc.

Root cause

Current main, end of IffInput::readimg() (src/iff.imageio/iffinput.cpp
1050-1061):

int bytespp = m_header.pixel_bytes();

std::vector<unsigned char> flip(m_spec.width * bytespp);
unsigned char *src, *dst, *tmp = flip.data();
for (int y = 0; y < m_spec.height / 2; y++) {
    src = &m_buf[(m_spec.height - y - 1) * m_spec.width * bytespp];  // int math
    dst = &m_buf[y * m_spec.width * bytespp];                        // int math

    memcpy(tmp, src, m_spec.width * bytespp);
    memcpy(src, dst, m_spec.width * bytespp);
    memcpy(dst, tmp, m_spec.width * bytespp);
}

m_spec.width/height are int, bytespp is int, so every product above is
computed in 32-bit signed arithmetic. check_open(m_spec, {0, 1<<16, ...})
(iffinput.cpp:287) admits width and height up to 65536, and
check_compression_ratio() (iffinput.cpp:288) only applies to declarations

= 1 GiB with a 10000:1 limit 鈥?a 2.0 GiB declaration backed by a 32 MB RLE
file (64x expansion, well under the reader's own plausibility check of 128x in
readimg()) passes all three gates. The signed overflow then yields a
negative index; operator[](size_type) converts it to a huge size_t, and
the memcpys read and write at m_buf.data() - ~2 GiB.

A minimal fix is computing the offsets in size_t / imagesize_t (e.g.
(imagesize_t)(m_spec.height - y - 1) * m_spec.width * bytespp), which keeps
the reader memory-safe for any dimensions that check_open() accepts. (The
other 32-bit multiplications in this function 鈥?tile_size, the
tw*th*channels span sizes 鈥?we believe are currently unreachable as memory
errors because their bounds checks share the same wrapped product, but they
are worth converting in the same pass.)

Impact

• Contract assertions enforced (OIIO 3.x default) or ASAN builds: process
  abort (SEGV) on a malformed file 鈥?denial of service for oiiotool,
  iconvert, maketx, iinfo, and any application decoding untrusted
  Maya IFF files.
• The wild memcpy is an out-of-bounds read and write at a
  attacker-chosen distance (up to 卤2 GiB from the buffer) 鈥?in a
  OIIO_HARDENING_NONE+NDEBUG build this is a potentially exploitable
  heap corruption, not merely a crash.

Related work

• #5284 鈥?dimension plausibility guards (limits raised to 65536 per side;
  did not audit downstream 32-bit arithmetic).
• #5458 鈥?"Guard nonsensical zbuf chunk size"; the commit message itself
  notes it is a single-point patch ("We had the analogous check for RGBA
  chunks, but missed ZBUF chunks"). This report came out of an
  incomplete-fix audit of that commit.
• CVE-2026-59956 / #5466 鈥?earlier IFF reader hardening (oversized memcpy,
  tile staging buffer).

Attachments

OIIO-01_poc.zip (34,521 bytes, SHA-256
ccf3cc0f93f136611ca2bf3238629f3c9617b23989ab323920fc832fa0775eb7)
containing:

• flip_int_overflow.iff (33,558,100 bytes) 鈥?the PoC described above.
• flip_int_overflow.asan.log (8,584 bytes) 鈥?full ASAN log of the crash.

Meta

Found during a manual audit of iffinput.cpp anchored on #5458, followed by
a targeted PoC construction; the accompanying AFL++ campaign could not reach
this path from small seeds (the trigger needs a multi-MB RLE payload that
decodes without error). 19 out of ~63M executions is not how this one was
found 鈥?the PoC is hand-built, deterministic, and attached. Full ASAN log
included in the attachment zip.

Happy to provide additional variants (RGBA/16-bit paths reach the same
overflow with different dimensions) if useful.

Credit

Discovered and reported by FortuneZ (GitHub: Fortune-zfc).