nextafter and nexttoward return the finite endpoint in directed rounding modes

Start with the overflow return expressions in src/s_nextafter.c, src/s_nextafterf.c, src/s_nextafterl.c, src/s_nexttoward.c, and src/s_nexttowardf.c; nexttowardl aliases nextafterl. Read the adjacent-value requirements cited in the issue, then run its endpoint testbench across the rounding modes and signs. Done means all six variants return the outward infinity and raise overflow and inexact for the reported endpoint cases.

At the largest finite input, an outward nextafter or nexttoward call can return the input unchanged instead of signed infinity. The same overflow-return expression affects all six type variants.

Version and configuration

Tested OpenLibm commit: 5fe399749f9276eaa0b8403e507470da05cbbb3f. It produces libopenlibm.so.4.0.
The default upstream branch, master, still points to this commit as of 2026-10-12.

Configuration: Linux x86-64, LP64, binary80 long double with 64 significand bits.
Library compilers: GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1) and Clang 18.1.3 (Ubuntu 1ubuntu1).
Both libraries were built with GNU Make 4.3 and the default -O3, without fast-math.
Both caller compilers used -O0 -fno-builtin -frounding-math.
The comparison library was glibc 2.39 (Ubuntu 2.39-0ubuntu8.9).

Reproduction

Run on Linux x86-64 in a fresh working directory:

git clone https://github.com/JuliaMath/openlibm.git openlibm-repro
cd openlibm-repro
git checkout --detach 5fe399749f9276eaa0b8403e507470da05cbbb3f
make -j3 USEGCC=1

Save this unchanged testbench as repro.c in the checkout. The example is provided under CC0-1.0.

#include <float.h>
#include <stdio.h>
#include <openlibm_math.h>
#include <openlibm_fenv.h>

/* The next representable value outward from a finite format endpoint is
 * signed infinity, independently of rounding mode (C Annex F.10.8.3-4). */
#define CHECK(name, expression) do { \
    feclearexcept(FE_ALL_EXCEPT); \
    long double actual = (expression); \
    int flags = fetestexcept(FE_ALL_EXCEPT); \
    int ok = isinf(actual) && !!signbit(actual) == (sign < 0) && \
             (flags & (FE_OVERFLOW | FE_INEXACT)) == (FE_OVERFLOW | FE_INEXACT); \
    if (!ok) printf("%s(%sMAX, %sinf), %s: got %La flags=%#x; expected %sinf: FAIL\n", \
                    name, sign < 0 ? "-" : "+", sign < 0 ? "-" : "+", \
                    mode_names[m], actual, flags, sign < 0 ? "-" : "+"); \
    failures += !ok; \
    checks++; \
} while (0)

#pragma STDC FENV_ACCESS ON

int main(void)
{
    const int modes[] = {FE_TONEAREST, FE_UPWARD, FE_DOWNWARD, FE_TOWARDZERO};
    const char *mode_names[] = {"nearest", "upward", "downward", "towardzero"};
    int failures = 0, checks = 0;
    for (int m = 0; m < 4; m++) {
        fesetround(modes[m]);
        for (int sign = -1; sign <= 1; sign += 2) {
            CHECK("nextafter", nextafter(sign * DBL_MAX, sign * INFINITY));
            CHECK("nextafterf", nextafterf(sign * FLT_MAX, sign * INFINITY));
            CHECK("nextafterl", nextafterl(sign * LDBL_MAX, sign * INFINITY));
            CHECK("nexttoward", nexttoward(sign * DBL_MAX, sign * (long double)INFINITY));
            CHECK("nexttowardf", nexttowardf(sign * FLT_MAX, sign * (long double)INFINITY));
            CHECK("nexttowardl", nexttowardl(sign * LDBL_MAX, sign * (long double)INFINITY));
        }
    }
    fesetround(FE_TONEAREST);
    printf("%d of %d endpoint checks failed\n", failures, checks);
    return failures != 0;
}

Compile and run:

gcc -std=c11 -O0 -fno-builtin -frounding-math -Iinclude -Isrc \
    repro.c -L. -Wl,-rpath,"$PWD" -lopenlibm -o repro
./repro

To repeat with a Clang library and caller:

make clean
make -j3 USECLANG=1 CC=clang
clang -std=c11 -O0 -fno-builtin -frounding-math -Iinclude -Isrc \
    repro.c -L. -Wl,-rpath,"$PWD" -lopenlibm -o repro
./repro

Expected and observed

N1570, 7.12.11.3, PDF page 274 and 7.12.11.4, PDF page 275 define the adjacent-value operations.
Annex F.10.8.3 paragraph 2 and F.10.8.4 paragraph 2 says: “Even though underflow or overflow can occur, the returned value is independent of the current rounding direction mode.”
The outward neighbor is signed infinity for each finite format endpoint.
Overflow and inexact must be raised.

Rounding mode	Positive endpoint toward +infinity	Negative endpoint toward -infinity
FE_TONEAREST	+infinity	-infinity
FE_DOWNWARD	Positive finite endpoint, wrong	-infinity
FE_UPWARD	+infinity	Negative finite endpoint, wrong
FE_TOWARDZERO	Positive finite endpoint, wrong	Negative finite endpoint, wrong

These observations apply to nextafter, nextafterf, nextafterl, nexttoward, nexttowardf, and nexttowardl.
All four library/caller compiler combinations report 24 of 48 endpoint checks failed and exit 1.
Overflow and inexact are raised even in the failing cases.
Both glibc caller controls pass all 48 checks.

Source and scope

The overflow branches return x+x in s_nextafter.c:67, s_nextafterf.c:57, s_nextafterl.c:70, s_nexttoward.c:62, and s_nexttowardf.c:50.
nexttowardl aliases nextafterl.
Source inspection explains the result: x+x follows the active rounding direction after the bit step has identified infinity.
The overflow return needs to preserve that adjacent value while generating the required exceptions.

Current FreeBSD msun has the same x+x overflow returns in all five implementations and the same long-double alias.
This is a source comparison. No FreeBSD binary was tested.

Both signs and all four rounding modes were tested on the stated Linux configuration.
Independent double-precision nextafter inward and neighboring finite-step controls pass for both OpenLibm builds and glibc.
Library binding, rounding-mode changes, full x87 precision, and disabled flush-to-zero were checked.
Beyond the endpoint tests and double-precision controls described above, other architectures, operating systems, long-double formats, and adjacency cases were not tested.
The Annex F comparison is for this supported IEC 60559 consumer configuration.

Found by ChipLab's AI verification agents and independently reproduced.

ChipLab ref: CLAB-2026-716