GS CPU rasterizer output depends on the caller's floating-point rounding mode (VU1 leaves FE_TOWARDZERO active during XGKICK)

Start in src/lib/vu/ps2_vu1_core.cpp: the fegetround/fesetround(FE_TOWARDZERO) pair around VU execution and its restore, plus finishXgkick(), which calls GS::processGIFPacket directly. Then read the rasterizer math in src/lib/gs/gs_cpu_backend.cpp. Done means GS output is the same under either caller rounding mode, and the ps2xTest/gs_cache suite still passes. The maintainer must first choose between a guard inside the GS entry points and a VU-side restore around the GS call.

Game: ICO (PAL, SCES-50760). Runtime: PS2Recomp eb8688b plus local fixes, Windows, MSVC (Visual Studio 2026), RelWithDebInfo. VU1 engine used for the measurements: a local copy of the VU1 engine from PR #254 (379b9ca, not merged), which our build uses by default. The base VU1 interpreter in eb8688b and main (e563698) has the same rounding-mode scope (read in the code, not measured).

The defect

The VU1 execution paths set the host thread's floating-point rounding mode to round-toward-zero while a VU program is executing, then restore the previous mode when execution completes. This pattern exists in the base VU1 implementation and in the VU1 engine of PR #254 (interpreted and compiled routines both run inside the same scope):

• Base interpreter, ps2_vu1_core.cpp: lines 1632-1633 (std::fegetround(), std::fesetround(FE_TOWARDZERO)) and line 1837 (restore), at e563698; same lines at eb8688b.
• PR #254, ps2_vu1_core.cpp: line 399 (set) and line 466 (restore), at 379b9ca.

XGKICK hands its packet to the GS while the VU program is still running: VU1Interpreter::finishXgkick() (lines 917-926) calls submitGifPacket(GifPathId::Path1, ...) or GS::processGIFPacket() directly.

The CPU rasterizer computes with floats and doubles, for example in gs_cpu_backend.cpp: barycentric weights (line 1321), Z (line 1328, double z = v0.z * w0 + v1.z * w1 + v2.z * w2;), colours truncated from floats (line 1333), and texture sampling. These operations follow the rounding mode of the calling thread.

Consequently, the same GS packet can produce different rasterized results when processed under different host rounding modes. In the current single-threaded path, a packet submitted synchronously from VU1/XGKICK can be processed while the caller still has FE_TOWARDZERO active. This is a host-side implementation leak; the GS hardware does not inherit the VU1's floating-point control state.

Measured effect (ICO)

Method: the GS stream was captured at the GS frontend entry points (initial VRAM and register state, every packet, end-of-window VRAM) for 4 windows of 20 vsyncs along the intro and menu. Each capture was then replayed offline on the same GS frontend and CPU backend sources, and the replayed VRAM was compared byte for byte with the VRAM the game produced.

Run	Replay rounding mode	Result (4 windows, 4 MB VRAM each)
Before the fix	round-to-nearest	Different in 4/4. Draw buffer: 74-79 % of pixels exact, the rest mostly off by 1 level out of 255 per colour channel, with occasional channel-value differences reaching 93-96 levels out of 255 in two windows, where the depth test selected a different primitive. Z buffer: 3-41 % of pixels differ; in 70-93 % of them the in-game Z is lower.
	round-toward-zero	Identical in 4/4, bit for bit.
After the fix below	round-to-nearest	Identical in 4/4, bit for bit.
	round-toward-zero	Different in 4/4 (88 k to 250 k bytes); now the in-game Z is the higher one.

The bias is what pointed to the cause: the in-game values were consistently lower (Z) or darker (colour). The decisive check is the RN/RZ control above, with the same replay tool; and the tool built with g++ on Linux matches the MSVC game output bit for bit once the rounding mode is the same. Builds using different optimization flags (-O2 and -O2 -mavx2 -mfma -ffast-math) also produced differences, but those differences were symmetric and did not show the systematic bias observed in the game/replay comparison. The -ffast-math build is only a secondary compiler-behavior comparison, not evidence about strict floating-point-environment compliance.

A counter at the GS entry points, with the fix active, shows that 66-72 % of the observed GS entries (per 600-vsync interval) arrive with FE_TOWARDZERO active. The remaining entries arrive with FE_TONEAREST; no other mode was observed. This is consistent with the VU1/XGKICK path accounting for most of the GS traffic, although the counter does not identify the GIF path for each individual entry.

Visually the difference is not noticeable in ICO (mostly one level out of 255, occasional depth-test flips). The capture/replay comparison is how it was found.

Fix tested

A small RAII guard at the top of each public GS entry point that can rasterize: processGIFPacket, processNativePackedGIFPacket, uploadImageNative, writeRegister, clearFramebufferContext, clearActiveFramebuffer. It reads the current mode, sets round-to-nearest if needed, and restores the caller's mode on every exit, so the VU program continues under its own rounding.

#include <cfenv>

struct RoundNearest
{
    int prev = FE_TONEAREST;
    bool changed = false;
    RoundNearest()
    {
        prev = std::fegetround();
        if (prev != FE_TONEAREST && prev >= 0)
            changed = std::fesetround(FE_TONEAREST) == 0;
    }
    ~RoundNearest()
    {
        if (changed)
            std::fesetround(prev);
    }
};

void GS::processGIFPacket(const uint8_t *data, uint32_t sizeBytes)
{
    RoundNearest roundNearest;
    std::lock_guard<std::recursive_mutex> lock(m_stateMutex);
    ...
}

Local checks of this fix (in our tree, not upstream tests):

• In game (ICO, intro, menu, second intro): picture unchanged to the eye. Frame rate per phase within 0.5 % of the previous run, VU1 cycles per frame within 1.3 %.
• The capture/replay results in the table above.
• The existing ps2xTest/gs_cache suite, built against our tree with the fix: 45/45 pass.
• A local test of 2,000 calls made under round-toward-zero and round-upward: the caller's mode was restored after every call.

Limits

• Tested on one game, on one machine, on 4 capture windows. Entry points not used by these windows are covered by reading the code, not by measurement.
• The cost of the guard was not measurable in ICO at its current frame rate. The standard rounding-mode functions go through the CRT. On MSVC x64, the floating-point control operation affects the SSE2 control word in MXCSR; on 32-bit x86, the CRT can affect both the x87 and SSE2 control words. Another project measured several percent of a thread when calling these functions per command with MinGW (see below). Reading and writing MXCSR directly (_mm_getcsr/_mm_setcsr) could avoid some CRT overhead on x86/x64, but would need to preserve the relevant control bits correctly.
• The fix does not change the rasterizer's formulas. It only makes the result independent of the caller.
• If std::fegetround() reports an error or std::fesetround(FE_TONEAREST) fails, the guard continues without forcing round-to-nearest. This did not happen on the configuration above (MSVC, x64), but it is not guaranteed on every platform; an assertion or a failure counter in a diagnostic build would make it visible.

Other ways to handle it, and related work

• VU side: restore round-to-nearest around the GS call in finishXgkick() instead of guarding the GS. This is narrower: it only covers the VU path.
• Opposite choice, in the Sinan-Karakaya fork (used by Rogue Galaxy Recompiled): its threaded GS backend records the caller's rounding mode with each command and applies it again on the GS thread. That keeps the frames identical to the single-thread path. Their notes report the same effect: "VU1 programs select rounding toward zero, the front end's Q/S/T arithmetic ran under it, and without it the logged frames differed in the last bit of those values" (victor-takai/rogue-galaxy-recomp, docs/STATUS.md). Their patch 0004-mxcsr-rounding-reads.patch also measured fegetround/fesetround through the MinGW CRT at 3 % and 4 % of their VIF1 thread.
• LRPS2 (libretro/parallel-ps2, a PCSX2-derived core, at ba43efb): the EE thread produces GS packets into a ring consumed by another thread (pcsx2/MTGS.cpp, comments around lines 59-64), and the VU's FP control register is set around VU execution and the host's restored afterwards (pcsx2/VUmicro.cpp lines 175-180). There, GS rasterization does not run under the VU's rounding mode.
• Not measured in this report: PR #359 moves rasterization to worker threads. The code search found no explicit fesetround/fegetround handling in its src/lib/gs path; the workers' actual rounding mode has not been measured. With the current single-thread path, PATH1 packets submitted synchronously from VU1/XGKICK can be rasterized while FE_TOWARDZERO is active. Making the GS independent of the caller would let the threaded and single-thread paths be compared exactly.

This report documents the defect; the local fix is offered as a tested reproduction and one possible solution, not as a merge request. The captures contain game data and are not attached; the tools and the numbers above are available on request.