ISO on VeraCrypt volume hangs Explorer when mounted (MapIrpDataBuffer MDL length check)

Start with MapIrpDataBuffer in src/Driver/EncryptedIoQueue.c and the MDL length check added in commit e4e6b16. Reproduce the short-MDL read using the Windows ISO and VeraCrypt volume described in the issue, then trace how the read buffer and requested length are handled. Done means the ISO can be mounted and its executable sections read without the driver writing beyond the MDL-described buffer.

Disclaimer: This was fully written by Claude Code Fable 5.2. I discovered the issue, and asked claude to find out what's causing this.

Windows: mounting a Windows-setup (UDF) ISO stored on a VeraCrypt volume hangs Explorer for minutes — driver fails a short-MDL read with STATUS_INVALID_USER_BUFFER (MapIrpDataBuffer)

Summary

When an ISO file that lives on a mounted VeraCrypt volume is mounted with the Windows built-in ISO mounter (double-click in Explorer, or Mount-DiskImage), and a PE file on that ISO is then mapped as an image, the VeraCrypt driver completes one of the resulting backing-file reads with STATUS_INVALID_USER_BUFFER (0xC00000E8) without doing any I/O.

The read is then re-issued from Windows' virtual-disk driver (vhdmp.sys) about 18–20 times per second for 160 seconds, after which the virtual CD-ROM reports STATUS_IO_DEVICE_ERROR. (I did not determine which Windows component drives the retries.) The requesting thread is blocked for the whole time. In practice Explorer freezes as soon as the ISO is mounted, because it maps setup.exe, which autorun.inf names as the drive icon (icon=setup.exe,0). Explorer retries afterwards, so the freeze repeats in 160-second cycles until the image is ejected.

The same ISO mounts and works normally from a non-VeraCrypt NTFS volume on the same machine.

The rejecting code appears to be the MDL length check in MapIrpDataBuffer (src/Driver/EncryptedIoQueue.c), added in commit e4e6b16 (2025-09-15):

if (irp->MdlAddress)
{
    if (MmGetMdlByteCount(irp->MdlAddress) < length)
        return STATUS_INVALID_USER_BUFFER; // caller asked for more than mapped

The VeraCrypt_1.26.24 tag does not contain this function or this status code.

Environment

Item	Value
OS	Windows 10 IoT Enterprise LTSC 2021 (21H2), build 19044.7725, x64, fresh install
VeraCrypt	1.26.29 (driver veracrypt.sys 1.26.29.3). Same symptom seen with 1.26.31 on a previous Windows installation of the same machine (no trace captured there)
VeraCrypt driver config	VeraCryptConfig = 6144 (0x1800), VeraCryptEnableMemoryProtection = 1
Volumes	Two non-system, device-hosted volumes (whole disk, no partition table), NTFS, mounted as fixed (not removable). Traced on the one hosted on an NVMe SSD
Sector sizes (C: and both VeraCrypt volumes)	512 logical / 4096 physical, 4096-byte clusters
File system filters on the volume	FsDepends, UCPD, Wof, FileInfo (all Microsoft). No third-party filter drivers. Microsoft Defender disabled
Windows drivers	vhdmp.sys 10.0.19041.7417, udfs.sys 10.0.19041.7725, cdrom.sys 10.0.19041.1, ntfs.sys 10.0.19041.5607
ISO	en-us_windows_10_iot_enterprise_ltsc_2021_x64_dvd_257ad90f.iso, 4,851,668,992 bytes, UDF 1.02 with ISO9660 bridge (CDIMAGE 2.56). Not sparse, not compressed

A second, custom-built (NTLite) Windows setup ISO with the same autorun.inf shows the same behaviour. Non-Windows ISOs do not freeze Explorer. The ones I inspected (FreeBSD, GhostBSD, a small PowerISO data image) are plain ISO9660 and have no autorun.inf entry pointing at an executable, so mounting them never exercises the failing read.

Steps to reproduce

With Explorer

1. Mount a VeraCrypt volume (here E:), NTFS.
2. Put a Windows installation ISO on it.
3. Double-click the ISO.

Result: the new drive appears, Explorer stops responding within about 5 seconds and is eventually reported as hung. The System log receives cdrom event 51 every 2 min 40 s.

Without Explorer (isolates the trigger)

Run elevated. The image is mounted without a drive letter, so Explorer is not involved; the script maps setup.exe the same way icon extraction does.

$iso = 'E:\base.iso'      # ISO on a VeraCrypt volume
Mount-DiskImage -ImagePath $iso -NoDriveLetter | Out-Null
$vol = (Get-DiskImage -ImagePath $iso | Get-Volume).Path      # \\?\Volume{...}\

Add-Type -Namespace P -Name N -MemberDefinition @'
[DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
public static extern IntPtr LoadLibraryExW(string path, IntPtr h, uint flags);
'@
# LOAD_LIBRARY_AS_DATAFILE | LOAD_LIBRARY_AS_IMAGE_RESOURCE
Measure-Command { [P.N]::LoadLibraryExW($vol + 'setup.exe', [IntPtr]::Zero, 0x22) }

Dismount-DiskImage -ImagePath $iso

Warning: on an affected system the LoadLibraryExW call blocks for minutes, and Dismount-DiskImage blocks until the pending read has failed (up to 160 seconds).

Results on my machine:

ISO location	Result
VeraCrypt volume (E:\base.iso)	LoadLibraryExW returned after 320 s (two retry cycles); touching the .data section then blocked again. 2 cdrom event 51 entries
Same VeraCrypt volume via \\localhost\E$\base.iso	Same hang
Plain NTFS on NVMe (C:), byte-identical ISO (same SHA-256)	93 ms, no errors
Plain NTFS, every PE-type file (.exe, .dll, .efi, .mui, .sys, …) on the ISO mapped the same way (256 files; second ISO 294 files)	All succeeded in under 1 s. At least 140 sections per ISO start on a 2048-byte boundary and end in a partial sector, like the failing one described below

Expected behaviour

The ISO mounts and files on it can be read and mapped, as with VeraCrypt 1.26.24 and earlier (by source inspection; I have not re-tested 1.26.24 on this installation) and as on a non-encrypted volume.

Analysis

Captured with a kernel ETW trace (WPR: FileIO, DiskIO, HardFaults, CSwitch with stacks) while double-clicking the ISO. Times are seconds from trace start; the mount happened at 63.27 s.

1. The read that fails

setup.exe starts at ISO byte offset 6,518,784. Its PE header has FileAlignment = 512, SectionAlignment = 4096:

Section	Raw offset	Raw size	Starts on a 2048-byte boundary	Shorter than one 2048-byte sector
.text	1024	6656	no	no
.rdata	7680	4608	no	no
.data	12288	512	yes	yes
.pdata	12800	512	no	yes
.rsrc	13312	51712	no	no
.reloc	65024	512	no	yes

Explorer's paging reads on \Device\CdRom0\setup.exe follow the section table one to one:

63.6503s explorer.exe Read off=0     size=4096  flags=0x60043    0.342ms st=0x00000000
63.6506s explorer.exe Read off=1024  size=6656  flags=0x60403    0.547ms st=0x00000000
63.6512s explorer.exe Read off=7680  size=4608  flags=0x60403    0.517ms st=0x00000000
63.6517s explorer.exe Read off=12288 size=512   flags=0x60403 160018.576ms st=0xC0000185
...after the first failure...
223.6703s explorer.exe Read off=12800 size=512   flags=0x60403    0.907ms st=0x00000000
223.6712s explorer.exe Read off=13312 size=51712 flags=0x60403    1.117ms st=0x00000000
223.6723s explorer.exe Read off=65024 size=512   flags=0x60403    0.373ms st=0x00000000
223.6727s explorer.exe Read off=12288 size=512   flags=0x60043 160018.105ms st=0xC0000185

Only the .data read fails: 512 bytes at a file offset that is a multiple of the 2048-byte media sector size. The other two 512-byte reads (.pdata, .reloc) do not start on a sector boundary and succeed.

2. What reaches the VeraCrypt volume

The corresponding reads issued by vhdmp.sys (System process) on the backing file, non-cached (IRP flags 0x101 = IRP_NOCACHE | IRP_READ_OPERATION):

63.6514s System Read off=6529024 size=2048 flags=0x101 0.135ms st=0x00000000 \Device\VeraCryptVolumeE\base.iso
63.6516s System Read off=6531072 size=2048 flags=0x101 0.100ms st=0x00000000 \Device\VeraCryptVolumeE\base.iso
63.6517s System Read off=6531072 size=2048 flags=0x101 0.015ms st=0xC00000E8 \Device\VeraCryptVolumeE\base.iso
63.6954s System Read off=6531072 size=2048 flags=0x101 0.087ms st=0xC00000E8 \Device\VeraCryptVolumeE\base.iso
63.7471s System Read off=6531072 size=2048 flags=0x101 0.020ms st=0xC00000E8 \Device\VeraCryptVolumeE\base.iso
... repeated every ~50 ms; more than 5,600 identical failures in the captured window

6,531,072 = 6,518,784 + 12,288, i.e. the sector holding the .data section. Note that a read of the same offset and length had succeeded 0.1 ms earlier (presumably issued for the preceding request, with a different buffer). Offset, length and sector alignment are therefore not the problem, which leaves the buffer.

Call stack of the failing read at issue time (module + offset, no symbols):

fileinfo.sys+0x282D
FLTMGR.SYS+0x64CC / +0x5F7A / +0x5021 / +0x4A2B
ntoskrnl.exe+0x26C505
vhdmp.sys+0xD831
vhdmp.sys+0x4C17 / +0x48AF / +0x4431 / +0x23EB5 / +0x758A / +0x74B6 / +0x5057 / +0x83B52
ntoskrnl.exe+0x22BD35 / +0x34E4C5 / +0x407738

3. Which component rejects it

Thread activity around one failing read (microseconds relative to the FileIo Read event):

   0.0us FileIo Read   tid=8212 (vhdmp worker)  off=6531072 size=2048
  10.5us CSwitch       cpu=13 new=6180 old=8212      <- wake stack of 6180 is in veracrypt.sys
  14.3us CSwitch       cpu=13 new=idle old=6180      <- back to waiting after < 4 us
  87.3us FileIo OperationEnd status=0xC00000E8

A VeraCrypt driver thread picks the IRP up and goes back to sleep in under 4 µs. No disk I/O is issued for these requests (the physical disk under the volume shows 136 I/Os in the whole 384-second trace, none slow). In MapIrpDataBuffer, the only path that returns STATUS_INVALID_USER_BUFFER for an IRP that carries an MDL is MmGetMdlByteCount(irp->MdlAddress) < length.

I have not attached a kernel debugger, so I have not read the MDL fields directly. My interpretation is:

• The memory manager asks UDFS for 512 bytes with an MDL describing 512 bytes.
• Because the request starts on a sector boundary but is shorter than a sector, the request that travels down the stack (UDFS → cdrom.sys → vhdmp.sys → NTFS → VeraCrypt volume device) has its length rounded up to one 2048-byte sector while still carrying the original MDL.
• VeraCrypt sees Length = 2048 with MmGetMdlByteCount = 512 and fails the IRP.

On a plain disk the same request is serviced without complaint, which is consistent with the 140-section test above.

4. Windows-side symptoms

cdrom event 51, "An error was detected on device \Device\CdRom0 during a paging operation", every 160 s. Decoded from the event data of the captured hang:

IRP_MJ_READ, FinalStatus = 0xC0000185 (STATUS_IO_DEVICE_ERROR), ErrorCode = 0x80040033
SRB: SrbStatus = 0x44, ScsiStatus = 0x02, DataTransferLength = 0x800, TimeOutValue = 10
CDB: 28 00 00 00 0C 75 00 00 01 00   -> READ(10), LBA 3189 (byte offset 6,531,072), 1 block

While that read is pending, Explorer's window thread is blocked opening the root of the new volume (Create \Device\CdRom0\ took 160,016 ms), which is what makes the shell appear dead. Application Hang event 1002 for explorer.exe was logged in an earlier occurrence.

Why I think this is worth fixing in VeraCrypt

The length check is a reasonable defence: without it a malformed request could make the driver write decrypted data past the buffer it was given. But this particular request shape is produced by Windows' own in-box ISO stack, and releases without the check (1.26.24 and earlier, by source inspection) would not have rejected it. For users it is a regression: an ordinary action (mounting a Windows ISO kept on an encrypted data disk) hangs the shell for minutes at a time.

Suggested fix (untested sketches)

Either of these would keep the protection while serving the request.

A. Clamp the copy-back instead of failing. For reads, when the MDL is shorter than length, read and decrypt into a driver-owned buffer, copy only MmGetMdlByteCount(irp->MdlAddress) bytes into the caller's buffer, and complete the IRP successfully with the full length. Nothing is written outside the described buffer, and the caller (which only described 512 bytes) gets everything it can use.

B. Accept the request when it stays inside pages the MDL already locks. This mirrors what a physical disk does with the same request:

if (irp->MdlAddress)
{
    ULONG mdlBytes = MmGetMdlByteCount(irp->MdlAddress);
    if (mdlBytes < length)
    {
        // Pages covered by the MDL are locked and are mapped in full by
        // MmGetSystemAddressForMdlSafe. Allow the transfer only if it ends inside them.
        SIZE_T mappedBytes = (SIZE_T) ADDRESS_AND_SIZE_TO_SPAN_PAGES(
                                 MmGetMdlVirtualAddress(irp->MdlAddress), mdlBytes) * PAGE_SIZE;
        if ((SIZE_T) MmGetMdlByteOffset(irp->MdlAddress) + length > mappedBytes)
            return STATUS_INVALID_USER_BUFFER;
    }
    ...

Option A is the stricter one. Option B changes less code but writes into the remainder of a page beyond the described byte range, so it may only be appropriate for read IRPs that are paging / non-cached kernel requests. The maintainers will know better which fits the queue design.

Workarounds

• Copy the ISO to a non-VeraCrypt volume before mounting it (tested, works).
• Open or extract the ISO with an archive tool, or attach it directly to a virtual machine, instead of mounting it in Windows.
• Accessing the ISO through a loopback UNC path (\\localhost\E$\...) does not help.

Available on request

• The ETW trace (.etl, about 500 MB) and the decoded report it was analysed from.
• The scripts used for the Explorer-free reproduction and the all-executables test.