Gigantic memory usage when using the SerialPort class under Citrix (but could happen everywhere else)

Description

A customer is complaining about our application consuming 27 GB of memory. We did an analysis of a dump and found that there were more than 96 million instances of the internal class SerialStreamAsyncResult (System.IO.Ports.SerialStream).

We think we have identified the problem in the method WaitForCommEvent in file SerialStream.Windows.cs.

There is an infinite loop

while (! ShutdownLoop)

You only leave the loop if you get any of three error codes:

  if (hr == Interop.Errors.ERROR_ACCESS_DENIED || hr == Interop.Errors.ERROR_BAD_COMMAND || hr == ERROR_DEVICE_REMOVED)
  {
      doCleanup = true;
      break;
  }

or if at the end of the loop and the completion callback decrements the variable _numBytes twice (the variable is initialized to 2):

if (Interlocked.Decrement(ref asyncResult._numBytes) == 0)
    threadPoolBinding.FreeNativeOverlapped(intOverlapped);

The completion callback is not being called because there is an error, so it never decrements the _numBytes.

Probably the error returned by Citrix is none of these three errors and the loop never leaves and creates all these instances killing the system.

There is even this code inside the loop that in reality documents that this could happen:

else if (hr != Interop.Errors.ERROR_INVALID_PARAMETER)
{
    // ignore ERROR_INVALID_PARAMETER errors.  WaitCommError seems to return this
    // when SetCommMask is changed while it's blocking (like we do in Dispose())
    Debug.Fail("WaitCommEvent returned error " + hr);
}

If the error code was ERROR_INVALID_PARAMETER or something other than the three error codes above or ERROR_IO_PENDING, then the loop would never end. There is a comment saying that there are at least three error codes Windows or drivers may return. I think we are getting a fourth one.

This is a huge issue as is consumes this incredible amount of memory till the session gets killed and the server gets really slow.

Reproduction Steps

Difficult to reproduce as it just happens sometimes under Citrix. We could imagine that using a driver that returns a different error code could at least show that there is an error in the logic. Anyways, if a driver returns an error that is not being handled in the method, it will enter an endless loop.

Expected behavior

The implementation should not enter an endless loop creating millions of objects. It would be better to really fail in this case, instead of what it is happening now.

Actual behavior

The memory dump is 27 GB, but we think that we already identified the issue.

Regression?

Yes, the customer has been working for years with our older version based on .NET Framework. We migrated to .NET 8 last year and now they are having the issues.

Known Workarounds

We didn't find any workaround.

Configuration

We are using .NET 8, but we looked at the current code and it has the problems we described, so it must be the same for at least all releases since .NET 8.

Other information

No response