dotnet/runtime

Safer failure with COM/C++/vtable interop and RTTI

Open

#86.458 geöffnet am 18. Mai 2023

Auf GitHub ansehen
 (7 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)C# (5.445 Forks)batch import
area-Interop-coreclrhelp wanted

Repository-Metriken

Stars
 (17.886 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 12T 11h) (661 gemergte PRs in 30 T)

Beschreibung

Interop with COM/vtables is currently performed by allocating an array of function pointers. However, this may be dangerous if some C++ code attempts to use dynamic_cast, typeid, or some other feature that requires the existence of RTTI. This is because the RTTI pointer is located just before the first entry of the vtable, which is out of bounds of the memory that was allocated.

To my knowledge, both the Microsoft and Itanium ABIs will treat a nullptr value for the RTTI pointer to mean "no RTTI is present for this type" (in the case of MSVC, __RTDynamicCast will catch an AV and treat that as "no RTTI"). I am not sure how the older GCC ABI deals with this.

Would it be desirable for the COM and vtable source generators (and the general guidance on vtable interop in .NET) to be changed from this:

void** vtable = new void*[methodCount];
// fill in vtable

to this?

void** vtable = new void*[methodCount + 1];
vtable[0] = nullptr;
vtable++;
// fill in vtable

It would ensure that the RTTI pointer is null and the proper failure messages can be displayed instead of potentially reading an invalid (but readable) pointer.

I think supporting RTTI is out of scope for .NET interop, but adding a dummy null pointer before the vtable seems like a good "best effort" to ensure that the failure is still deterministic if some code attempts to use RTTI with an interop vtable.

Contributor Guide