dotnet/runtime

Safer failure with COM/C++/vtable interop and RTTI

Open

#86.458 aperta il 18 mag 2023

Vedi su GitHub
 (7 commenti) (0 reazioni) (0 assegnatari)C# (5445 fork)batch import
area-Interop-coreclrhelp wanted

Metriche repository

Star
 (17.886 star)
Metriche merge PR
 (Merge medio 12g 11h) (661 PR mergiate in 30 g)

Descrizione

Interop with COM/vtables is currently performed by allocating an array of function pointers. However, this may be dangerous if some C++ code attempts to use dynamic_cast, typeid, or some other feature that requires the existence of RTTI. This is because the RTTI pointer is located just before the first entry of the vtable, which is out of bounds of the memory that was allocated.

To my knowledge, both the Microsoft and Itanium ABIs will treat a nullptr value for the RTTI pointer to mean "no RTTI is present for this type" (in the case of MSVC, __RTDynamicCast will catch an AV and treat that as "no RTTI"). I am not sure how the older GCC ABI deals with this.

Would it be desirable for the COM and vtable source generators (and the general guidance on vtable interop in .NET) to be changed from this:

void** vtable = new void*[methodCount];
// fill in vtable

to this?

void** vtable = new void*[methodCount + 1];
vtable[0] = nullptr;
vtable++;
// fill in vtable

It would ensure that the RTTI pointer is null and the proper failure messages can be displayed instead of potentially reading an invalid (but readable) pointer.

I think supporting RTTI is out of scope for .NET interop, but adding a dummy null pointer before the vtable seems like a good "best effort" to ensure that the failure is still deterministic if some code attempts to use RTTI with an interop vtable.

Guida contributor