external_ptr issues

Start by reviewing the existing external_pointer implementation and the attached safe_external_ptr.txt example, then read the discussion about type tags, protected data, and copy and move semantics. Done would require a decided scope for improving external_pointer or adding an alternative wrapper, with its intended semantics documented and validated.

In a recent project I had some issues with external_pointer that I wanted to discuss. First, a bit of context: My goal was to make a simple C++ library consisting of a bunch of classes that do stochastic simulations accessible from R. My approach was pretty straight-forward -- for every class T I created a C++ function that calls the constructor returns an external_pointer<T>, and for every member function I created a cpp11 function that takes an external_pointer<T> as its first argument.

The issues I faced were the following:

1. No type safety. On the R side, there is no distinction between external_pointer<T> and external_pointer<U>, so passing the wrong type of object to a cpp11 function is possible and will lead to crashes.

2. No support for protected data attached to the external pointer. This is relevant for C++ classes that keep references to instances of other wrapped classes around. For these references to stay valid, the referenced objects must live at least until the external pointer owning the referencing object is destructed. This is exactly what external pointers guarantee for the protected data SEXP they support, but the current external_pointer implementation does not make this functionality accessible.

3. While the semantics of R's external pointer are clear to me (I think), I didn't quite understand the semantics of cpp11's external_pointer class. I would have expected the class to have essentially the same semantics as sexp for copies and moves, i.e. basically those just copy the underlying SEXP around, but take care to protect it from garbage collection. However, it seems that external_pointer instead considers itself to own a particular external pointer SEXP, and copies it (using Rf_shallow_copy) when external_pointer is copied. What is the rational there?

4. Probably due to (3) my attempts at wrapping a class whose instances keep references to other objects around lead to crashes. Basically, what I did was to let the external_pointer point not to the class directly, but to a holder struct that contains external_pointer instances for the object plus all the objects it references. Whether the problem was my code or external_pointer I couldn't tell because I didn't really understand the intended proper usage of external_pointer in this case.

I ended up creating my own external pointer class, safe_external_pointer (code is attached). I used external_pointer as a template, but changed the following things

1. The tag mechanism of R's external pointers is used to provide rudimentary type safety.

2. The protected data mechanism is exposed to the user

3. The underlying external pointer SEXP is considered to be immutable, i.e. never changed after its creation. The copy and move semantics of safe_external_pointer are the same as that of sexp. Thus, safe_external_pointer does not really "own" the underlying external pointer SEXP, but rather is one of many references to it.

For my use case, this simplified things a lot, and it got rid of the crashes I wasn't able to debug before. Whether other use-cases would be made more complicated by these changes (in particular (3)), I don't know of course. Still, I figured I'd share my code and see if there's any interest in either including parts of it into cpp11's external_pointer, or including it as an alternative external pointer wrapper alongside the existing one.
safe_external_ptr.txt