Enhancing _jupyter_types_experimental in completion metadata

The issue discusses completion responses and the experimental _jupyter_types_experimental metadata, but names no implementation files or tests. First clarify the metadata design and expected class and module information with the maintainers; done would require an agreed response shape, implementation location, and coverage for instance completions.

Hi all

I was wondering whether this additional metadata is being used, and whether the ipykernel team is open to enhancing it. In particular, the 'type' field could be more useful; it seems to return 'instance' for any class instance, without actually returning the name of the class or other potentially useful information.

This came up in a discussion on the Python team at Microsoft. We are (slowly) working on creating type stubs for scientific Python packages. Pandas is done and handed off to pandas team; we have stubs for matplotlib that are in reasonable shape at https://github.com/microsoft/python-type-stubs, and we're working on much more complete ones for sklearn and scipy, and trying to create them in a semi-automated fashion from a combination of execution traces, numpydoc docstrings, and human-in-the-loop corrections. Besides loading them up and seeing how they fare with some sample code, it can be hard to know when these stubs are wrong, unless people file issues in that repo. Besides contributing them to the community we use these stubs in Visual Studio Code for code completions, and in the case of Jupyter notebooks the code completions we infer from these stubs are enhanced by completions suggested by the kernel. We thought this could be a great place to detect stub errors - if the stubs say a type is X and the kernel says it is Y and Y is not a (sub)type of X, that could be a stub error. We could surface this mismatch in some way, like a code action that opens an issue in the stub repo. We got very excited when we saw that there was a 'type' field in the 'metadata' field in completion responses, only to find it didn't actually give us the class name.

Ideally in the case of an 'instance' we would also want the class name and the module it comes from. While we could query the kernel for this, that involves additional CPU time that we don't think is reasonable to use. Our hope is that this is readily available information that could be added to these messages for minimal cost.