Async widget methods deadlock due to on_msg not being called

Beginne damit, den await-Fall von send/on_msg des Widgets zu reproduzieren und sein Verhalten mit IPython issue 12786 und ipykernel PR 589 zu vergleichen. Erledigt ist es, wenn der on_msg-Callback die Antwort verarbeiten kann, während die Zelle wartet, ohne Deadlock und ohne eine zweite Zelle zu benötigen.

We are developing a Jupyter widget which makes it possible to embed interactive Unfolded maps into notebooks:

Our widget is embedded via an iframe and so the communications with the JavaScript side are asynchronous. Some of the functions we want to add to the widget need to return data asynchronously. To communicate between Jupyter and JS we are using widget's send and on_msg methods. Our methods return futures. It works fine - the messages are being sent and received, but only if we do it in two steps:

When we are trying to use await so that we can have the results and do something with them within one cell, the execution is blocked forever:

We tried to investigate the cause. It turns out that the callback we register with widget.on_msg only gets called once the execution of the cell is finished. Hence, if we wait until we obtain the result of the calculation within the same cell where the method was called, we end up in a deadlock.

So far we've had no success in finding a workaround, despite trying many things. For instance, here we run the communication in a separate thread and wait until the query response is added to a blocking queue:

Calling the above function fails (after the 2 seconds timeout) with an error saying that the queue is empty:

And we don't see the _receive function being called.

However, if we check the queue immediately after getting this error, we can see that the queue is not empty - the response was added to it, apparently, just after the error caused the cell execution to be cancelled:

It might be the same issue as this one: https://github.com/ipython/ipython/issues/12786
but I am not completely sure. I tried installing and running the kernel with the fix from this PR, but I was getting errors related to traitlet messaging when trying to instantiate our widget.