Strange interaction with matplotlib magic

Reproduce the execution loop using jupyter_client and ipykernel with %matplotlib qt, comparing the iopub execution_state: idle message with wait_for_ready after an exception. Trace the kernel's abort behavior and determine which signal reliably permits the next execution request; done means the ordering is understood and the reported behavior has a verified resolution.

I have an app that uses jupyter_client to interact with an ipykernel. In that app I have an execution loop that sends an execute request and then waits for a reply on the iopub channel that has "execution_state": "idle". I assume that when this message is received it is done executing and ready to execute something else. I never send a second execute request until I get the idle-state reply.

This works until I use the %matplotlib magic to load a gui backend (e.g., %matplotlib qt). After that, things begin to behave strangely. If I send code to execute, and that code raises an exception, I get the error reply normally. But then whatever code I send next is aborted without ever being executed. I can only assume this is due to some interaction with the gui event loop that that matplotlib magic is creating, but I can't figure out exactly where the problem lies. It seems that the "stop aborting" event somehow is not processed until after the next execution request I send (even if the app just sits idle in between the exception-raising code and my next execution request). In other words things seem to happen out of order: I send a request, it raises an exception, the kernel "aborts the queue" (even though there are no other requests in the queue), I then send a new request, and the kernel then aborts that (even though it wasn't in the queue when the kernel decided to abort).

I'm not sure if this is a problem with ipykernel or with jupyter_client, but I'm posting here to see if there's something else I should be doing on the jupyter_client end. In particular, I want to know what message I need to wait for after submitting an execution request that tells me "the kernel is done processing, the next execution request you send will be processed and will not be aborted because of some lingering earlier error". I thought that was the idle-state message, but apparently not, because when I receive that message it seems the kernel might still be in a state where it is going to abort whatever I send it (because it thinks they were all part of one execution queue). Right now I fixed it by inserting a wait_for_ready call after each execution, but I'm not sure if that is overkill or could have any other unexpected effects.

Originally opened as jupyter/jupyter_client#904 by @BrenBarn, migration requested by @blink1073