Ipython: "Run All" in the notebook hangs on %matplotlib qt

I notice a similar thing with a Python 2.7 install from miniconda on windows. When I have a notebook where the first cell is:

%pylab

"Run All" will always hang even after having manually run that particular cell on its own. After this, I have to Kernel->Restart in order to get the notebook to respond to anything. My workaround is to run that cell manually by itself, move to the next cell and then "Run All Below."

I'm also happy to provide more info to help track this down further.

I can also confirm this (and it is rather annoying if you have a first cell with both %matplotlib and some imports (that take some time to execute), and you want to execute it together with the next cells).

A simple way to reproduce this:

1. A notebook with two cells:

%matplotlib import time time.sleep(10)

print "This will not print"

1. Execute both at once (press shift-enter twice)
2. The first cell will execute, but then the kernel hangs (second cell is not executed, you cannot execute anything, although the indicator says the kernel is ready).

So it is not necessarily related to "Run all", but just to executing multiple cells where one cell has %matplotlib.

Tested it last with IPython 2.1, Windows 64 bit, Firefox 31

After a little more experimentation, I found that this problem first shows up for me in version 2.0.0. If I roll back to version 1.2.1 with

conda install ipython=1.2.1

the problem goes away.

Windows 32 bit, python 2.7.7, Firefox 31

I also encountered this bug, and can reproduce using the above method using Python 3.4.1, Anaconda 2.1.0 (64-bit) on Windows 8, IPython 2.3.1
It's a pretty annoying bug. Is there something I can do to help track it down? When is the 4.0 milestone (roughly) expected to be completed?

I confirm the bug using recent ipython version from github (e2778c5f4047d03f5f025bd53d1704a0cf9f0337) on Linux Mint 17, python 3.4 and matplotlib 1.4.0.

Any news on this guys ? I can do more testing if needed. This bug is really annoying !!!

I can confirm that this happens to me as well, happens all the time on CentOS w/ python 2.7 and matplotlib 1.4.4

Does it work if you run the cell manually?

See my comment above (https://github.com/ipython/ipython/issues/5629#issuecomment-50603511), it also appears if you run multiple cells manually

Yes - it's basically a timing issue I believe. If you Shift+Enter run the cell, then wait a second, then continue with the others, then it works. If you press Shift+Enter multiple times repeatedly, then it hangs. Running further cells doesn't cause any code to actually be run (aka the little circle in the top right is still empty).

^ that's also what I observed.

Verified in Ipython Notebook 3.1.0. Workaround:

import time
%matplotlib
time.sleep(1)

Ugly, but it's too annoying giving up the option of run all.

I got this to work on my machine as well.

On Fri, Jun 12, 2015 at 7:02 AM, quicknir [email protected] wrote:

Verified in Ipython Notebook 3.1.0. Workaround:

import time
%matplotlib
time.sleep(1)

Ugly, but it's too annoying giving up the option of run all.

—
Reply to this email directly or view it on GitHub
https://github.com/ipython/ipython/issues/5629#issuecomment-111502934.

could we get this milestoned for 4.1 (or 5.0)?
Hanging notebooks sounds (to me) a bit more severe than appropriate for a wishlist milestone. Also, this bug is annoying when using matplotlib (despite the workaround).

Should we just have a sleep(1) in the matplotlib magic itself as a workaround until we figure out why ??

:+1:

The sleep workaround rarely works for me. I have seen it work a couple of times, but the vast majority of the times I've tried it, the notebook still hangs.

The only workaround that consistently works for me is to:

1. Manually run the cell containing %matplotlib
2. Wait for the kernel idle indication
3. select the following cell
4. "Run All Below"

FWIW I tried to test this a little to get a more reasonable estimate of how well this works.

I made a notebook with the following three cells:

import time

%matplotlib qt
time.sleep(1)

with open("test_log.txt", "a") as fid:
    fid.write("it worked\n")

then in the shell ran:

for i in {1..100}; do
    ipython nbconvert --execute test_qt.ipynb  --inplace;
done
wc -l test_log.txt

Maybe this test isn't exercising the right code, but it seems to work ok.

Marked as 4.1 so we look at it.

Thank you.
I just tried the repro of @mwaskom , and I got the same results. However, I then ripped out the time and sleep code, and ran the shell part again, and it _also_ worked! So, this does not seem to be a good repro procedure (or internal workaround). :-/
I also found, like @tanderl, that the sleep workaround does not consistently work for me. This is on ipython 3.2.1 on Ubuntu 64bit. I can try the same thing on Windows tomorrow.

On Windows 8.1, using Ipython 3.2.0 installed via Anaconda, I also found that the sleep workaround does not work for me, i.e. with sleep(1) (or even larger values) in the same cell as the matplotlib magic, Run All on a freshly restarted kernel will still hang (or do nothing, the UI still works), with a star on the following cell. Running the cells manually one-by-one, with waiting in between, still works.

I confirm what @bilderbuchi said with an updated anaconda stacks on Linux Mint.

Hi,
I've just experienced a similar issue. When running notebooks that use %matplotlib inline or notebook magics, the "run all cells" sometimes hangs and I have to run all cells by hand.
I'm running windows 7, Python 3.5, IPython 5.1.0, notebook 4.1.0 and matplotlib 1.5.3.
As a workaround, I clean all outputs and then it seems to work. I'm not able to provide a reproductible notebook, unfortunately (will do if I run across this in the future).

Best regards,
Florian

This is still in issue, in both jupyter notebook and lab.

@thomasaarholt do you have a reliable reproduction procedure?

@bilderbuchi see the original post in this issue, it's still the same issue. Here's a GIF of me recreating this on my machine (OSX w/ latest jupyter and qt4)

@Carreau, is that repro enough to work with? Should a hanging bug be more urgent than "wish list"?

@bilderbuchi @Carreau, I just spotted the wish list tag.
I'd like to make the case that this bug should be much, much higher priority:

Many new jupyter users will learn to use jupyter by pre-existing notebooks. For instance, they try running an example workbook provided by some plugin. With the current behaviour, users using the Run All command, or "Shift + Enter"ing quickly through the notebook will find that the notebook hangs from the very first moment, and this gives a very poor first impression. Such an impression can often really emphasise how "complicated coding is" to someone who is learning coding for the first time, or someone who has had a bad experience (class/exam) in the past.

I'd like to make the case that this bug should be much, much higher priority:

Well it's not a question of Priority anymore, the bug has been open for 2+ years now, and the issue is no one can figure it out. We did try to figure that out, but it seem to be an elusive fix, and we cannot be experts on all framework eventloop and libraries around. We would be happy to do something but the issue has been lingering around for so long that tagging it higher priority won't help. None of the devs know what's going on. Otherwise we would have fixed it.

@Carreau Can you point me at where in the ipython code the integration between the kernel server and the GUI should be happening?

Is there a way to programmatically open a notebook in a browser and run all cells? That would probably enable us to git bisect the code to nail down which change causes this.
I have tried executing the offending notebook via nbconvert --execute from the terminal and from within an IPython session, but couldn't get it to hang so far.

@bilderbuchi Just first off, are you able to reproduce the bug "by hand" on your machine?

Yes. I found some interesting behaviour trying different variations of the magic
First cell contents:

%matplotlib notebook -> works
%matplotlib inline -> works
%matplotlib -> prints Using matplotlib backend: Qt5Agg, then hangs
%matplotlib qt -> prints ModuleNotFoundError: No module named 'PyQt4' (I think that's a different unrelated bug)
%matplotlib qt5 -> prints nothing, then hangs

(The different bug when doing %matplotlib qt is ContinuumIO/anaconda-issues#1068 I think.)
The second cell only has

print("This will not print")

If I said "hangs" above, I mean that the second cell always has a star in the number instead of "2", and the message never gets printed.
All of this was executed with Kernel - Restart & Run All to ensure a clean slate. This is on Python 3.6.1 via Anaconda on Windows 10, IPython 5.3.0, Firefox 52.0.2.

Can you point me at where in the ipython code the integration between the kernel server and the GUI should be happening?

There's a description of how the event loop integration is meant to work here: http://ipython.readthedocs.io/en/stable/config/eventloops.html#event-loops-in-the-kernel

And the code for the Qt 5 event loop starts here: https://github.com/ipython/ipykernel/blob/20426cdad71820642dc69b95c30565d155521509/ipykernel/eventloops.py#L89

Just to check - is the problem definitely with the "matplotlib" part of %matplotlib and not with magics in general? Just in case the other magics take less time to run, and so the bug is less noticeable.

The problem also happens with the %gui qt magic, although I think that tries to import matplotlib under some circumstances so I am not sure.

This has nothing (directly) to do with matplotlib or magics. It is almost certainly a race condition when doing the hand off from one event loop running the show to the other.

My guess is that a similar fix to https://github.com/ipython/ipython/pull/10301/files is what is needed here.

I have faced a similar problem after updating Spyder

I can confirm this is still happening with %gui qt. I have a bunch of Qt stuff I call from notebooks and in those notebooks I have to run my first cell manually, wait for it to complete, then Run All Below.