Ipython: Global variables undefined in interactive use of embedded ipython shell

[ LP comment 1 by: Fernando Perez, on 2010-04-25 23:36:38.673176+00:00 ]

OK, I can confirm the problem (even on trunk), but it's hard. I am just for now making sure this bug is confirmed so we keep tracking it, but I'm not sure how to fix it.

The issue is that in embedded shells, we try to update the global namespace to use the surrounding scope (that's the point of an embedded shell, to be able to see what's around you). But this then causes python to fail to resolve the ipython interactive namespace when nested things (like local functions) are defined. See the mainloop() method of the embedded shell for details.

I need to think a lot more about how to fix this one, any ideas very much welcome.

Over at launchpad...

chairmanK wrote 20 hours ago:

Me Too. In addition to functions, this bug also appears in generator expressions.

The equivalent component in trunk seems to be IPython.frontend.terminal.InteractiveShellEmbed. But that's broken in other ways, and evidently hasn't been tested much. Does anyone know what its future is?

Could this be the same issue as #136?

This has been now fixed:

amirbar[ipython]> python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython import embed
>>> embed()
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: a = 1

In [2]: def f(x):
   ...:     return a*x
   ...: 

In [3]: f(3)
Out[3]: 3

In [4]: b = 1

In [5]: (lambda : b)()
Out[5]: 1

In [6]: 

Can someone explain a little more what went into the fix of this? I still encounter the issue, but only when I'm one layer removed via a method call, and only if I run from a script, not interactively.

python 2.7.2 on OSX 10.6.8, ipython 0.11 and 0.12 both exhibit similar behavior (using 0.12 for this comment)

This is a problem with our project which features (prominently) an embedded IPython shell.

testembed.py

from IPython import embed

def hi():
    embed()

if __name__ == '__main__':
    #embed()
    hi()

Run this on the command line with python testembed.py and see this session:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
   ...:     print time.time()
   ...:     

In [3]: tim()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
[snip] in <module>()
----> 1 tim()

[snip] in tim()
      1 def tim():
----> 2     print time.time()
      3 

NameError: global name 'time' is not defined

In [4]: 

However, comment out the call to hi() and replace it with the embed() call:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
    print time.time()
   ...:     

In [3]: tim()
1328639444.29

In [4]: 

After poking around, I think it has to do with the stack_depth parameter used here and this block: https://github.com/ipython/ipython/blob/master/IPython/frontend/terminal/embed.py#L185

Thoughts?

This is somewhat complicated, but I believe that's a restriction in Python itself.

In the failing case you show, time is not actually put into the global namespace: because you called embed inside the hi function, new variables you create interactively are local to that function. Ideally, tim() should work as a closure, closing over the reference to the time module. However, closures only appear to work when the containing function is compiled in one go. As far as I can tell, there is no way to define a closure dynamically. This simple example fails:

def outer():
    import time
    exec("def inner(): return time.time()")
    return inner

outer()()

This is probably because nested scopes were added to Python relatively late (they were a future import in 2.1, and always on in 2.2).

Ok, I think I understand. Looks like we can't really do anything about this then, other than dumping what I would write interactively to a file and then reading that file back in. Probably too complex for what we want to be able to do interactively.

Thanks, btw.

Sorry to keep harping on here, but it just makes interactive sessions feel very clunky:

Regular python:

>>> d={'one':1, 'two':2}
>>> getkeys=lambda: d.keys()
>>> getkeys()
['two', 'one']

Regular IPython:

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
Out[3]: ['two', 'one']

Embedded IPython:

>>> from IPython import embed
>>> embed()
Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <module>()
----> 1 getkeys()

/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <lambda>()
----> 1 getkeys=lambda: d.keys()

NameError: global name 'd' is not defined

I guess there isn't much that can be done, but I don't get why we can create a dynamic closure in regular python/ipython but not the embedded version.

By the oddities of Python scoping, in standard Python/IPython, we're not actually creating a closure. d is a global variable, and the rules for accessing those work differently from closures. Each function keeps a reference to the global namespace where it was defined (getkeys.func_globals), so it can access any variables defined there.

In contrast, when you make a closure, Python attaches references to each variable over which it has closed, as determined by the compiler - but that only works when the inner and outer functions are compiled at the same time. It looks like this:

In [8]: def outer():
    a = 1
    def inner():
        return a
    return inner
   ...: 

In [9]: f = outer()

In [10]: f
Out[10]: <function __main__.inner>

In [11]: f.func_closure
Out[11]: (<cell at 0x9f5e344: int object at 0x9a830b0>,)

This is possibly done to save memory - if the closure carried a reference to the local scope where it was defined, none of the variables from that scope could be freed while the closure was live.

I'm using Python 2.7.2 and IPython 0.12.1 on OSX 10.7.3 and having this issue still. When I run ./manage.py shell of Django which invokes IPython.embed(), the issue occurs. However, manually invoking the embed() in Python shell or from simple script file, there is no problem.

There's not a lot we can directly do about that, but we should probably encourage third parties away from embed for non-trivial uses.

@takluyver Do you mean that better to use ipython directly in this case?

It's possible for Django to start IPython in a way that won't cause this problem, but that's not the way we currently make easy. The problem occurs when IPython starts with separate local and global namespaces. There's no reason that Django requires separate local and global namespaces, but that's what calling embed() inside a function implies.

For reference, here's the code in Django:
https://code.djangoproject.com/browser/django/trunk/django/core/management/commands/shell.py

@takluyver That makes sense, thank! I'm opening a Django ticket for this.

@takluyver, now that we've merged embed_kernel so all the main pieces are in place, do you want to tackle a bit a cleanup of this to make slightly more fine-tuned uses easier?

I'll have a look at what interface might make most sense.

I am still having trouble with this issue. I've tried narrowing down exactly what's causing it, and the best I can determine is that it only happens on Ubuntu 12.04. I've tried all of the latest release versions on other servers, and it works fine. But any time I define a function in ipython or use %cpaste to paste in a function from another file, the inside of that function has no access to the global scope. It makes it basically impossible to do anything useful in terms of writing functions on the fly.

I am still having this problem with IPython 0.13 when it is called from other tools (e.g. Django's debugsqlshell command). It's very frustrating that something as basic as defining a function is totally broken.

I think embed() is the wrong interface for those to be using. embed() is
intended more for examining the state of a running program, so it uses
separate local and global namespaces. To get round this issue, ipython
needs to be started with a single interface. Sorry, I haven't had time to
work out what the best way to do it is.

Not only ubuntu. debian wheezy also shows that.

FYI, the Django ticket @liokm created above is https://code.djangoproject.com/ticket/18204, which now points to https://code.djangoproject.com/ticket/17078, which looks to have been fixed in trunk. It should land in 1.5.

I m having the same issue on Ubuntu with Ipython 0.13.2

@bkvirendra that is fixed in django 1.6

But 1.6 is not even stable yet!

But 1.6 is not even stable yet!

Software are not always stable, and between releases there might still be bugs. But there is nothing that should be fixed in IPython. Even if we do something here, the fix would not be in IPython stable before it is released either...

Problem still persists in ipython==4.2.0. Interestingly under windows it is no problem but together with ubuntu global variables are not recognized.

Use case:

from ipywidgets import interact, FloatSlider, IntSlider,RadioButtons, Dropdown

@interact(sv1 = radio_1, Scenario = drop_1, sv2 = slider_2)
def update_map(sv1,sv2, Scenario):

Problem remains. IPython 3.1.0, Debian Whezzy.

If I recall correctly, this was fixed for a little while, and seems to be reintroduced (reproduced using IPython 5.1.0 on macOS here: https://git.io/vPDrJ).

(edit: I may be mis-remembering the fix. It's possible I just threw everything I needed into a startup file instead of using embed)

Just in case this helps anyone I've been using this piece of code as a workaround

    def fix_embed_globals(N=0):
        # Get the parent frame
        import inspect
        frame_level0 = inspect.currentframe()
        frame_cur = frame_level0
        N = 2  # I may have this value off by one. I rewrote this function to use only standard calls
        strict = False
        for _ix in range(N + 1):
            frame_next = frame_cur.f_back
            if frame_next is None:
                if strict:
                    raise AssertionError('Frame level %r is root' % _ix)
                else:
                    break
            frame_cur = frame_next
        parent_frame = frame_cur
        # Add locals to parent globals
        parent_frame.f_locals.update(parent_frame.f_globals)
        parent_frame.f_globals['_didfixipyglobals'] = True

I just call that function whenever I get the name error. It puts all locals in your current frame into the global dictionary. Its hacky but it works in most circumstances.

As noted at https://github.com/inducer/pudb/issues/103, a temporary workaround while using an embedded shell is: globals().update(locals()) (after defining the locally global variables, only).

Hmm, can the milestone of this bug be changed to one of the next versions?

Done.

will this ever get fixed?

AFAIK, my comments from a few years ago stand:

1. I believe we are limited by the way Python itself works. Closures and dynamic evaluation don't play well together, and IPython can't fix it. People have found workarounds to move local variables to a global namespace, but those are hacks which can cause other problems, because they're modifying the global namespace. I don't plan on putting such workarounds into IPython; I'd rather leave an issue with something we're not doing than introduce one by trying to be too clever.

2. Many places where people thought they wanted embed(), they should actually use start_ipython() instead and avoid this kind of issue.

I don't understand, and it makes me angry.

sorry, I was frustrated that things which would work if you typed them into an empty .py file didn't work here

but re-reading the history, it seems this is an issue with Django's manage.py shell specifically, 90% of the time I'm in ipython I'm doing that but it's easy to forget that's the case

embarrassingly I'm working a lot in obsolete version of Django (1.4)

according to earlier comment, newer versions of Django shell use ipython differently and may not have the problem? eg https://code.djangoproject.com/ticket/17078

apologies for the misunderstanding

Yup, I think it was fixed for Django 1.6. If you really can't upgrade, you might want to apply the fix manually. It looks like this was it: https://github.com/django/django/commit/3570ff734e93f493e023b912c9a97101f605f7f5

I found a workaround today that is posted at #10695. A simpler treatment for the case where the IPython is not embedded inside a function is shown at this thread. I am not an expert so please help to check the validity.