Ipython: IPythonShellEmbed fails to recognize local variables

Created on 19 Jul 2010  ·  18Comments  ·  Source: ipython/ipython

Issue

Embedded IPython shells can lose track of local variables.

Test Case

Minimal test case:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.Shell.IPShellEmbed()()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

To see the error, first run the code in Python (or IPython) and exit from the spawned shell; the final print statement correctly displays '15'. Run the code again, but this time type sum(foo.values[x] for x in foo.indices) in the spawned shell, and we receive the error

" NameError: global name 'foo' is not defined".

bug

Most helpful comment

I looked into this issue a bit, and it's definitely fixable (though maintaining fixes for both Python 2 and 3 may be messy).

The ChainMap solution would be easiest to include into IPython proper. However, there's a slight catch that eval/exec require globals to be a dict. Creating a class MyChainMap(ChainMap, dict): pass can work around this.

I also wrote a Python 3.5+ fix based on a different strategy of simulating closure cells and forcing the python compiler to emit the correct bytecode to work with them. The relevant file is here, part of my xdbg demo. It works by replacing get_ipython().run_ast_nodes.

As far as I can tell, the two approaches differ only in their handling of closures. When xdbg is embedded at a scope that has closed over some variables, it can correctly access those variables by reference and mutate them. Additionally, if any functions are created in the interactive interpreter, they will close over any local variables they need while allowing the rest of the local scope to be garbage collected.

All 18 comments

Hmm, where did you define 'foo'?

If you just run call foo outside of the function bar, foo should not exist (and obvously it doesn't).

When you instead run "sum(f.values[x] for x in f.indices)" you get 15 again...

Quite correct. However, I'm refering to the use of foo _inside_ the spawned IPython shell, which is in turn spawned inside the function definition where foo is a local variable.

Is this likely to be the same issue as #62?

No, @takluyver: this is a separate issue and indeed a real bug in our embedding code. I was hoping your recent work with namespaces would have fixed it, but it didn't. For reference, here's the example code to run with the current embedding api:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.embed()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

Then, in the spawned, embedded IPython, this fails:

In [1]: sum(foo.values[x] for x in foo.indices)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/home/fperez/tmp/junk/ipython/foo.py in <module>()
----> 1 sum(foo.values[x] for x in foo.indices)

/home/fperez/tmp/junk/ipython/foo.py in <genexpr>((x,))
----> 1 sum(foo.values[x] for x in foo.indices)

NameError: global name 'foo' is not defined

And it doesn't work even if we pass to the embed call user_ns=locals() explicitly, but in that case we get in addition a crash on exit:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2702, in atexit_operations
    self.reset(new_session=False)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1100, in reset
    self.displayhook.flush()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/displayhook.py", line 319, in flush
    self.shell.user_ns['_oh'].clear()
KeyError: '_oh'

It looks like our embedding machinery is in pretty bad shape...

Assigned this to myself to look at.

Excellent, thanks.

Unfortunately, I think this is a limitation of Python itself. It appears code compiled dynamically cannot define a closure, which is essentially what we're doing here with a generator expression. Here is a minimal test case:

def f():
   x = 1
   exec "def g(): print x\ng()"

f()

Which gives:

Traceback (most recent call last):
  File "scopetest.py", line 5, in <module>
    f()
  File "scopetest.py", line 3, in f
    exec "def g(): print x\ng()"
  File "<string>", line 2, in <module>
  File "<string>", line 1, in g
NameError: global name 'x' is not defined

Note that you can still see local variables in IPython - in the example given, plain print foo works. But you can't close a new scope over them.

I think it may be possible to make this work using collections.ChainMap from Python 3.3 so that IPython sees both local and global variables where it is embedded as globals. However, from the lack of noise about this over the last two years, I don't think it is high priority, so I'm retagging accordingly, and hopefully getting to this some time after 1.0.

Is it acceptable to up-vote, :+1: ? This affects me as well. I can add my use-case if requested.

I would be 100% okay if the fix for this only worked on Python 3.

I am also having the same problem under Python 3. Thanks for re-openning.

Same problem here :+1: both in Python 2 and 3.

I have this issue in both Python 2 and 3. It affects me on a daily basis.

I looked into this issue a bit, and it's definitely fixable (though maintaining fixes for both Python 2 and 3 may be messy).

The ChainMap solution would be easiest to include into IPython proper. However, there's a slight catch that eval/exec require globals to be a dict. Creating a class MyChainMap(ChainMap, dict): pass can work around this.

I also wrote a Python 3.5+ fix based on a different strategy of simulating closure cells and forcing the python compiler to emit the correct bytecode to work with them. The relevant file is here, part of my xdbg demo. It works by replacing get_ipython().run_ast_nodes.

As far as I can tell, the two approaches differ only in their handling of closures. When xdbg is embedded at a scope that has closed over some variables, it can correctly access those variables by reference and mutate them. Additionally, if any functions are created in the interactive interpreter, they will close over any local variables they need while allowing the rest of the local scope to be garbage collected.

IPython 6.0 and later versions only work in Python 3, so if @nikitakit or anyone else wants to open a pulll request with a test case and a fix for this, that would be welcome.

It's been a year since my last comment here, and during this time my understanding of the issue has changed a bit.

Interactively inspecting local scope remains an important feature to me, but there are actually a number of inter-related issues regarding local variables and the embedding machinery. For example, modifying local variables inside an embedded shell doesn't work:

>>> import IPython
>>> def test():
...     x = 5
...     IPython.embed()
...     print('x is', x)
...
>>> test()
Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:24:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: x
Out[1]: 5

In [2]: x = 6

In [3]:
Do you really want to exit ([y]/n)? y

x is 5

A ChainMap-based approach wouldn't help in this situation.

Another question is what happens when closures defined inside an embedded shell are leaked out into the global scope. Consider running the same code as above, but inputting something along the lines of IPython.get_my_x = lambda: x in the embedded IPython shell. A ChainMap-based solution will avoid causing a NameError in this situation, at the expense of potentially introducing two simultaneous copies of x that exist independently of each other (one being the ChainMap, and the other the local variable/closure cell used by the python interpreter).

Given the complexity of the situation, I've decided to focus my efforts on a more comprehensive approach to the problem (which also better aligns with my own usage of IPython). This has led to the development of xdbg, which is essentially a debugger that integrates with IPython. The key idea is to extend the IPython shell by offering debugger commands via magics (e.g. %break to set breakpoints). The fact that breakpoints are set externally, rather than by calling the function embed in-place, has allowed an implementation that addresses many of these problems with local variables.

I don't currently plan on pull-requesting a narrowly-targeted bugfix to core IPython. However, I'm very interested to know what IPython users and devs think about using a debugger-inspired interface for local code inspection. IPython (and now also Jupyter) have had a huge impact on the ability to do interactive coding in Python, but there are still many improvements to be made in how it interacts with heavy encapsulation using classes/functions/modules.

I've been burned a bunch of times by this bug. E.g.

[mcgibbon@xps13:~]$ cat test.py 
x = 1
def func():    
    x = 2
    import IPython
    IPython.embed()

if __name__ == "__main__":
    func()
[mcgibbon@xps13:~]$ python test.py 
Python 3.7.6 (default, Dec 18 2019, 19:23:55) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: [x for _ in range(2)]                                                                                                                                                                                                                 
Out[1]: [1, 1]  # lol whoops. looked up in wrong scope.

I'm not sure what can be done, but just adding my voice to the choir.

Based on https://bugs.python.org/issue13557 and the fact that explicitly passing locals() and globals() to an exec fixes @takluyver's reproduction of the bug, it seems like this _ought_ to be possible to fix in IPython by passing the correct local and global namespaces.

x = 1

def func():    
    x = 2
    exec("def g():\n print(x)\ng()")  # prints 1 :(
    exec("def g():\n print(x)\ng()", locals(), globals())  # prints 2 yay!


if __name__ == "__main__":
    func()
Was this page helpful?
0 / 5 - 0 ratings

Related issues

frenzymadness picture frenzymadness  ·  3Comments

zhao-ji picture zhao-ji  ·  4Comments

ericdill picture ericdill  ·  3Comments

alvations picture alvations  ·  4Comments

quchunguang picture quchunguang  ·  3Comments