Ipython: Weird character after first import

Hum I've seen this one but with ^[[43;1R, but I assumed it was due to me being on master of my terminal emulator. Not sure where this come from

I've seen similar things in latest https://github.com/jonathanslenders/pymux (not released, uses prompt-toolkit 2). Maybe @jonathanslenders has some idea?

I get the same:

$ ipython
Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os                                                                                             

^[[19;1RIn [2]:

If I then try to tab complete:

In [2]: os.p                                                                                                  
os.pardir         os.pathconf       os.pathsep        os.popen          os.putenv         
os.path           os.pathconf_names os.pipe           os.pread          os.pwrite         
^[[19;1RIn [2]: os.p

This makes tab completion almost unusable.

Is it possible that this was fixed by the following commit: https://github.com/jonathanslenders/python-prompt-toolkit/commit/e226d640177aa1d2cf293e4de382f171586173a2 ?
It's merged in prompt_toolkit master, but I'm about to release a new prompt_toolkit 2.0 which includes it.

I installed prompt-toolkit from master and the problem still seems to be there, but I'm not sure if something else needs to be done on the ipython side.

No, I'm pretty sure this isn't IPython. I'll try to reproduce with iterm2

I'm sure I'm not understanding the complexities here, but just for information, this does occur with IPython 7.0 and 7.1, and doesn't occur with 6.0 or 6.5, testing in the same virtualenv.

Can you all post which terminal you've seen it in?
I'm able to reproduce on iTerm2 – 3.2.1beta6 – osx; alacritty v0.2.0-35-ga53cabf osx, but not on bare macos terminal.app (sierra 10.12.6)

Could it be mismatch between advertized features of terminal and what they actually can do ?

It also not reproducible (to me) with ipython --colors=nocolor

Just bare Terminal.app on High Sierra 10.13.6 . nocolor doesn't fix it for me.

It also not reproducible (to me) with ipython --colors=nocolor

Scratch that, it seem to be random, but indeed, nocolor did not fix it.

Can you guys try by removing this patch_stdout context manager ?

If I remove it, it _seem_ fine to me, and if I extra flush stdout I get the issue twice ..

Removing the context manager seems to fix it for me.

This happens to me in either the macOS Terminal.app and in iTerm2. However, the character appeared quite consistently in iTerm2 3.2.1beta. This morning I upgraded to 3.2.2beta1 and now the character seems to appear and immediately disappear, replaced by the normal iPython prompt. But at least in one case the character has been persistent, I'm not sure what was the difference.

Yup, fixes it for me too. The effect was always consistent for me.

@jonathanslenders could it be that patch stdout had a race condition and that stdout/err are restored, and start being written to, before the flush have happened ?

The raw parameter of patch_stdout seem to go nowhere also in the ptk code.

So there is reason to have this, or otherwise printing in background threads would mess-up the rendering, but it looks (to me), that is is a race condition between flushing captured stderr/out and restoring to their initial value.

Not sure if you localised the bug or still experimenting but for your information:

• macOS 10.13.6
• iTerm 3.2.1 - shows always ^[[41;1R after first input (import, expression, even blank line)
• Terminal.app 2.8.2 (404) - works fine!

Python 3.7.0 (default, Sep 18 2018, 18:47:22)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

^[[41;1RIn [1]:

I'm also getting this (macOS 10.11.6, iTerm 3.2.0beta5) very frequently, though not 100% of the time, and with the 43;1R sequence.

Python 3.7.0 (default, Jun 29 2018, 20:13:53)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sqlalchemy

^[[43;1RIn [2]:

Yeah, this one is really killing me. I'd downgrade, but that breaks Jupyter notebooks. For me, tab completion is more or less completely broken, because these characters appear very frequently.

Sadly readline is not an option at the moment : https://github.com/ipython/rlipython/issues/21

No change running current master of prompt_toolkit (version 2.0.5).

I can reproduce it.

iTerm2 build 3.2.3 (I think the latest), IPython 7.0.1, Python 3.6.1.

I had the bug with Python 3.6.3 on Ubuntu, and it disappeared with Python 3.7, although I can see some glitches (like the characters are printed and then quickly removed).

I have a fix here: https://github.com/jonathanslenders/python-prompt-toolkit/commit/09de545476be985b95ae2690ef8393efdd65b7e5

Actually what you see in this commit are two individual fixes that would both solve the issue (for what I could reproduce).

• For some reason, an empty string is captured and written to stdout, right before accepting the first input. This triggered the patch_stdout code. Actually, there was no need to go through the effort of erasing the prompt, and drawing it again, just for printing an empty string. (I still have to check why this is actually happening.)

• The real fix is to wait for CPR responses before rendering the captured content. CPR works asynchronously. We ask for the cursor position by writing something to stdout, and we receive the response from the terminal on stdin. There is always a little delay. But it's important to keep the terminal in RAW mode and read this input until this response arrives. We didn't do that, and it rendered the terminal's own response with the cursor position.

The timing should have been slightly different between terminals, but I guess this should do it.

Could you all try with the latest prompt_toolkit commit? If it works, I'll push a new release.

That fixes the bug on my side. Thanks @jonathanslenders !

Yes, many thanks, latest master fixes it for me too.

That patch fixes some [[39;1R chars I was having too. Thanks!

EDIT: Scratch that. I was testing the wrong version. Yes, this seems to fix it for me as well.

No, this doesn't seem to fix it for me. Instead I get a different character

AlbireoProˇalbireo: Downloads » ipython                 (3.7.0 2.7.15)

In [1]: from can.interfaces.slcan import slcanBus

^[[21;1RIn [2]:

Could you all try with the latest prompt_toolkit commit? If it works, I'll push a new release.

works for me. You also change the handling of colors AFAICT, we were unwillingly relying on ansi mapping to ansi code in some places in IPython, I'll fix that as well. Thanks !

@Carreau: I just pushed prompt_toolkit 2.0.6.
Does IPython expect that certain RRGGBB sequences map to certain ANSI colors, even in 256color mode?

By the way @Carreau, one one feature of prompt_toolkit now is the ability to increase/decrease the brightness of the colors. This makes it easy to adjust to terminals with light or dark background. I've added this to the menu of ptpython for testing (interactively) and it works pretty well.

expect

I wouldn't say "expect" but the themes seem to have a mix of #ansixxx and #00ff00 that looked good together, and with 2.0.6 are a bit stronger in their differences.

I think we just need more consistencies as to whether we use #ansi or #hex.

I'll look at the brightness options at some point. I just have started a new position a few weeks ago and have a bit less time on developing IPython/jupyter itself.

Interesting, but it makes sense. When a color is defined as #00ff00, I look at the closest color from the 256 color palette, but I'm excluding the 16 ansi colors. Which means it takes the closest from the remaining 240 colors.

The reason is that people these days have often custom color schemes defined for the ANSI colors - but not for the 240 remaining colors. So, while it could be a little off in your situation, it could actually be much closer to the real color for other people.

Here is an example of how the basic ANSI colors are rendered in different terminals:
https://en.wikipedia.org/wiki/ANSI_escape_code#Colors

closing as fixed upstream