Xterm.js: Problem with emojis/unicode (assumed double-width?)

I can't reproduce this on macOS/bash. I wouldn't expect it to happen unless we tried to tweak with the emoji width (make it non-1). Any more details on repro steps would be useful.

Reproduce (OSX, MacOS High Sierra)

• start v3 demo
• paste 😀 multiple times
• hit left arrow key multiple times, see how it jumps two cells each time and runs into the cell boundaries.

Are you already on MacOS High Sierra, because I am? Maybe it is also related to node-pty - I can see that when inserting an emoji in Terminal.app it automatically adds a space after the emoji, and even more, I cannot select that space, it's like High Sierra would threat it as a double-width character...

Terminal.app on MacOS High Sierra

This might be a high sierra thing where they tried to make Emojis better in the terminal? I can't update currently as it messes up rendering in Electron apps.

I hate to say that I've upgraded and now I'm stuck with tons of rendering glitches in all those chromium based apps 😤

Yeah I was careful to track feedback before updating as similar things happened with Sierra I think.

I notice that Terminal.app has a nice emoji experience where they're treated as proper wide characters, we can probably do it if they can 😄

UAX #11 East Asian Width, Revision 31 changed emoji _East Asian Wide_ (double-width), and wcswidth in wchar.h returns the width of strings. I guess macOS High Sierra would have supported the update.
xterm.js's wcwidth in src/CharWidth.ts doesn't have supported it yet.

@mandel59 thanks for the info!

@mandel59 I think this issue is specific to emoji, I think CJK chars should be correctly categorized as double width chars (some components may be broken though like https://github.com/xtermjs/xterm.js/issues/1387)

Reference: https://github.com/gnachman/iTerm2/blob/master/sources/NSCharacterSet+iTerm.m has some mapping solutions too.

VS Code issue: https://github.com/Microsoft/vscode/issues/59145

Apparently flutter run demonstrates this behavior

Here's a small Dart repro for what we're seeing that doesn't require Flutter (does require Dart though):

import 'dart:async';

import 'dart:io';

// Windows console font has a limited set of Unicode characters.
final _animation = Platform.isWindows
    ? <String>[r'-', r'\', r'|', r'/']
    : <String>['🌕', '🌖', '🌗', '🌘', '🌑', '🌒', '🌓', '🌔'];
final _backspace = '\b' * _animation[0].length;

main() {
  int ticks = 0;
  void update(Timer timer) {
    if (ticks % 50 == 0) {
      stdout.write('\nDoing thing... ');
    }
    stdout.write('$_backspace${_animation[ticks++ % _animation.length]}');
  }

  new Timer.periodic(const Duration(milliseconds: 100), update);
}

Save as a .dart file and then run it with dart xxx.dart. This code here issues two backspaces for this character, which in a standard macOS terminal correctly removes the character. However, in the VS Code terminal it will backspace the emoji and then another character. If the emoji was at the start of the line this does nothing, but if it's not, the rendering slowly creeps to the left by a character (and leaves the right-half of the icon after it).

If I you remove the * _animation[0].length from the backspace so that it always does a single backspace, it works fine in the VS Code terminal, but no longer backspaces enough in the macOS Terminal (so the moons then creep to the right!).

Also applies to Hebrew/Arabic as mentioned in https://github.com/Microsoft/vscode/issues/60470

VSCode 1.31.1 on Ubuntu 18.04 experiences this issue.

When you scroll up or down your command history with the arrow keys, artifacts are picked up such that when you get to a command you want to run, chances are its illegible.

Also, after you pick up an artifact, editing the command is buggy, since what's displayed is different from what's in memory in xterm.

Steps:

Added an emoji to my prompt:

Hit up arrow once to get to command npm i

Notice the rogue n that was added to the prompt.

Remove the emoji from the prompt in .bashrc and reload the term and the bug goes away.

First things first, I wanted to thank you guys for your amazing work.

VSCode 1.31.1 on macOS Mojave 10.14.3 experiences the same. The n thing mentioned by @juliusecker happens to me too, exactly as he described.

I have the same bug as mentioned by @juliusecker.
VSCode Version 1.33.1 (1.33.1) 51b0b28134d51361cf996d2f0a1c698247aeabd8 2019-04-11T08:14:39.158Z
System Version: macOS 10.14.4 (18E226)
zsh 5.7.1 (x86_64-apple-darwin17.7.0)

Very simple to reproduce in VSCode and Powershell.

emojiterminaltest.ps1

"Emoji Normal"
"😊1😊2😊3😊4"
"Emoji with Padding Spaces - What it should look like"
"😊  1😊  2😊  3😊  4"

Are there any resources assigned to this item? Looks like lots of regressions in other programs (Hyper, etc.) as seen on this issue thread.

@jerch is this a duplicate of https://github.com/xtermjs/xterm.js/issues/1709?

@Tyriar Yes kinda, its just an example where ppl most likely will face it - emojis :smile_cat:

I guess we'll keep them both opened as this describes the problem well and #1709 looks more at the solution, plus this one has a bunch of upvotes.

It is replicatable in Windows 10, the following version:

The above is what I see in a VS code build-in terminal,

the above is what I see when the same text in a text file opened in VS code.

Issue has been around nearly two years, any update?

@jerch wants to get to it soon, he has a bunch of stuff ongoing atm though. One of the big blockers for this was the new addon system which is now released.

Closing in favor of https://github.com/xtermjs/xterm.js/issues/1709 as they're essentially the same thing and we want to keep our issue count down.