Xterm.js: ZModem support?

Could you please get into some more details on how this would work on xterm.js?

I use xterm.js to connect to a bash(a full-feature bash) and let user interact with the bash. I'd like support user to upload/download file through web terminal by using rz/sz.

If I connect to a machine' shell with SecureCRT/Xshell, I can upload/download files with rz/sz command. I'd like use zterm.js to connect to a machine's shell, and upload/download files with rz/sz command

Zmodem is a protocol feature of is the underlying serial line abstraction of the PTY system. This is already usable with tools like zssh in xterm.js. xterm.js itself is just forwarding the master IO of the PTY to the browser. In reality you are operating on the system where the xterm.js server part is hosted.

This is quite different to a terminal emulator operating on the same machine like SecureCRT does.

So the question is - From which machine to which machine do you want to move files through the terminal? (Remember your local machine has no terminal channel to the server part of xterm.js).

|       local machine               |
|   +----------------------------+  |
|   |       browser              |  |
|   |   +-------------------+    |  |               +---------------------------+          +------------------------+
|   |   |                   |    |  |               |   proxy                   |          |                        |
|   |   |   web terminal    >----------------------->   server part of xterm.js >----------| romote machine terminal|
|   |   |                   |    |  |               +---------------------------+          +------------------------+
|   |   |                   |    |  |
|   |   +-------------------+    |  |
|   +----------------------------+  |
+-----------------------------------+

I'd like to move files from local machine to remote machine. The remote machine can not be accessed by local machine directly, since it has no external ip.

Remember your local machine has no terminal channel to the server part of xterm.js

Is there any workaround for this?

Well you can use zssh (with zmodem support), sftp or scp from the proxy.

Unless someone implements it there is no way to up/download files directly from the webterminal in your browser to the remote machine. One of the modem protocols could be used that way (there is a XModem JS implementation), but since the browser has very limited filesystem access I question the usefulness. The hard part would be to provide a download interaction with the browser for big files (since the data is landing directly in the browser through the websocket, JS would have to hold all data until it can be saved.)

Another approach would be to enhance the server part with rz/sz functionality and map that transparently to the browser. This would circumvent the FS limitations of the browser but would need to hook another parser into the terminal stream to catch the 'rz\r' initialisation of the modem protocols.

I’ve been working on a ZMODEM implementation in JavaScript and have it transferring files in both directions via xterm.js.

The browser does indeed have to buffer the entire file when it receives. :( That’s really the only downside, though, and it’s not bad unless you’re sending very large files. Uploads can be done via FileReader, and downloads happen via the “download” attribute of <a> elements.

I’m still testing and documenting it, but is ZMODEM integration a feature that would interest xterm.js?

@FGasper Nice :) Sounds promising, if you implemented it with node's stream API it could also be useful for big files as a server side extension, that proxies those files to the browser endpoint via download link.

The library itself is platform-agnostic, so it should work anywhere. (Knock on wood.)

@FGasper sounds neat 😄!

We can definitely consider a zmodem addon, if it works in the browser.

What are usually the use cases for using zmodem?

I use it to do file transfers within a terminal session to/from my workstation. It allows you not to have to scp/sftp or what not.

Similar to this for iTerm2:
https://github.com/mmastrac/iterm2-zmodem

@parisk The X/Y/Z-MODEM protocols allow to "abuse" a terminal connection for direct file transfer from and to the other side. This is very useful if you want to orchestrate a server only through a single terminal session. For POSIX systems there are the rz/sz tools to accomplish this.

Well for xterm.js it is a perfect addon, it will raise the usefulness of xterm.js for server admins in restricted environments. :+1:

NB: XMODEM and YMODEM both have the drawback of not “prompting” the other side that there’s a file transfer ready. (It’s unfortunate because they’re much simpler!)

ZMODEM sends what is essentially a “magic string” that can be watched for; then you prompt the user, “ZMODEM detected; proceed with file transfer?” At that point the implementation provides means either to provide files to transfer or to accept/skip files as the sender offers them.

For xterm.js it would be neat to just drag'n drop files into the terminal - lets say a file gets saved at the current terminal shell path. But thats very tricky to accomplish unless xterm.js can control the remote shell itself or at least grab the current working directory at any point. Same goes for backwards - if a ls listing could be spotted as referring to remote files, drag'n drop out of the terminal would be neat. Well just dreaming :wink:

I don’t see why drag/drop rz wouldn’t be doable once the Zmodem session is started. It could be implemented multiple ways, but maybe:

1) Type rz into your console.
2) Drag/drop files in.
3) Start the transfer.

Question: How set is xterm.js still on IE support?

No more IE support in v3 due to https://github.com/sourcelair/xterm.js/pull/938, developers should be off it anyway.

I’d like to show you folks a demo.

I’ve been testing against my own terminal server but would like to use whatever you guys have as default.

^^ Am I missing something here? npm install errors out for me …

felipe@Macintosh-4 18:00:56 ~/code/p5-Net-WebSocket/demo
> sudo npm install
npm ERR! install Couldn't read dependencies
npm ERR! Darwin 16.7.0
npm ERR! argv "/opt/local/bin/node" "/opt/local/bin/npm" "install"
npm ERR! node v8.3.0
npm ERR! npm  v2.15.12
npm ERR! path /Users/felipe/package.json
npm ERR! code ENOPACKAGEJSON
npm ERR! errno -2
npm ERR! syscall open

npm ERR! package.json ENOENT: no such file or directory, open '/Users/felipe/package.json'
npm ERR! package.json This is most likely not a problem with npm itself.
npm ERR! package.json npm can't find a package.json file in your current directory.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/felipe/code/p5-Net-WebSocket/demo/npm-debug.log

Maybe you forgot git clone ...? And why with sudo?

@FGasper

npm ERR! package.json ENOENT: no such file or directory, open '/Users/felipe/package.json'

You're running npm install in your home directory, not the repo directory.

Is there a way to get node-pty to output binary rather than strings? ZMODEM is a binary protocol, but the demo seems to think everything is UTF-8. In particular, something seems to be converting 0x8a to UTF-8 \ufffd

This might be a bit more involving - for a quick fix you can try to disable the encoders of node-pty or set the string type to 'binary', but this might have unwanted impacts on the websocket transfer to xterm.js. As far as I can tell the whole chain node-pty <---> websocket <---> xterm.js relies on UTF-8 bytes that get decoded to JS strings on the fly. Might need a greater patch to get binary data flowing.

Where is that binary string type setting? (I don’t see such in node-pty?)

As best I can tell, the data is already corrupted by the time it hits app.js, so the change in behavior would have to be in node-pty.

ZMODEM does spell out an encoding that escapes all high-bit characters down to 7-bit, but it was apparently never actually implemented … Forsberg probably figured it would never be needed as all the lines became 8-bit safe, heh. :)

You can try with {encoding: 'binary'} in ctor options or .setEncoding('binary') on the fly.

It looks like the most recent node-pty allows setting or not setting UTF8 mode …

  if (info[8]->ToBoolean()->Value()) {
#if defined(IUTF8)
    term->c_iflag |= IUTF8;
#endif
}

https://github.com/Tyriar/node-pty/blob/master/src/unix/pty.cc

Hm … the node-pty that xterm.js pulls in (0.4.1) is pretty old … maybe the most recent version would include that flag.

Calling .setEncoding('binary') doesn’t work … and neither does passing in encoding:"binary" to pty.spawn().

IUTF8 of termios does a different thing - it enables correct handling of multi byte UTF8 chars in the pty device (for line width and erase).
Hmm maybe the hard way - try to remove the encoder:

delete ptyObj._socket._readableState.decoder;
delete ptyObj._socket._readableState.encoding;

This should give you buffer objects instead of strings (most likely to break upwards to xterm.js).

OK, I got it by upgrading to node-pty 0.6.4 and setting encoding to null. (Didn’t need the _readableState stuff.) It does still send the first line of the shell session as text, but everything after is binary, so cool.

It does still send the first line of the shell session as text...

Even if you apply it to the constructor as {encoding: null}?

Yes, even with {encoding: null} the first line is sent as a text frame.

I’ve got this into what I think is a reasonable state to try it out.

1) Set up https://github.com/FGasper/xterm.js.git as a remote.
2) Check out that repo’s zmodem branch.
3) git submodule init; git submodule update
4) npm install (See below.)
5) npm start, then load localhost:3000 in the browser. (Chrome is what I’ve tested.)
6) ssh to a machine that has lrzsz installed.
7) Type rz, and send one or more files from your workstation to the remote.
8) sz <filename1> <filename2> … will send files in a batch to your workstation.

(The UI is minimal by design; assumedly a “real” deployment would polish it more.)

Concerning step 4: When I tested just now on my workstation I had to fix a permissions issue with the node-gyp package.

Nice, works like a charm (tested with text files in Firefox).

Just a few remarks:

• Is it possible to trigger the init sequence by accident (e.g. random data output)? If so, the terminal would IMHO need an "enter file transfer" setting to make this explicit.
• The "Start ZMODEM session" should be interruptable, selecting "No" does not abort rz on the other end atm.
• Is it possible to draw some progress bar / indicator into the terminal widget while the transfer is ongoing? Or some other more fancy stuff, atm rz/sz prints some weird status numbers into the terminal.

@jerch

1) Yes, it’s possible to trigger the init sequence by accident. That’s what the “Start ZMODEM session” prompt is for: the user can still back out if needs be.

2) Should be fixed now.

3) The application receives progress events in sync with the browser’s FileReader API. Chrome seems to give content with progress; however, Firefox doesn’t actually give the file content in those events. If you send yourself a suitably large file in Chrome you’ll see something like:

4) The weird characters at the start of a session are the printable parts of the ZMODEM receive-init sequence: ** + ASCII CAN + B01 + 10 hex characters + CR + 0x8a + XON. I agree that it’s ugly that they go to the screen; however, it’s pretty standard in ZMODEM-savvy terminals that I’ve used. I suppose it could send enough BS characters to the terminal to delete those characters? Or maybe just filter them out if they do arrive together (as I’d think they generally will). (UPDATE: “Problem” with sending BS characters is that it’ll only happen after the user accepts ZMODEM; the characters have to show on the terminal first because the user hasn’t yet confirmed that ZMODEM is what’s desired.)

There’s also a need to add a control to cancel an in-progress transfer. I’m still looking at the best way to handle that.

There’s also a need to add a control to cancel an in-progress transfer. I’m still looking at the best way to handle that.

Maybe this from the specs helps?

A ZFIN, ZABORT, or TIMEOUT terminates the session; a ZSKIP terminates the processing of
this file.

Potentially. The spec isn’t always the most helpful thing; for example, it mentions an Attn sequence that is sent after a ZSINIT. The spec seems to suggest that that’s how to get a sender to pause a data send, but apparently nothing actually uses Attn. Likewise with the ESC8 option: it’s not actually implemented in lrzsz, and since that’s the de facto reference implementation, ESC8 is unusable—which is a shame because it would nicely work around that problem with the termios IEXTEN flag.

I’m in a bit of a time crunch on other projects right now but hope to return to this next week.

If I may add my two cents regarding the shape of this for making it an xterm plugin: I would propose to not add an UI at all, but to dispatch things through an event, similar to node streams. Something like

term.on('transfer', (transfer) => {

  // accept or reject the transfer
  transfer.accept();

  // listen for progress
  transfer.on('progress', ...);

  // data chunks arrive
  transfer.on('data', ...);

  // transfer has finished
  transfer.once('end', ...);

  // cancel transfer
  transfer.abort();

});

This way a consumer can build its own UI on top of the plugin.

@mofux That’s how I would want this to work as well. The UI components that I’ve put into the demo are meant just to demonstrate the controls.

@FGasper Good job 👍

I'm going to add ZModem support for ttyd when your api is ready for use (https://github.com/tsl0922/ttyd/issues/37), thanks for your work.

https://www.npmjs.com/package/zmodem.js

I’ve made an ALPHA release of zmodem.js. From here I’ll look at the plugin interface for xterm.js, but anyone who wants to look at zmodem.js, please feel free to do so and let me know how it works for you.

The ZMODEM addon is now merged, FYI.

win7 system

$ npm run start-zmodem

> [email protected] start-zmodem E:\test\xterm\xterm.js
> node demo/zmodem/app

App listening to http://127.0.0.1:3100

why can't open File Explorer?