Fabric: Strange input handling on Windows in Fab2

Created on 14 Sep 2018  ·  11Comments  ·  Source: fabric/fabric

Fabric 1 handles manual input correctly, but Fabric 2 behaves strangely, with pty or without.

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

# contents of test_input.sh
# #!/bin/bash
# read -p "enter value: " var
# echo "you entered $var"

# Fabric 1 handled input correctly
>>> run("~/test_input.sh")
[riskapp.comm.equabank.loc] run: ~/test_input.sh
[riskapp.comm.equabank.loc] Login password for 'user':
[riskapp.comm.equabank.loc] out: enter value: foo # typed "foo" ENTER
[riskapp.comm.equabank.loc] out: you entered foo
[riskapp.comm.equabank.loc] out:

'enter value: foo\r\nyou entered foo'

# Fabric 2 without pty doesn't show prompt and behaves strange, must hit enter twice to finish command
>>> c = Connection(SERVER, connect_kwargs={"password": PASSWORD})
>>> c.run("~/test_input.sh")
bar # typed "bar" ENTER
bar # only "b" appeared, the rest after another ENTER


you entered bar
<Result cmd='~/test_input.sh' exited=0>

# Fabric 2 with pty shows prompt but input behavior is still strange, still must hit enter twice
>>> c.run("~/test_input.sh", pty=True)
enter value: baz  # typed "baz" ENTER
b  # another ENTER
az

you entered baz
<Result cmd='~/test_input.sh' exited=0>

# Fabric 2 automatic response works ok with pty
>>> c.run("~/test_input.sh", pty=True, watchers=[Responder(pattern="enter value", response="foo\n")])
enter value: foo
you entered foo
<Result cmd='~/test_input.sh' exited=0>

Tested on Windows 7 64b in cmd, Python 3.7 64b, with Fabric versions:
Fabric3 1.14.post1, Fabric 2.3.1, Paramiko 2.4.1, Invoke 1.1.1

Bug Needs investigation Nonstandard platforms
Source
picture kuba-baku
👍1

Most helpful comment

Here is a breaking change of invoke from 1.2 to 1.3 verion.
https://github.com/pyinvoke/invoke/issues/654

picture baconYao on 12 Aug 2019
👍4

All 11 comments

Thanks for the report. I'm unable to perform diagnosis on Windows systems myself but hopefully somebody else may be able to reproduce. I can say we've definitely squashed a number of terminal & encoding related issues related to Windows in the past so things _should_ be pretty stable there, but you may have found another one.

Another question triggered by another ticket - has this always happened or did it only start recently? Another user reported some weird behavior (but different than this reported symptom) after a recent Windows update.

bitprophet picture bitprophet on 14 Sep 2018

I can't say, whether it started recently or not, because I have not tested it before.
Now I tried it on Windows 10 in Python 3.5.3 and no change.
I tried it several times and once it ended up on this error:

>>> c.run("~/test_input.sh")
asv  # i typed "asv" ENTER
asv  # another ENTER
you entered asv

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<decorator-gen-3>", line 2, in run
  File "C:\Python35\lib\site-packages\fabric2\connection.py", line 30, in opens
    return method(self, *args, **kwargs)
  File "C:\Python35\lib\site-packages\fabric2\connection.py", line 702, in run
    return self._run(self._remote_runner(), command, **kwargs)
  File "C:\Python35\lib\site-packages\invoke\context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "C:\Python35\lib\site-packages\invoke\runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "C:\Python35\lib\site-packages\invoke\runners.py", line 365, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (OSError):

Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp852'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp852'>},
 'target': <bound method Runner.handle_stdin of <fabric2.runners.Remote object at 0x000001BF0189C908>>}

Traceback (most recent call last):
  File "C:\Python35\lib\site-packages\invoke\util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()
  File "C:\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python35\lib\site-packages\invoke\runners.py", line 648, in handle_stdin
    self.write_proc_stdin(data)
  File "C:\Python35\lib\site-packages\invoke\runners.py", line 784, in write_proc_stdin
    self._write_proc_stdin(data.encode(self.encoding))
  File "C:\Python35\lib\site-packages\fabric2\runners.py", line 69, in _write_proc_stdin
    return self.channel.sendall(data)
  File "C:\Python35\lib\site-packages\paramiko\channel.py", line 846, in sendall
    sent = self.send(s)
  File "C:\Python35\lib\site-packages\paramiko\channel.py", line 801, in send
    return self._send(s, m)
  File "C:\Python35\lib\site-packages\paramiko\channel.py", line 1180, in _send
    raise socket.error("Socket is closed")

OSError: Socket is closed
kuba-baku picture kuba-baku on 26 Sep 2018

Hi,

Any fix on it. We have the same issue as above shown.

2019-08-07 11:31:30,590 - vm-1.0-GA-x86_64-minimal-template-1.0-Update-917-1565202528 - INFO - exec [cd /root || exit $?; mkdir -m 777 -p /mnt/shared-1565202528]
testlist: {'tests': [{'test_catagory': 'upgrade', 'test_json': 'runlists/upgrade.json', 'priority': 'P0'}]}
Traceback (most recent call last):
File "harness.py", line 395, in
main()
File "harness.py", line 392, in main
sys.exit(harness.start_run())
File "harness.py", line 59, in start_run
result_map = self.run()
File "harness.py", line 84, in run
result_map[test['test_catagory']] = self.run_tests(test)
File "harness.py", line 127, in run_tests
self.mount_storage_on_vm(vm_object)
File "harness.py", line 241, in mount_storage_on_vm
"mkdir -m 777 -p %s" % (self.log_location)).return_code == 0:
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/photontools/runner/guest.py", line 36, in run_with_cd
return conn.run("cd {} || exit $?; {}".format(quote(dir), command), *args)
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/photontools/runner/vm.py", line 242, in _pre_run
return _orig_run(command, *kwargs)
File "", line 2, in run
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/fabric/connection.py", line 30, in opens
return method(self, args, *kwargs)
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/fabric/connection.py", line 702, in run
return self._run(self._remote_runner(), command, *kwargs)
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/context.py", line 101, in _run
return runner.run(command, *kwargs)
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/runners.py", line 291, in run
return self._run_body(command, **kwargs)
File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/runners.py", line 399, in _run_body
raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (NotImplementedError):

Thread args: {'kwargs': {'echo': None,
'input_': <_io.TextIOWrapper name='' mode='r' encoding='UTF-8'>,
'output': <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>},
'target': >}

Traceback (most recent call last):

File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/util.py", line 233, in run
super(ExceptionHandlingThread, self).run()

File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(self._args, *self._kwargs)

File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/runners.py", line 706, in handle_stdin
self.close_proc_stdin()

File "/root/workspace/upgrade-test-CI-in-harness/update-basic-in-harness/.venv/lib/python3.6/site-packages/invoke/runners.py", line 939, in close_proc_stdin
raise NotImplementedError

NotImplementedError

michellew-vmware picture michellew-vmware on 7 Aug 2019
👍2

Same problem occurred at 2019/08/07

def ssh_login_target_nas():
    """ Use ssh to login the target NAS 

        return 
            type: <class 'fabric.connection.Connection'>
    """
    # Set up fabric's login information
    host = SSH_ACCOUNT + "@" + SELF_NAS_IP_ADDRESS              
    # Connect to your nas
    c = Connection(host = host, connect_kwargs={"password": SSH_ACCOUNT_PASSWORD})
    try:
        hostname = c.run('hostname').stdout.strip()
        logger.debug("Connect to NAS: " + hostname)
    except Exception, error:
        logger.error(error)
        raise SystemExit
    return c

I could get hostname from stdout. But I still got the exception like below.

05:57:13 [ERROR] - 2019-08-08 05:58:17,108 - auto_update: ssh_login_target_nas 54   - 
05:57:13 Saw 1 exceptions within threads (NotImplementedError):
05:57:13 
05:57:13 
05:57:13 Thread args: {'kwargs': {'echo': None,
05:57:13             'input_': <open file '<stdin>', mode 'r' at 0x7fbf8f39b0c0>,
05:57:13             'output': <open file '<stdout>', mode 'w' at 0x7fbf8f39b150>},
05:57:13  'target': <bound method Remote.handle_stdin of <fabric.runners.Remote object at 0x7fbf8c6d3e10>>}
05:57:13 
05:57:13 Traceback (most recent call last):
05:57:13 
05:57:13   File "/home/vagrant/workspace/qsirch-v4.1-ui-autotester-chrome/qpkg-update/uenv/local/lib/python2.7/site-packages/invoke/util.py", line 233, in run
05:57:13     super(ExceptionHandlingThread, self).run()
05:57:13 
05:57:13   File "/usr/lib/python2.7/threading.py", line 754, in run
05:57:13     self.__target(*self.__args, **self.__kwargs)
05:57:13 
05:57:13   File "/home/vagrant/workspace/qsirch-v4.1-ui-autotester-chrome/qpkg-update/uenv/local/lib/python2.7/site-packages/invoke/runners.py", line 706, in handle_stdin
05:57:13     self.close_proc_stdin()
05:57:13 
05:57:13   File "/home/vagrant/workspace/qsirch-v4.1-ui-autotester-chrome/qpkg-update/uenv/local/lib/python2.7/site-packages/invoke/runners.py", line 939, in close_proc_stdin
05:57:13     raise NotImplementedError
05:57:13 
05:57:13 NotImplementedError

My content of requirements.txt like below:

requests==2.11.1
fabric==2.4.0
xmltodict

I found that I would install different version of packages after executing pip install -r requirements.txt since 2019/08/07.

2019/08/06. It could execute properly

Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 enum34-1.1.6 fabric-2.4.0 invoke-1.2.0 ipaddress-1.0.22 paramiko-2.6.0 pycparser-2.19 pynacl-1.3.0 requests-2.11.1 six-1.12.0 xmltodict-0.12.0

2019/08/07. Failed

Successfully installed asn1crypto-0.24.0 bcrypt-3.1.7 cffi-1.12.3 cryptography-2.7 enum34-1.1.6 fabric-2.4.0 invoke-1.3.0 ipaddress-1.0.22 paramiko-2.6.0 pycparser-2.19 pynacl-1.3.0 requests-2.11.1 six-1.12.0 xmltodict-0.12.0

The version of invoke package is different.

baconYao picture baconYao on 12 Aug 2019
👍1

Here is a breaking change of invoke from 1.2 to 1.3 verion.
https://github.com/pyinvoke/invoke/issues/654

baconYao picture baconYao on 12 Aug 2019
👍4

great find @baconYao, this was blocking me as well. downgrading invoke for now to 1.2 has gotten me past this issue. Much appreciated.

scott-estes picture scott-estes on 20 Aug 2019
👍1

Same here, on any key press in continuously running threaded task, I get socket error.

Saw 1 exceptions within threads (OSError):

Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>},
 'target': <bound method Runner.handle_stdin of <fabric.runners.Remote object at 0x7fcad97aeed0>>}

Traceback (most recent call last):

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/invoke/util.py", line 233, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib64/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/invoke/runners.py", line 694, in handle_stdin
    self.write_proc_stdin(data)

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/invoke/runners.py", line 832, in write_proc_stdin
    self._write_proc_stdin(data.encode(self.encoding))

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/fabric/runners.py", line 67, in _write_proc_stdin
    return self.channel.sendall(data)

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/paramiko/channel.py", line 846, in sendall
    sent = self.send(s)

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/paramiko/channel.py", line 801, in send
    return self._send(s, m)

  File "/home/ds/.local/share/virtualenvs/port_error_histogram-BRcAd8l-/lib/python3.7/site-packages/paramiko/channel.py", line 1198, in _send
    raise socket.error("Socket is closed")

OSError: Socket is closed
dhruvinsh picture dhruvinsh on 22 Aug 2019

I had the same problem with 2.4.0. I was using fab inside a celery task. For some reason upgrading to 2.5.0 solved it.

Gabriel-Desharnais picture Gabriel-Desharnais on 26 Aug 2019

This issue could also be connected to Windows or WSL environments. We had the same stacktrace using latest Ubuntu WSL on latest Windows 10, and solved it by running our tasks on macOS.
Upgrade or downgrading fabric and or invoke as mentioned above did not work for us.

The strange thing is, that most tasks also worked on WSL, only a very long task (many run() commands) did fail with this exception non-deterministically, i.e., after some uncertain number of commands run.

JulianEberius picture JulianEberius on 18 Sep 2019

I am so glad I was able to find this.

L1ghtman2k picture L1ghtman2k on 18 Sep 2019

On further investigation, it could also be connected to the Responder class. We used a responder to answer "yes" on a couple of quick consecutive .run() commands, and this would fail on WSL. Removing the Responder objects and calling "yes | the-command" instead worked on WSL as well.

JulianEberius picture JulianEberius on 19 Sep 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jamesob picture jamesob  ·  3Comments
peteruhnak picture peteruhnak  ·  6Comments
Grazfather picture Grazfather  ·  4Comments
shadyabhi picture shadyabhi  ·  5Comments
neemxyang picture neemxyang  ·  6Comments