Fabric: Fabric 2 alpha/beta feedback meta-ticket!

Small mistake in the URL, it should be http://bitprophet.org/blog/2017/04/17/fabric-2-alpha-beta/

Goes to show how well I remember my own website, eh? Thanks! I even ran a link checker on my post, but...not against this ticket ;)

I have a proposal for how fabric 1's roledefs system can be implemented in a way that I believe to be in line with fabric 2's philosophy. Each Collection of tasks can also have @group decorated functions in the same namespace that return populated Group objects against which tasks can be run. Namespaces would matter, so if fabfile.py contained a Collection called deploy who had @groups called web and db, one could use deploy.web.execute(mytask) or fab -G deploy.web mytask to execute mytask on each host in the web group. These decorated functions would be called lazily and memoized to prevent unnecessary API calls, should a host list lookup be a slow operation as implemented by the user.

Is this in line with the design philosophy for fabric 2? I'd love to take a swing at implementation if so.

That's a neat idea, @RedKrieg! I've been deferring my own brainstorm for how "best" to generate Group objects and/or how to refer to them on the CLI, but offhand that sounds like a reasonable way to go. I just crapped out far too many words in #1594 and included your idea there (+ a link). Let's continue discussion there, but tl;dr yes I'd love to see a PoC PR.

What is the best approach to run a command locally despite -H argument being present, for example if I want to combine local build and rsync to remote host into a single task?

@max-arnold Connection objects have a .local attribute that acts like .run on the local machine: http://docs.fabfile.org/en/v2/api/connection.html#fabric.connection.Connection.local

Ok, I guess an example is better than words:

@task
def build(ctx):
    # should always run locally
    ctx.local('uname -a')


@task
def deploy(ctx):
    build(ctx)
    # this one should run on remote host
    ctx.run('uname -a')

Combined task runs fine:

fab -H host deploy

Local task alone fails with AttributeError: No attribute or config key found for 'local':

fab build

Basically I want to have a task which does something locally, no matter how it was invoked (with or without -H).

From the other side, some commands are only intended to run remotely. If no host is present,ctx.run will attempt to run them locally, which can lead to unexpected consequences.

| User runs task | Command/task author wants it to be runned | How it should behave? |
|---------------------|------------------------------|----------------|
| locally | locally | run() behavior is fine (but will violate author's intention if run remotely) |
| locally | remotely | It should fail (or ask for host string like old fabric did) |
|locally | locally or remotely | run() behavior is fine |
| remotely | locally | It should always run locally, but Context has no local() method to ensure it |
| remotely | remotely | run() behavior is fine (but problems are expected during local invocation) |
| remotely | locally or remotely | run() behavior is fine |

@max-arnold that's exactly #98! which I haven't actually fully solved yet. Putting some modern thoughts in there as a comment...edit: this one

I've noticed what appears to be inconsistency with sudo.

Where connection is a remote server, authenticated as root.

Trying to expand tilde as another user other than root:

c.sudo("echo bar > ~/foo", user="builder")

This succeeds, but rather than writing /home/builder/foo, it instead wrote /root/foo.

On the other hand, if I just try to ls:

c.sudo("ls", user="builder")

I get Permission denied.

Something feels off.

My guess when Dustin reported the above out-of-band, was that this is a sudo (the command, not the method) specific wrinkle, since last I looked we are using -H and apparently it doesn't behave 100% like we're expecting.

Hi Jeff,
since i'm already working in python 3, right now i'm using fabric3 (the porting of fabric 1.x).

I was tempted to try the migration to fabric 2, but i immediately hit a show stopper. I make extensive usage of fabric.contrib functions, but migration doc says it no longer exists.

For sure i don't want to bloat the code reverting to their shell equivalent, and since, judging from the number of issues on contrib.* of fabric 1.x, it looks it is used a lot, i think its lack could be a serious hindrance for migrating for many others.
I found patchwork project that apparently contains the equivalent of fabric.contrib, but it's code is not updated since years.
Do you have any plan to port fabric.contrib to fabric 2?

Thanks,
Gabriele

@garu57 Yes, the plan right now is to use patchwork as basically "2.0's contrib". At the moment it's Fabric 1 based but that'll change after Fabric 2.0.0 comes out. I expect the most commonly used contrib bits to get ported over quickly.

Hi everyone,

Sometimes my program runs a remote command e.g.

source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./00-update-kernel.sh'

I see that scripts is executed like that:

033[0;32m[DONE]\033[0m'
+ return 0
+ alt_test_done_msg 'Prepare evironment'
+ echo -e '* Prepare evironment    \033[0;32m[DONE]\033[0m'
+ exit 0

and I should see also my debug output "?" according to my code :

class GenericFabric(object):
    def __init__(self, host, user, key_filename=None, port=22):
        connection_string = "{u}@{h}:{p}".format(u=user, h=host, p=port)
        self.connection = Connection(connection_string)
        self.key_filename = key_filename

    #@with_settings
    def generic_cmd(self, command_str, timeout, fabric_timeout, **kwargs):
        """
        Creating remote container from template

        @type command_str:     str
        @param command_str:    command for execute with VM
        @type timeout          int or float
        @param timeout         number of seconds for pause
        @type fabric_timeout   int or float
        @param fabric_timeout  number of seconds for timeout fabric run

        @rtype:                FabricResponse
        @return:               Return remote status of operation VM
        """
        if fabric_timeout > 0:
            command = self.connection.run(command_str.format(**kwargs),
                                          timeout=fabric_timeout,
                                          warn=True, echo=True)
        else:
            command = self.connection.run(command_str.format(**kwargs),
                                          warn=True, echo=True)
        print("?")
        if timeout > 0:
            sleep(timeout)
        return FabricResponse(command)

    def simple_generic_cmd(self, command_str, **kwargs):
        """
        @type command_str:     str
        @param command_str:    command for execute with VM

        @rtype:                FabricResponse
        @return:               Return remote status of operation VM
        """
        return self.generic_cmd(command_str, 0.1, 0, **kwargs)

But there are no ones "?". So I suggest there were hanging in the run().

Actually, my program is huge and there is multi-process programming. To catch hanging I'm executing remote commands with the following code:

class TestingSystemVM(GenericFabric):
#######
    @signal_alarm_down
    def run_rpm_test(self, command_test, package, type_of_test="base"):
        """

        """
        signal.signal(signal.SIGALRM, alarm_handler)
        signal.alarm(self.__timeout)
        returned_value = Queue()
        start_time = datetime.utcnow()
        directory, command_test = os.path.split(command_test)
        try:
            if command_test.endswith(".yml"):
                directory = directory.replace("/opt/QA", self.ansible.git_qa_repo)
                output = self.ansible.play_ansible(command_test,
                                                   package,
                                                   directory)
            else:
                vm_instance = (self.host,
                               self.user,
                               self.key_filename,
                               self.os_name,
                               self.platform,
                               self.arch)
                running_test = Process(target=separate_process_running_test,
                                       args=(vm_instance,
                                             returned_value,
                                             directory,
                                             command_test,
                                             package,
                                             type_of_test))
                running_test.start()
                running_test.join(self.__timeout)
                if running_test.is_alive():
                    running_test.terminate()
                    command_test_res = "FAIL: Timeout\n"
                    return command_test_res, work_time(start_time), 1
                elif returned_value.empty():
                    command_test_res = "FAIL: Problem while getting result\n"
                    return command_test_res, work_time(start_time), 1
                else:
                    output = returned_value.get()
        except TimeOut:
            command_test_res = "FAIL: Timeout\n"
            return command_test_res, work_time(start_time), 1
        if output.failed or (package.name in ("lve-utils", "lve-stats") and
                                     "FAIL" in output.stdout):
            res_output = "FAIL: " + output.stdout
        else:
            res_output = output.stdout
        return res_output, work_time(start_time), 0 if output.succeeded else 1

def separate_process_running_test(vm_instance, return_value_queue,
                                  directory, command_test, package,
                                  type_of_test="base"):
    """

    """
    sleep(0.5)

    signal.signal(signal.SIGTERM, kill_fabric_runner)
    signal.signal(signal.SIGINT, kill_fabric_runner)
    (host,
     user,
     key_filename,
     os_name,
     platform,
     arch) = vm_instance
    child_vm_instance = TestingSystemVM(host,
                                        user,
                                        key_filename,
                                        os_name,
                                        platform,
                                        arch,
                                        FakeAnsible())
    if command_test.endswith(".bats"):
        command_test = "/usr/bin/bats --tap " + command_test
    else:
        command_test = os.path.join("./", command_test)

    output = child_vm_instance.simple_generic_cmd(child_vm_instance._c_run_test,
                                                  envvars=child_vm_instance.env_vars,
                                                  exec_test=command_test,
                                                  dir=directory,
                                                  package=package.name,
                                                  pver=package.version,
                                                  prel=package.release,
                                                  type_test=type_of_test)
    print("!")
    return_value_queue.put(output)

When I use fabric2, some commands launched from child processes are hanging from time to time.
Upd. Actually, it was my mistake in the code. So, fabric2 doesn't contain hanging

Hello everybody,
I did some launch experiments and can share my experience.

Firstly, I ran command from my console

ssh [email protected] -t "source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'"

and that ended perfectly and fast with exit=0:

###
+ echo -e '* Prepare evironment    \033[0;32m[DONE]\033[0m'
* Prepare evironment    [DONE]
+ exit 0
Connection to 192.168.0.34 closed.

Good.
Then I executed another command:

ssh [email protected] "source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'"

That was finished so fast with exit=0 too:

###
+ echo -e '* Prepare evironment    \033[0;32m[DONE]\033[0m'
+ exit 0

But there isn't "Connection to 192.168.0.36 closed." I don't know it is important or not.

After that I recreated VMs and launched command by fabric2 (from IPython):

In [1]: from fabric import Connection
In [2]: Connection('[email protected]').run("source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'", pty=True)
....
+ echo -e '* Prepare evironment    \033[0;32m[DONE]\033[0m'
* Prepare evironment    [DONE]
+ exit 0
Out[2]: <Result cmd="source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'" exited=0>

Also, I ran with pty=False:

In [1]: from fabric import Connection

In [2]: Connection('[email protected]').run("source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'", pty=False)
+ echo -e '* Prepare evironment    \033[0;32m[DONE]\033[0m'
+ exit 0
Out[2]: <Result cmd="source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'" exited=0>

There wasn't "Connection to closed." on both tests. Again I don't know it is important or not.

Secondly, I decided that is not a good environment for my stand, because I need launch from child process. So, I recreated VMs again and ran the script with next code:

In [1]: import subprocess

In [2]: t = subprocess.Popen(""" ssh [email protected] -t "source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'" """, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True); r = t.communicate()

In the output of this running there was "Connection to 192.168.0.34 closed."

On the other hand

In [1]: import subprocess

In [2]: t = subprocess.Popen(""" ssh [email protected] "source /opt/centos/vars && su - -c 'cd /opt/QA/rpm_tests/p_lvemanager && ALT_TEST=base ALT_PACKAGE_NAME=lvemanager ALT_PACKAGE_VERSION=3.0 ALT_PACKAGE_RELEASE=11.el6.cloudlinux.21100.1.1505113712 ./01-prepare-environment.sh'" """, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True); r = t.communicate()

Both of them haven't hanged.

Tonight I'll launch my scripts with fabric2 on the process that was forked and I'll write.
Upd. Actually, it was my mistake in the code. So, fabric2 doesn't contain hanging.

Would be good to get use_sudo support for Transfer.put.

I guess the simplest workaround for now is just to do a put somewhere I do have permissions then a sudo mv?

@ned2 that's all use_sudo can really do in get/put anyways, eg it's how v1 does it. Not actually possible to upload a file "with sudo" as far as I'm aware! (Not without connecting as root, which is a bad idea and ideally is not even allowed.)

Given the attempt to have a somewhat cleaner API in v2, I'm almost more likely to implement a wrapper instead of bundling that behavior inside get/put themselves. Maybe just something like Transfer.sudo_put (which would call self.put). That way the "pure" put is kept minimal.

@bitprophet I would like to start off by thanking you for some pretty stable and useful software with Fabric1 and your approach to releasing Fabric2.

I've started to play around with v2 but I've hit an issue with Connection. I'm using my SSH config from Fabric 1, Fabric 1 would prompt me for a password when attempting to connect. Fabric 2 is not doing this and fails stating that Authentication has failed. I cannot see anywhere in the v2 docs that mentions the authentication mechanism for Connection. There is a mention of passwords with sudo but from what I can gather this is after the connection has been successful anyway..

My question is whether I've missed something in the docs or whether this is a missing feature or an issue.

The code:

@task
def testing(c):
    with Connection('MyHostname') as cxn:
        print("Connected")

        cxn.run('ls -l')

The error:

  File /lib/python3.4/site-packages/paramiko/auth_handler.py", line 223, in wait_for_response
    raise e 
paramiko.ssh_exception.AuthenticationException: Authentication failed.

My SSH config (Replaced values to post here):

Host MyHostname
  HostName replacedhostname.co.uk
  Port 22
  IdentityFile ~/.ssh/id_rsa
  User replaceduser

I'm on the latest v2 version: https://github.com/fabric/fabric/commit/fec3a22ee89900500ae731913fd33f9b56e89f46

Hopefully solving this will aid anyone else with the same issue.

Thank you for your time.

@Aiky30 It's probably not you, the auth stuff still needs work (in large part due to needing some in-progress Paramiko work so auth exceptions aren't horribly hard to interpret correctly.) I think in this case it may simply be that Fab 2 isn't interpreting IdentityFile - we grab host, user, port and a bunch of other settings but IdentityFile is still a TODO.

That's partly because we also don't deal with passphrases right now (see above - want to avoid a big Fabric 1 kludge where we have to make wild guesses if a given auth fail means password or passphrase is needed). So you'd probably just transition to a "I can't unlock this file" error after that. And both of those are because I use an ssh-agent most of the time - which would be my 1st suggestion for an immediate workaround.

That said, I think it's probably worth me hacking in both IdentityFile and explicit passphrase config support, because non-agent keys are likely the number 1 most common auth setup out there so lacking it probably means most alpha/beta users are left in the lurch. Will see if I can bang that out today.

Took 2.0a for a spin.

1) I'm also a bit lost on how to actually define and use hosts. I can run:

   fab -H user@host:22 some-task

But if I don't want to pass in host details (user, host, port) every time, I'm not sure how to configure hosts in fabfile.py (Connections or Group) and then just refer to them. If I understood https://github.com/fabric/fabric/issues/1591#issuecomment-296343613 correctly, it's just not supported at the moment (and tracked in https://github.com/fabric/fabric/issues/1594)?

2) Also, I hit issue where I have encrypted SSH key (at ~/.ssh/id_rsa) and I have ssh-add'd that to ssh-agent. It shows up if I list the keys via ssh-add -l. When I run some task, giving out proper username:

   fab -H musttu@host sometask

Things work ok. But if I use incorrect user, e.g. fab -H bad_user@host sometask, then I get:

Traceback (most recent call last): ... File "/home/maximus/.virtualenvs/testenv/lib/python3.6/site-packages/paramiko/pkey.py", line 326, in _read_private_key raise PasswordRequiredException('Private key file is encrypted') paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

But I assume that falls under https://github.com/paramiko/paramiko/issues/387 and is not really specific to fabric 2.0. But yeah, it does feel weird to get PasswordRequiredException: Private key file is encrypted error, when the ssh-agent already has the decrypted key. If I understood correctly, after invalid login, paramiko will fall back to using key directly, and without passphrase provided will throw this error.

3) Would be nice to allow type annotating task functions (for better auto-completion in IDE). Have you had time to consider https://github.com/pyinvoke/invoke/pull/458 ?

4) I like the idea of patchwork library. For fabric 1.x, there's https://github.com/sebastien/cuisine (that seems pretty dead) that contains lots of extra functions (haven't used it myself). Is the idea with patchwork to build something similar (so Chef/Ansible/SaltStack-style declarative functions, though with reduced scope)? I have always hated the multi-dir/multi-file YAML/DSL approach that other tools follow, and would just like to stay in python, to get more flexibility, less verbose syntax, IDE auto-completion, easy debugging, etc.

5) Just drop py 2.6 and 3.2-3.3 support for fabric 2.0. Shouldn't people just move on?

@tuukkamustonen - thanks for the feedback! Responses:

1. Roles are indeed covered under #1594 - and I think there's another ticket out there for the related issue of just having per-host config data, aside from how ssh_config support does so at its own level.

   Deciding how exactly to reconcile fabric-level config with ssh_config and then also with anything from Group or runtime data...isn't trivial. Though it definitely needs to happen soon; I just want to have a half-thought-out solution instead of a first-draft.

2. Again, yes, this is the Paramiko ticket you mentioned, and your guess is accurate, the bad-username situation will always eventually fail, and since Paramiko only tracks the last error it encountered, it happens to be that the last thing it tried was the on-disk, encrypted copy of the key. (If you moved that key elsewhere, for example, the last error would be something else.)

3. I just commented on that ticket, it may (or may not) duplicate existing tickets for the same overall feature. I've commented on those older ones and IIRC it's a "yea I think it'd be nice as long as it doesn't foul up Python 2." However it's not as high priority as most of the bigger missing features so it falls in the "needs a great, all boxes checked PR that I can just-merge" bucket :)

4. Yes, see #461 which is very old but still on my mind. These days the big question is how such a lib would be written to span both Invoke and Fabric; in many/most cases one can simply write generic Invoke tasks that don't "know about" any split local/remote contexts, which can be handed a Fabric connection context when one wants to run them against a remote system.

   Such "context agnostic" tasks would want to live in the 'invocations' library (or some other still Invoke-specific lib); but some (e.g. anything involving file transfer and not just shell commands) would need to be "Fabric-aware". Figuring out how to bridge that gap is the problem.

   Could go the route of "that lib only requires invoke, does not hard-require fabric, _but_ a subset of tasks will complain if you don't install fabric & hand them a Connection context." Could split up into two libs (with the Fabric-oriented one requiring the more generic Invoke one.) Etc.

5. As of recently, this is definitely in the cards, see paramiko/paramiko#1070 and/or pyinvoke/invoke#364. Invoke 1.0, Fabric 2.0 and Paramiko 3.0 (or, _maybe_, a 2.4/2.5/whatever) will all be Python 2.7 / 3.4 and up.

About (1):

I'm sure you know this already, but you may want to grab some ideas from Ansible's inventories. Or maybe not :).

About (4):

I think https://github.com/pyinvoke/invocations is way to opinionated for this. Invocations provides _conventions_ and may build on top of patchwork/cuisine-style package, but I wouldn't mix conventions (how to release or doctest) with utilities (how to copy file or alter permissions).

I don't see problem in putting utils for local-only, remote-only and local+remote operations into the same library (patchwork). What operation supports which mode can be tackled as documentation (as docstring, auto-generated from annotation, etc.).

This feels better than splitting logic into 2+ packages. Because what if you first add an operation to your "invoke-only" package (and there might be even "remote-only" operations), and then later add support for it to work also on remote servers? Would you move, copy or extend the code? In either case, it's more work, and user needs different import, etc. so it sounds a bit cumbersome.

What can I do to get this polished enough to go on PyPi?

@haydenflinner It's happening over the next week or two!! (Aim is to release before I get on my plane to PyCon, which is May 10.)

For example, please see the recently updated upgrade docs I was just working on this week: http://docs.fabfile.org/en/v2/upgrading.html

In fact, I may as well close this ticket now, since I'll be accepting actual real ones for 2.0.0 and above soon 👍

Do note, y'all can still comment here if you like. Note that I'm hoping to get at least a few chunks of feature work done before release, alongside all of the project management prep.

I looked at fabric v2 today and I'm sorry, I just can't bring myself to use it.

1. Where did roledefs go? Why can't I pass a role name on the command line anymore? Am I supposed to implement this myself?
2. Why can't I pass arguments to tasks on the command line anymore? Am I supposed to implement this myself as well?
3. Why does the -H parameter have to come before the task parameter? It should be the other way around. This is so awful.

Tasks do take arguments, but the syntax changed. See http://docs.pyinvoke.org/en/1.1/concepts/invoking-tasks.html#task-command-line-arguments

@dgarstang You may want to consider taking a more neutral or empathetic tone with folks who give you their free labor! Just sayin 😉

1. See #1594, which IIRC is linked in the upgrading doc
2. @ploxiln got your back here
3. (Partly) see #1772 which is merged to master and will be out soon; it adds ability to use @task(hosts=xxx).

(re: 3: In terms of giving --hosts and friends as if they were per-task arguments, that's something I might add sometime, have been torn on whether it's worth it. Similar to per-task --help, the more "magic" exclusions we add, the higher chance a user will try using that same name for their own task args and get confused when it breaks.)