Fabric: Remote Python execution hooks

Seán Hayes (Sean_Hayes) posted:

I created a fork on GitHub: https://github.com/SeanHayes/fabric

In fabric.decorators I added a decorator called runs_on_remote. It's sort of a poor man's remote execution.

Basically, when you wrap a function with it, if env.hosts isn't empty, the decorator will run 'fab func_name' on the host instead of allowing the function to execute locally. For example:

from fabric.api import *

env.hosts = []

def remote():
    env.hosts = ['your_host']

@runs_on_remote(remote_fabfile_path='/path/to/fabfile/dir/')
def foo():
    print local('ls /')

Running 'fab remote foo' locally will cause 'fab foo' to be run on 'your_host'.

on 2011-01-16 at 08:27pm EST

Any more progress on this. After using fabric for a while, I see this as one of the big missing pieces. Being able to write remote utility methods in pure python could certainly make it easier to write remote tasks.

No comments == no progress, sorry :) This is not a huge blocker for most users -- the majority are basically comfortable with shell scripting. So while this is _definitely_ near the top of the "nice to have" list, it's still in the "nice to have" list.

I expect it will get a closer look when we get into organizing version 2.x (sometime in the next few months if not sooner), since that's a great time to make "do X at Y location" a more abstract operation.

One simplification that may be worth consideration is simply making it possible to run a temporary python script on the remote system. I had even considered a manually way to do this by making a 'remote_utils.py' file or something that I put to the remote system on startup and then call that script with the name of the function I want to run within the module.

Assuming you mean some form of "copy local .py to remote system, execute it, then probably remove it again", yea that gets kicked around occasionally too. (See some of the comments above, even.)

But that doesn't require core support, so it would probably belong in one's local code library or in eg Patchwork.

I'd just like to point out that there's already a library, forked from 'sshuttle', that implements this:

Remote Exec lets you ship python code to a remote machine and run it there, all without installing anything other than the standard Python interpreter on the server.

It connects to the remote host using SSH, sends the python files you specify, compiles them on the server, and passes control to the specified main function.

Additionally, the client arranges for stdin/stdout on the server side to be connected to a network socket on the client side, so that you can communicate with the uploaded server binary as if you'd connected to it normally.

https://bitbucket.org/danderson/py-remoteexec

That said, the original code (and therefore, the fork too) is GPLv2 licensed, and I'm not sure if the authors are willing to re-license it.

@bitprophet I was thinking the same thing that this could be a local extension module. I just haven't found a good implementation of it that I could copy and refine. If anyone knows of one, I would love to see it and try it out.

@abierbaum Check the libraries listed in #461, I'd bet at least one of them may have some implementation of this.

The fantasy implementation would just have a @remote decorator on any python function and when called, the actual function is run on the remote target. Of the libs pointed out execnet seems like the best best, but also the most complex. (Just skipping through the documentation).

The various, "put code on the server and run it" approaches don't seem like they add a lot of value over just doing a "put" of the file, sudo running it, and deleting it. Not to mention this method allows you to run any code, not just Python code.

This is something that would be really helpful for me if it worked well. e.g. I would love to use something like _psutils_ to manipulate processes rather than the ps > grep > awk > kill methods I have to use now.

If somebody were to want to implement this, what method would you be most likely to pull into the contrib?

Why not just

print run("""python -c 'import re
m = re.search("1", "m1m")
print m'""")

[localhost] run: python -c 'import re
m = re.search("1", "sfsdf1231231231dfads")
print m'
[localhost] out: <_sre.SRE_Match object at 0x1086cd8b8>

I actually added the workaround (put, run, remove) to Patchwork a short while back, https://github.com/fabric/patchwork/blob/master/patchwork/commands.py#L11 - it's a first draft so could use some love sometime I'm sure.

This does the job if your functions are self contained, thus you need to import in the function body.

def rpc(f):
    @wraps(f)
    def wrapper(*l, **kw):
        c = 'python -c "import marshal,pickle,types;types.FunctionType(marshal.loads(%r),globals(),%r)(*pickle.loads(%r),**pickle.loads(%r))"'
        c = c % (marshal.dumps(f.func_code),f.func_name,pickle.dumps(l),pickle.dumps(kw))
        run(c)
    return wrapper

Example:

@task
@rpc
def f(): print 1+1

Works ok but fabric fails when serializing long functions in its bash escaping mecanism.
The design of fabric is totally flawed because its api, run(string) implies escaping.
The correct api of run should be run(cmd=[]) where cmd is a list.
Fabric should not escape but serialize and execv must be used instead of exec /bin/sh -c string.
run(string) could be supported but it's usage should be discouraged.

If ssh does not support execv char *const argv[], then a small helper should be used on the remote side, we could do this helper in python or in perl. This helper could also support python rpc like the example above

So I tried what @antonylesuisse suggested above but ran in to some issues. For whatever reason I couldn't import modules directly, and even if I did import them on the remote side with an exec as soon as I tried to access them the interpreter would segfault. It looks like that may be due to me running different versions of python, with the version of python actually running fabric being in a miniconda environment.

Here are a few functions that put the source code, executes it after pickling/unpickling the arguments, then gets the pickled response back over the wire directly from the output file. I think this should avoid any shell escaping issues.. of course the responses still have to be pickleable/unpickleable but as long as they are it seems to work ok.

An example script is attached.
rpc_example.py.zip
.
This still isn't interactive, but at least you can run non-interactive code with it. Has anyone dived in to execnet or Pushy integration?

@raddessi It seems that Chopsticks might be the library you are looking for.

i used fabric like chopsticks. and i think it's much simple.
在 Fri, 18 Aug 2017 14:23:20 +0000 (UTC)
Fabien Meghazi notifications@github.com 已寫入：

@raddessi It seems that
Chopsticks might be the
library you are looking for.

https://mitogen.readthedocs.io/en/latest/ is a newcomer on the scene here and while I don't remember anymore how exactly the others listed above work, this one literally bootstraps a remote, in-memory Python interpreter (does not require Python installed remotely; uses SSH - but not Paramiko, sob) and pushes code into it for execution.

It can apparently also force subprocesses to use the local process for SSH which lets it do things like SFTP-over-sudo (I don't quite follow how that works but haven't read the source yet.)

However it's not Python 3 compat and claims not to have plans there, so could be a blocker as Fabric 2 / Invoke are explicitly 2+3 compatible.

@bitprophet "Mitogen is a Python library for writing distributed self-replicating programs." Does this really fit. Why not Ansible? Have you looked at Ansible? I am curious what you think. We switched from fabric to salt, but I am not 100% happy with it. I guess I would use Ansible the next time.

I examined both Salt and Ansible a while back and didn't really like either. That said, I consider Fabric to be mostly orthogonal to 'full' config management systems like those, so it's not really a fair comparison. (You can get, say, 75-85% of the way there with Fabric, many folks do, and I'd like to improve that where we can - but Fabric will always be more of a bottom-up building-block type library whereas most CM tools like Salt/Ansible/Chef/Puppet are more like top-down frameworks.)

@bitprophet thank you for your reply.

I am not happy with salt, I am not happy with fabric.

Fabric is sometimes like "remote shell execution". I switched from shell scripts to python several years ago since I love tracebacks and exceptions. The shell like execution (on error write to stdout/stderr and continue with the next line) is not acceptable in my context. I prefer uncaught exceptions to "on error continue" especially in production systems.

Fabric felt like a step back.

We used it some times, but switched to salt.

But Salt is to complicated.

In my team most developers make a big bow around salt.

That was different with fabric.

I am sitting between the chairs.

I have not tried Ansible up to now. But it seems simpler and that is important for me.

I hope the following words don't hurt you . ... I think I won't switch back to fabric.

To me, I think the fabric is a remote execution tool. SaltStack/Ansible/Puppet/Chef are a kind of state management platform. The goal is different.
Fabric provides a straightforward way to manage the remote host. There are some libraries offer a batch of code snippet for Fabric. It's a direction of "State Management."
If we want Fabric to resolve a more sophisticated remote execution problem, it asks users to write their components to handle the connection or other staff. Mitogen suits this case.

The shell like execution (on error write to stdout/stderr and continue with the next line) is not acceptable in my context. I prefer uncaught exceptions to "on error continue" especially in production systems.

To be fair, Fabric should not be continuing on error, so if you're seeing that, something else is going on.

That said, it's still not as clean exception-wise as we would like (basically all errors result in SystemExit) and version 2 is being built to be a lot more exception-friendly (with SystemExit only being raised at the very outer edges of the program.)

Hi @bitprophet, just to note Python 3 support for Mitogen is a more explicit goal these days. It shouldn't require more than a couple more weeks to hit, and it's necessary anyway to support Ansible on upcoming versions of Fedora.

Hi @bitprophet, another note to say Python 3 has at long (loooong) last reached master, and the result is going to be on PyPI as the first stable series release by tomorrow evening. The 'fakessh' support is presently broken and not in scope, but I'm dying to get stuck to fixing it, and generalizing it for TCP/UDP/pipe forwarding -- hoping after getting this Python 3 doldrums behind me, energy will return and lots of new stuff will happen real quick now :)

Awesome, glad to hear it @dw! I don't know when I'll have time to look at Mitogen integration on my end but super stoked that the interpreter-version barrier is gone now 👍

@bitprophet any updates on this ?

@bitprophet this feature is very useful for our purpose (remote debugging), would love to help out in the implementation in any way possible.

i just found out about mitogen, and i think it would be an excellent backend for fabric.

in fact, I think it could advantageously replace a lot of fabric code (and possibly most of pyinvoke as well). it removes the "special case" distinction between fabric/invoke (ssh/local) and merges all of this in a magic "context" that also works across backends like "sudo" more transparently than fabric (IMHO).

right now I really feel the pain of this feature missing in fabric. i'm looking at implementing more and more complex business logic on the remote end. right now this involves transfering files back and forth with sftp to make sure things are correctly setup, and it's kind of clunky. it would be so much better to just do that remotely: just open the file, look for content, search/replace and so on becomes much more intuitive.

it also makes it possible to write Python code that is reusable outside of fabric. right now I can't really do that, unless i squint really hard, because fundamentally, the python code only runs locally. so i need to copy files (or processes?!) back and forth between hosts to do what i need. it's far from ideal...

right now I'm at a junction: either I keep going the fabric route, or I just port all my code to mitogen and make a small executor wrapper around it.. i don't use the fabfile feature enough to warrant the complexity of fabric, while mitogen gives me much more powerful features for my main use case (which is bootstrapping and recovering servers from heterogeneous environments)...

but I understand that there are other use cases for fabric and invoke as well, of course. I just feel it would be much more natural to write native python code, even for existing functionality in Fabric (like code execution), because it would make error handling, pipe control, and existing reflexes work more or less normally...