Fabric: How to read results in ThreadingGroup during partial failures?

Hey @shadyabhi ,

Not sure if you are trying to catch a specific exception or if you just want to catch all exceptions happening in separate threads. I think the ThreadingGroup object already passes around exception objects using queues and raises if any are encountered during execution. I was able to see stack traces using the following code snippet:

def run_data(c):
    c.run('date', hide='both')

@task
def thread_exception_example(c):
    hosts = ['localhost', 'fakehost', 'localhost']

    # python list comprehension
    {c: run_data(c) for c in Group(*hosts)}

On the command-line I ran: fab thread-exception-example

Hi @bossjones,

Thanks for replying. However, I'm not using the fab command but rather executing all this by using fabric as a library. So, an answer is still pending and would be appreciated.

In other words, fab is handling what I want to handle in my code myself. There are reasons I don't want to use fab command directly as I would like to have more flexibility in how I invoke tasks.

Thanks

Hi everyone,

I'm sorry, I overlooked the details. Checked out the fabric code and got details.

The argument with GroupException is a dictionary with the key as fabric.connection.Connection object and value as fabric.runners.Result object.

except GroupException as e:
         for c, r in e.result.items():
             print "Connection: {}, Result: {}".format(c, r)

Hello,

In addition of shadyabhi's comment, here is a way I use to deal with GroupException.
I hope this will help !!

import logging, socket, paramiko.ssh_exception
from fabric import Connection, Config, ThreadingGroup, exceptions, runners

...

g = ThreadingGroup.from_connections(c)
try:
    result = g.run('hostname' )  # (just a basic command)
except exceptions.GroupException as e:
    i = 0
    for c, r in e.result.items():
        print("Host[{}] = [{}]".format(i,c.host) )
        if isinstance(r,runners.Result) :
            print("ok, fine")
        elif isinstance(r,socket.gaierror) :
            print("Network error")
        elif isinstance(r,paramiko.ssh_exception.AuthenticationException) :
            print("Auth failed")
        else:
            print("something else")
        i+=1

(got if from ActivCloud support )

Thanks @akiuni for your example! I'm new to Python/Fabric and that helped me understand how to make requests using ThreadedGroup, and collect the results when there was an exception. Here's my expanded version of your example. Much appreciated! For what it's worth, I posted this to Stackoverflow as well.

# requires fabric 2.x - run 'pip install fabric' to install it
import logging, socket, paramiko.ssh_exception
from fabric import Connection, Config, SerialGroup, ThreadingGroup, exceptions, runners
from fabric.exceptions import GroupException


# Note: You need to supply your own valid servers here to ssh to of course!
def main():
    testHosts("All should succeed", "validServer1,validServer2,validServer3")
    testHosts("Some should fail", "validServer1,validServer2,BADSERVER1,validServer3,BADSERVER2")

def testHosts(message, hostsAsString):
    print("")
    print(message)

    # Get list of hosts from somewhere, and convert them to connections
    hosts = hostsAsString.split(",")
    servers = [Connection(host=host) for host in hosts]

    # Create a thread group to run requests in parallel
    g = ThreadingGroup.from_connections(servers)
    try:
        command = "df -h / | tail -n1 | awk '{print $5}'"
        results = g.run(command, hide=True)
        for r in results:
            connection = results[r]
            print("{}".format(r.host) )
            print("  SUCCESS, " + connection.stdout.strip())
    except GroupException as e:
        # If an exception occurred, at least one request failed. 
        # Iterate through results here
        for c, r in e.result.items():
            print("{}".format(c.host) )
            if isinstance(r,runners.Result) :
                print("  SUCCESS, " + r.stdout.strip())
            elif isinstance(r,socket.gaierror) :
                print("  FAILED,  Network error")
            elif isinstance(r,paramiko.ssh_exception.AuthenticationException) :
                print("  FAILED,  Auth failed")
            else:
                print("  FAILED,  Something other reason")

main()