Fabric: Parallel execution under python3

http://docs.fabfile.org/en/2.4/api/group.html

Yes, however, Threading group doesn't support configs, in particular:

config = Config(overrides={'sudo':{'password': 'changeme'}})

c = ThreadingGroup("[email protected]", "[email protected]", connect_kwargs={"password": "changeme"}, config=config)

c.run("whoami")

c.sudo("whoami") #Throws AttributeError: 'ThreadingGroup' object has no attribute 'sudo'

ThreadingGroup only allowed run command.
https://github.com/fabric/fabric/issues/1899#issue-377714688

You can rewrite and add the command support.

What do you think about the following patch to enable sudo in SerialGroup and ThreadingGroup ?
( applied and tested on fabric 2.4 )

--- fabric-orig/group.py    2018-10-30 14:35:22.000000000 +0100
+++ fabric-patched/group.py 2019-02-18 10:16:29.186000000 +0100
@@ -183,12 +183,28 @@
             raise GroupException(results)
         return results

+    def sudo(self, *args, **kwargs):
+        results = GroupResult()
+        excepted = False
+        for cxn in self:
+            try:
+                results[cxn] = cxn.sudo(*args, **kwargs)
+            except Exception as e:
+                results[cxn] = e
+                excepted = True
+        if excepted:
+            raise GroupException(results)
+        return results

 def thread_worker(cxn, queue, args, kwargs):
     result = cxn.run(*args, **kwargs)
     # TODO: namedtuple or attrs object?
     queue.put((cxn, result))

+def thread_worker_sudo(cxn, queue, args, kwargs):
+    result = cxn.sudo(*args, **kwargs)
+    # TODO: namedtuple or attrs object?
+    queue.put((cxn, result))

 class ThreadingGroup(Group):
     """
@@ -208,6 +224,49 @@
             )
             threads.append(thread)
         for thread in threads:
+            thread.start()
+        for thread in threads:
+            # TODO: configurable join timeout
+            # TODO: (in sudo's version) configurability around interactive
+            # prompting resulting in an exception instead, as in v1
+            thread.join()
+        # Get non-exception results from queue
+        while not queue.empty():
+            # TODO: io-sleep? shouldn't matter if all threads are now joined
+            cxn, result = queue.get(block=False)
+            # TODO: outstanding musings about how exactly aggregate results
+            # ought to ideally operate...heterogenous obj like this, multiple
+            # objs, ??
+            results[cxn] = result
+        # Get exceptions from the threads themselves.
+        # TODO: in a non-thread setup, this would differ, e.g.:
+        # - a queue if using multiprocessing
+        # - some other state-passing mechanism if using e.g. coroutines
+        # - ???
+        excepted = False
+        for thread in threads:
+            wrapper = thread.exception()
+            if wrapper is not None:
+                # Outer kwargs is Thread instantiation kwargs, inner is kwargs
+                # passed to thread target/body.
+                cxn = wrapper.kwargs["kwargs"]["cxn"]
+                results[cxn] = wrapper.value
+                excepted = True
+        if excepted:
+            raise GroupException(results)
+        return results
+
+    def sudo(self, *args, **kwargs):
+        results = GroupResult()
+        queue = Queue()
+        threads = []
+        for cxn in self:
+            my_kwargs = dict(cxn=cxn, queue=queue, args=args, kwargs=kwargs)
+            thread = ExceptionHandlingThread(
+                target=thread_worker_sudo, kwargs=my_kwargs
+            )
+            threads.append(thread)
+        for thread in threads:
             thread.start()
         for thread in threads:
             # TODO: configurable join timeout

That would honestly be awesome @akiuni

It would be good to get the execute() feature back into ThreadingGroup for Fabric2.

@akiuni is there a PR for this? https://github.com/fabric/fabric/issues/1912#issuecomment-464652728

@akiuni is there a PR for this? #1912 (comment)

Hello, I don't think so.. To be honest, I know how to check that and I don't know how to proceed to submit such a modification...

We need to get this merged into Fabric asap.

mm looking at the current code in v 2.5.0
https://github.com/fabric/fabric/blob/2.5.0/fabric/group.py#L127 I can see that this code is no longer current, and that the authors are thinking about a best way to implement this. Also with no PR they have no platform to discuss the changes the code introduce. I'm looking at this, but I'm not an expert python coder, but I will try to give it a shot as a PR

PR here: https://github.com/fabric/fabric/pull/2052

Hello, I'm not used to git developpemen (PR is a strange word for me), feel free to tell me if I can help

hey @akiuni
PR stands for Pull Request, and it means, that, if you want to make a change, you bring that change into the source code via a pull request, which can then be treated as a conversation, and reviewed, until it's approved. When it has been approved, the changes the PR introduces can be traced back to a commit like I have done here: #2052
A clever trick is to add ".patch" to the pull request (PR) like here: https://patch-diff.githubusercontent.com/raw/fabric/fabric/pull/2052.patch which will then give you a diff of the changes (as you have supplied in your earlier comment) but now, the benefit is, the conversation / review / single point of change as I mentioned above - thank You for being the first with a patch for this though, I was looking at it while I created the PR

Sudo is being added to Group right now! See #1999