Gsutil: 'ForkAwareLocal' object has no attribute 'connection' for multithreaded cp

Created on 9 Sep 2020 · 7Comments · Source: GoogleCloudPlatform/gsutil

When running gsutil -m cp -r gs://example/ ./ on a fairly large folder on macOS with the system Python 3.7 (to prevent the issues from #961), I see many instances of the following error:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 788, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2348, in run
    cls = copy.copy(class_map[caller_id])
  File "<string>", line 2, in __getitem__
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 792, in _callmethod
    self._connect()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 779, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused

As a result the command seems to hang after starting a first batch of downloads.

Source

djc

👍24

Most helpful comment

The -o "GSUtil:parallel_process_count=1" workaround works for me on Big Sur (macOS 11.2) but it's frustrating to me that this continues to persist as an issue on the Mac platform.

dweekly on 10 Feb 2021

👍9

All 7 comments

I have the exactly same problem in OSX 10.15.6, gcloud {308.0.0, 297.0.1}, gsutil {4.53, 4.51}, python 3.7.8.

peinan on 14 Sep 2020

Same in OSX 10.15.7, gcloud 303.0.0, gsutil 4.52, python 3.6.5

waltaskew on 28 Oct 2020

Same in gcloud 317.0.0, python 3.7.7

mobuchowski on 7 Nov 2020

Thanks for reporting this! I was able to reproduce this bug as well and found that disabling multiprocessing helped. You can do this by setting parallel_process_count=1 in the GSUtil section of your boto config file, or by adding the following flag to your command: -o "GSUtil:parallel_process_count=1". Though this disables multiprocessing, multithreading should still be enabled, so you'll still be able to parallelize your transfers.

I'm guessing this issue is related to more general issues with multiprocessing on MacOS that PR #1107 left us vulnerable to (https://github.com/GoogleCloudPlatform/gsutil/pull/1107#issuecomment-698555319). If you're still having this issue after disabling multiprocessing please let us know!

rrauber on 9 Nov 2020

👍4

Thank you for providing a workaround, it works for me.

This tool is a joke though. Not a single version since 297 works out of the box without some kind of patching.

alamothe on 1 Dec 2020

👍2

The -o "GSUtil:parallel_process_count=1" workaround works for me on Big Sur (macOS 11.2) but it's frustrating to me that this continues to persist as an issue on the Mac platform.

dweekly on 10 Feb 2021

👍9

Doesnt work for me, I get CommandException: Destination URL must name a directory, bucket, or bucket subdirectory for the multiple source form of the cp command. please fix your joke product