gsutil not using gcloug oauth details

The pip install and the gcloud install are separate from each other. If you are using the pip install, then gsutil config will set up your .boto file. But if you are using a gcloud install, then gcloud auth login should be used. It sounds like you are mixing the two configurations? Could you try the following steps:

1. Remove /Users/domster83/.boto
2. Run gcloud auth login
3. Run gsutil

I had completely uninstalled gsutil via pip and deleted the .boto file before installing the standard sdk.

On 2 Aug 2016, at 20:01, thobrla [email protected] wrote:

The pip install and the gcloud install are separate from each other. If you are using the pip install, then gsutil config will set up your .boto file. But if you are using a gcloud install, then gcloud auth login should be used. It sounds like you are mixing the two configurations? Could you try the following steps:

1. Remove /Users/domster83/.boto
2. Run gcloud auth login
3. Run gsutil

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

So does gsutil version -l display a config path with a .boto file in .config/gcloud now?

@thobrla ah! No, but you've helped me figure it out. Although this seems like a bug.
Currently that outputs

gsutil version: 4.19
checksum: 67da5fbdef140f1663fbb11e96bb9f90 (OK)
boto version: 2.39.0
python version: 2.7.12 (default, Jun 29 2016, 14:05:02) [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]
OS: Darwin 16.0.0
multiprocessing available: True
using cloud sdk: True
config path: /Users/domster83/.boto
gsutil path: /Users/domster83/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

And that made me think about me ENV. I had BOTO_CONFIG defined in my bash_profile as

export BOTO_CONFIG="$HOME/.boto"

Which gave me BOTO_CONFIG=/Users/domster83/.boto as an ENV VAR.
If I comment out this line and reload my terminal, without the ENV VAR I get

gsutil version: 4.19
checksum: 67da5fbdef140f1663fbb11e96bb9f90 (OK)
boto version: 2.39.0
python version: 2.7.12 (default, Jun 29 2016, 14:05:02) [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]
OS: Darwin 16.0.0
multiprocessing available: True
using cloud sdk: True
config path: /Users/domster83/.boto
gsutil path: /Users/domster83/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

BUT....gsutil now works. It shows the same config path, same file however. Neither of which are .config/gcloud

I do think there is bug here - when looking for the config path, we break when discovering the first valid file, but multiple files could be specified via the BOTO_PATH variable (which is how gcloud passes credentials to gsutil).

I believe I'm running into this config file issue too. I want to set use_magicfile in a version-controlled repo rather than a global config file, in order to serve webpages without .html extensions.

If I set BOTO_CONFIG or BOTO_PATH variables, I lose the credentials I've set with gcloud auth login. A gsutil rsync gives:

401 Anonymous users does not have storage.objects.list access to bucket foo.example.com.

Note that this happens even if BOTO_CONFIG points to /home/blah/.boto

Looking at the boto.pyami.config code, we only load multiple .boto files if:

1. BOTO_CONFIG is not set in your environment variables
2. BOTO_PATH is set in your environment variables

So in order to utilize multiple config files, you'll want to add the path of your repo-specific .boto config (which, from your description above, should only contain the config options I've shown below) to the set of paths in your BOTO_PATH variable. Note that they're loaded in order, so you should append that path, rather than prepend it.

[GSUtil]
use_magicfile = True

I tested this myself with gsutil v4.22 by overriding the default_project_id option in my second .boto file, running export BOTH_PATH=/path/to/second/boto/config, and then running gsutil ls (the gsutil binary included with the gcloud SDK, which utilizes a wrapper script to set the BOTO_PATH env var) to ensure that it listed buckets in the default project specified by my second config.

As for thobrla's comment, I'll get a fix out to ensure that when conditions 1 and 2 above are true, we show all readable config files in the output for gsutil version -l.

Thanks @houglum! I'll give that a try.

No such luck:

~$ head ~/.boto
# This file contains credentials and other configuration information needed
# by the boto library, used by gsutil. You can edit this file (e.g., to add
# credentials) but be careful not to mis-edit any of the variable names (like
# "gs_access_key_id") or remove important markers (like the "[Credentials]" and
# "[Boto]" section delimiters).
#
# This file was created by gsutil version 4.22 at 2017-02-26 10:29:13.
#
# You can create additional configuration files by running
# gsutil config [options] [-o <config-file>]
~$ cat /Users/foo/bar/.boto
[GSUtil]
use_magicfile = True
~$ unset BOTO_PATH
~$ gsutil ls
~$ echo $?
0
~$ export BOTO_PATH="/Users/foo/.boto"
~$ gsutil ls
ServiceException: 401 Anonymous users does not have storage.buckets.list access to project 9999999.
~$ export BOTO_PATH="/Users/foo/.boto:/Users/foo/bar/.boto"
~$ gsutil ls
ServiceException: 401 Anonymous users does not have storage.buckets.list access to project 9999999.
~$ export BOTO_PATH="/Users/foo/bar/.boto"
~$ gsutil ls
ServiceException: 401 Anonymous users does not have storage.buckets.list access to project 9999999.

Oh hey, I can totally reproduce this today. I think the shell I was using yesterday had some leftover alias for gsutil such that I was invoking a non-gcloud version. Apologies for the confusion :)

Anyway, upon realizing that, I figured out that this is a bug with the gsutil bootstrapping script that gcloud uses, /path-to-cloud-sdk/bin/bootstrapping/gsutil.py. Here's the culprit snippet:

    # We construct a BOTO_PATH that tacks the refresh token config
    # on the end.
    if boto_config:
      boto_path = os.pathsep.join([boto_config, gsutil_path])
    elif boto_path:
      boto_path = os.pathsep.join([boto_path, gsutil_path])
    else:
      path_parts = ['/etc/boto.cfg',
                    os.path.expanduser(os.path.join('~', '.boto')),
                    gsutil_path]
      boto_path = os.pathsep.join(path_parts)

      if 'BOTO_CONFIG' in os.environ:
        del os.environ['BOTO_CONFIG']
      os.environ['BOTO_PATH'] = boto_path

That last line should be shifted left one indentation level so that it gets applied regardless of which predicates above were true. I'll let our gcloud folks know so we can get this fixed.

(Also, WRT my previous comment, it looks like the gcloud script passes its configured default project as a top-level option, i.e. "-o 'GSUtil:default_project_id=...'", so one cannot overwrite the default project using a local .boto file.)

This fix as noted above is included in the most recent version of gcloud, and ce02232 fixes the config listing issue. Closing this.