Gsutil: support for aws_session_token

Created on 12 Jun 2018 · 6Comments · Source: GoogleCloudPlatform/gsutil

Hi, I'm using gsutil v. 4.3.1 and there isn't any apparent support for the AWS_SESSION_TOKEN, which is a required configuration setting when working with AWS pre-signed URLs.

Use case: I'm trying to upload an object from a GCP bucket (in my account) to S3. I have been provided with an AWS pre-signed URL and corresponding credentials that include

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

I have tested with these variables in the .boto file, and even in different tests as environment variables, and the response I get is "403 AccessDenied" when running my cp command that is of the form

gsutil cp gs://mybucket/obj s3://s3bucket/obj

Things work fine when not using a pre-signed URL.
After lots of testing, I wanted to verify that the AWS_SESSION_TOKEN is not supported, and to request for such support.

Thanks,
-Nathan

Feature Request dependency

Source

nathankw

👍1

Most helpful comment

+1 - It would be extremely useful to be able to sync from S3 (with an STS Assumed Role) to GCS using gsutil.

ggiill on 15 May 2020

👍4

All 6 comments

Hi,
Just wanted to check back. Is this something that could make it to the feature list?
Let me know if you need anymore details.

nathankw on 8 Aug 2018

It looks like there are two asks here, neither of which gsutil supports at the moment:

Uploading to a GCS Signed URL or an S3 Pre-Signed URL
Adding support for AWS_SESSION_TOKEN (I also mentioned this in the Stackoverflow post at [1], which mentions this is unlikely because it's not supported in the Boto library).

[1] https://stackoverflow.com/questions/51353850/configure-gsutil-boto-file-to-use-aws-sts-assume-role

houglum on 10 Aug 2018

The last answer is from Aug 2018. Is there any update since then?

iampat on 21 May 2019

These features would rely on us extending or decoupling from the old deprecated boto library. Currently we have it on our roadmap to explore our relationship with boto and possibly decoupling from it, along with swapping oath2client with google-auth and swapping httplib2 with requests library.

Likely this would be explored after we figure out and implement a path away from boto. These dependency changes are currently goals for us, although the boto change is a slightly lower priority than the oauth2client -> google-auth change.

Likewise, gsutil is transitioning maintainers, which may mean reprioritizing goals depending on the new owner's priorities and roadmap.

In short, we will definitely keep you updated on this! We have not forgotten you. :slightly_smiling_face: