mc doesn't work with GCS buckets containing underscores in the name

Created on 5 Apr 2016 · 6Comments · Source: minio/mc

The validation mc does on bucket names restricts them to a DNS-able subset of characters. However Google Cloud Storage seems to allow other characters. The leads to the situation where mc can't perform operations on some GCS buckets.

Is there a way to relax that restriction on a per-alias basis? My use-case here is to be able to transfer data between GCS and S3, and I can't currently do that for all the buckets I'm working with. See below for the error I'm getting.

$ ./mc config host add gcs https://storage.googleapis.com GOOGXXXXXXXXXXXXXXXX eb4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX S3v2
$ ./mc ls gcs
[2016-03-21 15:30:02 GMT]     0B dw_export/
$ ./mc ls gcs/dw_export/
mc: <ERROR> Unable to stat ‘gcs/dw_export/’. Bucket name contains invalid characters.

wontfix

Source

d5ve

👍2

Most helpful comment

$ ./mc config host add gcs https://storage.googleapis.com GOOGXXXXXXXXXXXXXXXX eb4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX S3v2
$ ./mc ls gcs
[2016-03-21 15:30:02 GMT] 0B dw_export/
$ ./mc ls gcs/dw_export/
mc: Unable to stat ‘gcs/dw_export/’. Bucket name contains invalid characters.

Currently yes buckets are restricted based on S3 bucket naming requirements i.e following bucket restrictions from here http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Will see if there is a way to relax this just for gcs.

harshavardhana on 5 Apr 2016

👍2 😕1

All 6 comments

$ ./mc config host add gcs https://storage.googleapis.com GOOGXXXXXXXXXXXXXXXX eb4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX S3v2
$ ./mc ls gcs
[2016-03-21 15:30:02 GMT] 0B dw_export/
$ ./mc ls gcs/dw_export/
mc: Unable to stat ‘gcs/dw_export/’. Bucket name contains invalid characters.

Currently yes buckets are restricted based on S3 bucket naming requirements i.e following bucket restrictions from here http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Will see if there is a way to relax this just for gcs.

harshavardhana on 5 Apr 2016

👍2 😕1

Can we have bucket naming/validation conventions specific to each storage provider. It can be very handy when we support multiple providers each with their own restrictions. One's policy changes will not break the other.

sreeram-boyapati on 18 Apr 2016

Can we have bucket naming/validation conventions specific to each storage provider. It can be very handy when we support multiple providers each with their own restrictions. One's policy changes will not break the other.

The problem with Google Cloud Storage is that it is not fully S3 compatible. 'mc' is a tool only written for 'S3' compatible object storage like S3, Minio, Ceph or Swift and Filesystems. - that is why adding verbatim solutions per provider doesn't make sense since it's not a generic data transfer tool for all types of providers.

What can be done is relaxing bucket restrictions just for 'GCS' specifically, it requires code changes in minio-go.

harshavardhana on 18 Apr 2016

To make minio work, I transfer (rename) my backet with name without underscore.
So after that I was able to mirror this bucket.

skipp on 18 Apr 2016

We should only encourage users to follow best practice. If we relax it, data migration between GCS, AWS and Minio becomes harder and users will always be dependent on mc.