mc doesn't work over nginx proxy

Created on 13 Oct 2018  ·  9Comments  ·  Source: minio/mc

I am running 3 minio servers (Debian 9, with the binary) on dedicated machines in distributed mode. On a separate machine I am running nginx (nginx version: nginx/1.15.5) as a proxy, as described in the official documentation.

MC works locally on the machine running nginx, but as soon as I am running it on a remote machine, it does not work. I can neither copy, heal or do anything else. Other clients like s3cmd, the Minio Web Browser, Transmit, ... work like a charm. Only the MC client ist not capable of transferring files etc.

mc version

Version: 2018-09-26T00:42:43Z
Release-tag: RELEASE.2018-09-26T00-42-43Z
Commit-id: 87f7e65c4c837c8886bf2dd8800c445983b36187

System information

Here is the output of mc admin info

➜ mc admin info minio --debug
mc: <DEBUG> GET /minio/admin/v1/info HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) madmin-go/0.0.1 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T103300Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/json
Date: Sat, 13 Oct 2018 10:33:00 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D248043802835
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  647.517578ms

●  192.168.100.11:9000
   Uptime : online since 1 day ago
  Version : 2018-10-06T00:15:16Z
   Region :
 SQS ARNs : <none>
    Stats : Incoming 2.6MiB, Outgoing 47KiB
  Storage : Used 496MiB
    Disks : 6, 0


●  192.168.100.10:9000
   Uptime : online since 1 day ago
  Version : 2018-10-06T00:15:16Z
   Region :
 SQS ARNs : <none>
    Stats : Incoming 5.3MiB, Outgoing 102KiB
  Storage : Used 496MiB
    Disks : 6, 0


●  192.168.100.12:9000
   Uptime : online since 1 day ago
  Version : 2018-10-06T00:15:16Z
   Region :
 SQS ARNs : <none>
    Stats : Incoming 60KiB, Outgoing 42KiB
  Storage : Used 496MiB
    Disks : 6, 0

Here is the nginx access log for a mc cp command

**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "GET /tests/?location= HTTP/1.1" 200 139 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "HEAD /tests/ HTTP/1.1" 200 0 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "GET / HTTP/1.1" 200 661 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "HEAD /tests/ HTTP/1.1" 200 0 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "GET / HTTP/1.1" 200 661 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:34 +0200] "HEAD /tests/ HTTP/1.1" 200 0 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:35 +0200] "GET / HTTP/1.1" 200 661 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:35 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:36 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:37 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:38 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:45 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:48 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:43:50 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:44:04 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:44:13 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"
**redacted-ip** - - [13/Oct/2018:12:44:18 +0200] "PUT /tests/Scannable-Dokument.jpg HTTP/1.1" 400 279 "-" "Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z" "-"

Here is the nginx error log for that command

2018/10/13 12:43:48 [error] 16018#16018: *409 recv() failed (104: Connection reset by peer) while sending to client, client: **redacted-ip**, server: **redacted-domain**, request: "PUT /tests/Scannable-Dokument.jpg HTTP/1.1", upstream: "http://192.168.100.11:9000/tests/Scannable-Dokument.jpg", host: "**redacted-domain**"
2018/10/13 12:43:50 [error] 16018#16018: *411 recv() failed (104: Connection reset by peer) while sending to client, client: **redacted-ip**, server: **redacted-domain**, request: "PUT /tests/Scannable-Dokument.jpg HTTP/1.1", upstream: "http://192.168.100.11:9000/tests/Scannable-Dokument.jpg", host: "**redacted-domain**"
2018/10/13 12:44:04 [error] 16018#16018: *413 recv() failed (104: Connection reset by peer) while sending to client, client: **redacted-ip**, server: **redacted-domain**, request: "PUT /tests/Scannable-Dokument.jpg HTTP/1.1", upstream: "http://192.168.100.11:9000/tests/Scannable-Dokument.jpg", host: "**redacted-domain**"
2018/10/13 12:44:13 [error] 16018#16018: *415 recv() failed (104: Connection reset by peer) while reading upstream, client: **redacted-ip**, server: **redacted-domain**, request: "PUT /tests/Scannable-Dokument.jpg HTTP/1.1", upstream: "http://192.168.100.11:9000/tests/Scannable-Dokument.jpg", host: "**redacted-domain**"
2018/10/13 12:44:18 [error] 16018#16018: *417 recv() failed (104: Connection reset by peer) while sending to client, client: **redacted-ip**, server: **redacted-domain**, request: "PUT /tests/Scannable-Dokument.jpg HTTP/1.1", upstream: "http://192.168.100.11:9000/tests/Scannable-Dokument.jpg", host: "**redacted-domain**"

Here is the /etc/nginx/nginx.conf file

user  nginx;
worker_processes  8;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    server_tokens off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA!aNULL:!eNULL:!MEDIUM:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SEED:!DES';
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    ssl_dhparam /etc/ssl/certs/dhparam.pem;

    gzip on;
    gzip_disable "msie6";

    gzip_comp_level 9;
    gzip_types text/plain text/css application/javascript text/csv application/xml application/json application/vnd.ms-excel;

    include /etc/nginx/sites-enabled/*.conf;
}

Here is the config for the site 

upstream minio_servers {
    server 192.168.100.10:9000;
    server 192.168.100.11:9000;
    server 192.168.100.12:9000;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name **redacted-domain**;

    ssl_certificate /etc/letsencrypt/live/**redacted-domain**/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/**redacted-domain**/privkey.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384';

    ssl_prefer_server_ciphers on;

    add_header Strict-Transport-Security max-age=15768000;

    ssl_stapling on;
    ssl_stapling_verify on;

    resolver 1.1.1.1;

    client_max_body_size 1000m;
    client_body_buffer_size 1000M;

    location / {
        proxy_set_header Host $http_host;
        proxy_pass http://minio_servers;
        proxy_buffering off;
    }
}

Here is the log from mc cp

➜  mc cp Scannable-Dokument.jpg minio/tests --debug > minio.log
mc: <DEBUG> GET /tests/?location= HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104333Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513B9130E6B
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  703.140014ms

mc: <DEBUG> HEAD /tests/ HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z

mc: <DEBUG> HTTP/1.1 200 OK
Connection: close
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513C091F556
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  123.206481ms

mc: <DEBUG> GET / HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513C7E90B6B
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  117.662247ms

mc: <DEBUG> HEAD /tests/ HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z

mc: <DEBUG> HTTP/1.1 200 OK
Connection: close
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513CEFB2FD8
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  122.064239ms

mc: <DEBUG> GET / HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513D646E2A5
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  117.517384ms

mc: <DEBUG> HEAD /tests/ HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z

mc: <DEBUG> HTTP/1.1 200 OK
Connection: close
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Sat, 13 Oct 2018 10:43:34 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513DD5815F9
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  120.675639ms

mc: <DEBUG> GET / HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181013T104334Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 13 Oct 2018 10:43:35 GMT
Server: nginx
Strict-Transport-Security: max-age=15768000
Vary: Origin
X-Amz-Request-Id: 155D2513E48F76D8
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  118.697509ms

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104335Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104335Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104336Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104337Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104344Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104347Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104349Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104403Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104412Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <DEBUG> PUT /tests/Scannable-Dokument.jpg HTTP/1.1
Host: **redacted-domain**
User-Agent: Minio (darwin; amd64) minio-go/v6.0.6 mc/2018-09-26T00:42:43Z
Content-Length: 429688
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181013/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-com.apple.quarantine, Signature=**REDACTED**
Content-Type: image/jpeg
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181013T104417Z
X-Amz-Meta-Com.apple.quarantine: 0083;57e15bf0;Safari;DB1870C5-FB08-4A00-A022-E8C216CFDD86
Accept-Encoding: gzip

mc: <ERROR> Failed to copy `Scannable-Dokument.jpg`. Put https://**redacted-domain**/tests/Scannable-Dokument.jpg: Connection closed by foreign host https://**redacted-domain**/tests/Scannable-Dokument.jpg. Retry again.
 (3) cp-main.go:414 cmd.doCopySession(..) Tags: [Scannable-Dokument.jpg]
 (2) common-methods.go:196 cmd.uploadSourceToTargetURL(..) Tags: [https://**redacted-domain**/tests/Scannable-Dokument.jpg]
 (1) common-methods.go:130 cmd.putTargetStream(..) Tags: [minio, https://**redacted-domain**/tests/Scannable-Dokument.jpg]
 (0) client-s3.go:684 cmd.(*s3Client).Put(..)
 Release-Tag:RELEASE.2018-09-26T00-42-43Z | Commit:87f7e65c4c83 | Host:Philips-MacBook-Pro.local | OS:darwin | Arch:amd64 | Lang:go1.10.2 | Mem:5.1MB/14MB | Heap:5.1MB/9.6MB
mc: <ERROR> Session safely terminated. To resume session `mc session resume rDXVeuhO`

Thanks!

fixed medium
Source
picture philipkozeny

Most helpful comment

Apple adding the quarantine attribute was added sometime back (2007'ish according to this - https://developer.apple.com/library/archive/releasenotes/Carbon/RN-LaunchServices/index.html). It seems to want to do this to any downloaded file, if you check your default downloads folder you should see that attribute set on quite a few files. By default nginx strips what it considers to be invalid characters (including "."). So, when the com.apple.quarantine attr gets sent as a header, by default nginx strips it - https://trac.nginx.org/nginx/ticket/629. So, you can add ignore_invalid_headers off;, something like:

server {
 listen 80;
 server_name ebox;
 ignore_invalid_headers off;
 location / {
   proxy_set_header Host $http_host;
   proxy_pass http://hbox:9000;
   client_max_body_size 1000m;
 }
}

As an aside, this would also affect any attribute manually set via say setfattr (since it requires prepending user. to the attribute).
We should add this config to the documentation.

picture eco-minio on 18 Oct 2018
👍2

All 9 comments

Can you remove xattr -d com.apple.quarantine on the file and try again ?

harshavardhana picture harshavardhana on 13 Oct 2018
👍1

Good catch, that's it - thanks! It seems that all files / folder I have tested have that attribute somehow. Any changes in Mojave that could have triggered that or just bad luck? I think it would be great if the upload would not throw an error when a file has that attribute and complete the upload. What do you think?

philipkozeny picture philipkozeny on 13 Oct 2018

Good catch, that's it - thanks! It seems that all files / folder I have tested have that attribute somehow. Any changes in Mojave that could have triggered that or just bad luck? I think it would be great if the upload would not throw an error when a file has that attribute and complete the upload. What do you think?

It is nginx filtering out these HTTP headers, have to figure out why nginx is stripping this header.

harshavardhana picture harshavardhana on 14 Oct 2018

Apple adding the quarantine attribute was added sometime back (2007'ish according to this - https://developer.apple.com/library/archive/releasenotes/Carbon/RN-LaunchServices/index.html). It seems to want to do this to any downloaded file, if you check your default downloads folder you should see that attribute set on quite a few files. By default nginx strips what it considers to be invalid characters (including "."). So, when the com.apple.quarantine attr gets sent as a header, by default nginx strips it - https://trac.nginx.org/nginx/ticket/629. So, you can add ignore_invalid_headers off;, something like:

server {
 listen 80;
 server_name ebox;
 ignore_invalid_headers off;
 location / {
   proxy_set_header Host $http_host;
   proxy_pass http://hbox:9000;
   client_max_body_size 1000m;
 }
}

As an aside, this would also affect any attribute manually set via say setfattr (since it requires prepending user. to the attribute).
We should add this config to the documentation.

eco-minio picture eco-minio on 18 Oct 2018
👍2

This is a Nginx issue as they are not fully RFC compliant in this scenario when proxying requests. With the above option turned off as suggested by @eco-minio we can now use Nginx properly.

We will update this in our documentation as well.

harshavardhana picture harshavardhana on 18 Oct 2018

For reference, set ignore-invalid-headers to false if you are using Kubernetes nginx ingress controller (maybe add that in the documentation as well).

zllovesuki picture zllovesuki on 18 Oct 2018
👍1

@eco-minio I just hate whoever responded in the original nginx ticket: "Whether or not these headers match a grammar of a particular RFC is mostly irrelevant." Well we don't like the RFC grammar so we are going to do it OUR way. Then why do we have standards and/or RFC?

Does Caddy or Traefik suffer from the same problem?

zllovesuki picture zllovesuki on 22 Oct 2018
👍1

Does Caddy or Traefik suffer from the same problem?

They don't @zllovesuki

harshavardhana picture harshavardhana on 22 Oct 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

lock[bot] picture lock[bot] on 25 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ramosisw picture ramosisw  ·  4Comments
zllovesuki picture zllovesuki  ·  19Comments
deekoder picture deekoder  ·  13Comments
teto picture teto  ·  7Comments
mausch picture mausch  ·  8Comments