Restsharp: port is being stripped off host header

Created on 21 Feb 2018 · 35Comments · Source: restsharp/RestSharp

Expected Behavior

Restsharp should send Host header as is without modifying port in Host header value.
https://github.com/minio/minio-dotnet/pull/212 is bringing in Restsharp 106.2.1 instead of Restsharp.Netcore (non official version with .netcore support). However, we found signature mismatch errors because the custom authenticator in use by minio dotnet sdk creates authorization signature using ip:port as value for the "Host" header. Restsharp httpclient seems to be stripping off the port, as the server trace shows only ip in the host header value.

Actual Behavior

port is being stripped off the "Host" header value in Restsharp 106.2.1

Steps to Reproduce the Problem

clone minio-dotnet sdk and get patch https://github.com/minio/minio-dotnet/pull/212
➜ minio-dotnet git:(02cbcb5) ✗ dotnet restore;dotnet build
➜ minio-dotnet git:(02cbcb5) ✗ dotnet run --project SimpleTest
2.
3.

Specifications

Version:106.2.1
Platform: Linux/Windows
Subsystem:

StackTrace

minio-dotnet client side trace
---------------------------------------
Full URL of Request http://192.168.1.157:9000/
Request completed in 90.1376 ms, Request: {
  "resource": "/",
  "parameters": [
    {
      "name": "x-amz-content-sha256",
      "value": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "type": "HttpHeader"
    },
    {
      "name": "Host",
      "value": "192.168.1.157:9000",
      "type": "HttpHeader"
    },
    {
      "name": "x-amz-date",
      "value": "20180221T205958Z",
      "type": "HttpHeader"
    },
    {
      "name": "Authorization",
      "value": "AWS4-HMAC-SHA256 Credential=minio/20180221/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=db0fbcc9aa66975a530a2621962a87787559289a218da881420686263da862df",
      "type": "HttpHeader"
    },
    {
      "name": "Accept",
      "value": "application/json, application/xml, text/json, text/x-json, text/javascript, text/xml",
      "type": "HttpHeader"
    }
  ],
  "method": "GET",
  "uri": "http://192.168.1.157:9000/"
}, Response: {
  "statusCode": 403,
  "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key></Key><BucketName></BucketName><Resource>/</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>",
  "headers": [
    {
      "Name": "Accept-Ranges",
      "Value": "bytes",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "Server",
      "Value": "Minio/DEVELOPMENT.GOGET, (linux; amd64)",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "Vary",
      "Value": "Origin",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "X-Amz-Request-Id",
      "Value": "151572E9DCF94747",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "Date",
      "Value": "Wed, 21 Feb 2018 20:59:58 GMT",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "Transfer-Encoding",
      "Value": "chunked",
      "Type": 3,
      "ContentType": null
    },
    {
      "Name": "Content-Type",
      "Value": "application/xml",
      "Type": 3,
      "ContentType": null
    }
  ],
  "responseUri": "http://192.168.1.157:9000/",
  "errorMessage": null
}
I'm not handling the Minio.Exceptions.MinioException.

Unhandled Exception: System.AggregateException: One or more errors occurred. (Minio API responded with message=The request signature we calculated does not match the signature you provided. Check your key and signing method.) (Minio API responded with message=The request signature we calculated does not match the signature you provided. Check your key and signing method.) ---> Minio.Exceptions.MinioException: Minio API responded with message=The request signature we calculated does not match the signature you provided. Check your key and signing method.
   at Minio.MinioClient.ParseError(IRestResponse response) in /home/kris/code/minio-dotnet/Minio/MinioClient.cs:line 475
   at Minio.MinioClient.<>c.<.ctor>b__78_0(IRestResponse response) in /home/kris/code/minio-dotnet/Minio/MinioClient.cs:line 70
   at Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable`1 handlers, DateTime startTime) in /home/kris/code/minio-dotnet/Minio/MinioClient.cs:line 502
   at Minio.MinioClient.<ExecuteTaskAsync>d__81.MoveNext() in /home/kris/code/minio-dotnet/Minio/MinioClient.cs:line 349
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<ListBucketsAsync>d__0.MoveNext() in /home/kris/code/minio-dotnet/Minio/ApiEndpoints/BucketOperations.cs:line 52
   --- End of inner exception stack trace ---
   at System.AggregateException.Handle(Func`2 predicate)
   at SimpleTest.Program.Main(String[] args) in /home/kris/code/minio-dotnet/SimpleTest/Program.cs:line 47

Server side trace: 
--------------------------
➜  minio git:(debugnet) minio server ~/ex                               
Drive Capacity: 6.9 GiB Free, 221 GiB Total

Endpoint:  http://192.168.1.157:9000  http://172.17.0.1:9000  http://172.18.0.1:9000  http://172.19.0.1:9000  http://172.20.0.1:9000  http://127.0.0.1:9000
AccessKey: minio 
SecretKey: minio123 

Browser Access:
   http://192.168.1.157:9000  http://172.17.0.1:9000  http://172.18.0.1:9000  http://172.19.0.1:9000  http://172.20.0.1:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.1.157:9000 minio minio123

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
signed headers .... map[X-Amz-Content-Sha256:[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855] X-Amz-Date:[20180221T205638Z] Host:[192.168.1.157]]

Source

poornas

Most helpful comment

Let’s raise it!

niemyjski on 22 Mar 2018

👍4

All 35 comments

Is there any way to reproduce it in a simple test?

alexeyzimarev on 24 Feb 2018

https://github.com/poornas/restsharp-test - this is a bare bones project using restsharp to make a List buckets call to minio server. It uses a custom authenticator - you will see that it fails on account of signature mismatch. once the request is passed to restsharp, the port is stripped off before it is passed to server.

poornas on 26 Feb 2018

Thanks. I will have a look and try to fix it.

alexeyzimarev on 26 Feb 2018

👍2

Thanks!

niemyjski on 27 Feb 2018

@alexeyzimarev Any ideas on what could be causing this?

niemyjski on 28 Feb 2018

@alexeyzimarev, did you have a chance to look into this yet? Thanks

poornas on 5 Mar 2018

Checking now

alexeyzimarev on 5 Mar 2018

I cloned your repro and I see this:

  "parameters": [
...
    {
      "name": "Host",
      "value": "play.minio.io:9000",
      "type": "HttpHeader"
    }

Do you mean that parameters in the request are fine but the value is not being properly sent to the server?

alexeyzimarev on 5 Mar 2018

@alexeyzimarev , that's correct - the trace you see above is before calling executeAsync. When ExecuteAsync is called, Restsharp invokes the Authenticate method, where we are setting signature authorization header based on url:port in the custom authenticator. port seems to be stripped by Restsharp before hitting the server and after the Authenticate() call.

poornas on 5 Mar 2018

Any updates on this? We are blocked pretty hard on this.

niemyjski on 14 Mar 2018

Sorry I was unable to look at this just yet. If it is very critical, may be you guys can spend an hour debugging it?

alexeyzimarev on 14 Mar 2018

I had a quick look into this. As far as I can tell, the problem here isn't actually with RestSharp, but is actually HttpWebRequest in System.Net!

By the time RestSharp has handed off the request to System.Net, the Host property (Uri) in the HttpWebRequest still contains the port information. However, when .SendRequest() is called, the Host header that is used is populated by the URI Host only, not host and port.

Here is a screenshot of my debugger at the above link.
screen shot 2018-03-22 at 11 22 52 am

This clearly goes against the HTTP spec which respects the use of ports.

Someone please correct me if I'm wrong, because I have never found a bug in the dotnet framework, and usually when I think I have - I haven't. Otherwise I suppose the next steps would be to raise this issue on the CoreFX repo?

twgraham on 22 Mar 2018

Let’s raise it!

niemyjski on 22 Mar 2018

👍4

Done - let's see if this is legit where the bug is or not.

twgraham on 22 Mar 2018

Just curious - does anybody know if it is a regression with recent .NET Core? (2.1) Or with last stable (2.0)? Or difference of .NET Framework vs. .NET Core, or difference between OS Windows vs. Linux? ("don't know" is fine answer, thanks!)

karelz on 22 Mar 2018

@karelz TBH i looked into this issue to solve another issue so I could stop getting compiler warnings 😆 for me its a "don't know", I've only tried this on .NET Core 2.0 on MacOS. This library only recently (months maybe) upgraded to support .NET Core over just .NET Framework. Maybe @alexeyzimarev might know more?

twgraham on 22 Mar 2018

OK, so you know at least: It works on .NET Framework (Windows) and fails on .NET Core 2.0 (Mac). Is that correct summary?

karelz on 22 Mar 2018

No - from other threads I'm in I'd say it fails all platforms on .NET Core 2.0, and I don't know about .NET Framework.

twgraham on 22 Mar 2018

@karelz I'd say that assumption would be correct. I just changed from full framework to .net core/standard and then the tests broke. I'm not sure if it's a regression from 1.x as we only tried migrating to .net standard 2.0

niemyjski on 22 Mar 2018

👍1

@niemyjski thanks for confirmation. That is sufficient info at this point. I was just scouting known facts / experiences.

karelz on 22 Mar 2018

Are we keeping this issue open?

alexeyzimarev on 22 Mar 2018

The fix in CoreFx is in the PR: https://github.com/dotnet/corefx/pull/28375

caesar-chen on 22 Mar 2018

🎉2

Yes :)

Thanks
-Blake Niemyjski

On Thu, Mar 22, 2018 at 3:57 PM, Caesar Chen notifications@github.com
wrote:

The fix in CoreFx is in the PR: dotnet/corefx#28375
https://github.com/dotnet/corefx/pull/28375

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/restsharp/RestSharp/issues/1085#issuecomment-375454963,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-So67a2QfPq72jMHNxaSCBhDXiWXOMks5thBAygaJpZM4SOXLx
.

niemyjski on 23 Mar 2018

Was there any reason we couldn't switch to HttpClient which is much faster
and easier to work with? It wouldn't have this issue?

Thanks
-Blake Niemyjski

On Fri, Mar 23, 2018 at 7:14 AM, Blake Niemyjski bniemyjski@gmail.com
wrote:

Yes :)

Thanks
-Blake Niemyjski

On Thu, Mar 22, 2018 at 3:57 PM, Caesar Chen notifications@github.com
wrote:

The fix in CoreFx is in the PR: dotnet/corefx#28375
https://github.com/dotnet/corefx/pull/28375

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/restsharp/RestSharp/issues/1085#issuecomment-375454963,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-So67a2QfPq72jMHNxaSCBhDXiWXOMks5thBAygaJpZM4SOXLx
.

niemyjski on 23 Mar 2018

@niemyjski you mean you switch of RestSharp uses HttpClient instead of HttpWebRequest?

alexeyzimarev on 23 Mar 2018

Yes :)

Thanks
-Blake Niemyjski

On Fri, Mar 23, 2018 at 9:22 AM, Alexey Zimarev notifications@github.com
wrote:

@niemyjski https://github.com/niemyjski you mean you switch of
RestSharp uses HttpClient instead of HttpWebRequest?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/restsharp/RestSharp/issues/1085#issuecomment-375680769,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-So0cbK66_Nq-MRdk0xwdb7sLPSGUoks5thQUUgaJpZM4SOXLx
.

niemyjski on 23 Mar 2018

@niemyjski the question was one _or_ another, _yes_ is really not a valid answer :)

If you mean that RestSharp should use HttpClient - I am pro to it but last time I tried it was a lot of work to match all parameters in the RestClient.

alexeyzimarev on 25 Mar 2018

Yes, we should switch to HttpClient which as much more flexibility, has better performance and a modern api which we know works everywhere. We could do some breaking changes and bump the major :)

niemyjski on 25 Mar 2018

Yeah, I am fine doing this but I don't do it alone. I tried and it was too much work. I would also remove Async calls that accept callbacks. ContinueWith should be just fine.

alexeyzimarev on 25 Mar 2018

👍1

Looks like this has been merged into 2.1, should we issue a nightly which references the new preview 2 bits and we can move other packages a long? Is there any work arounds we can do for < 2.1 runtimes/libs?

niemyjski on 20 Apr 2018

I don't have any idea about workarounds

alexeyzimarev on 20 Apr 2018

I need to do a release first before I can change prerelease to reference 2.1 preview

alexeyzimarev on 20 Apr 2018

Has this been resolved? 2.1 rtm packages are on nuget.

niemyjski on 1 Jun 2018

So, you need to try using 2.1 and see if they fixed it?

alexeyzimarev on 1 Jun 2018

@niemyjski my comment from the 20th of April is stupid. I don't need to point anything anywhere. The library compiles to .NET Standard 2.0. The error is caused by a bug in the .NET Core 2.0 runtime so as soon as you rebuild your application with .NET Core 2.1, if the issue is fixed, it should start working. Sorry about the confusion.

alexeyzimarev on 1 Jun 2018

Was this page helpful?

0 / 5 - 0 ratings