Oauthlib: Client web application does no longer send client_id

My first thinking is to revert back the changes in prepare_request_body to, by default, use the self.client_id set in the WebApplicationClient constructor.
Then, inline docs should be changed in adequation to add &client_id=xx to the prepare_request_body output.

Finally, to replace the original fix, I'll suggest to remove client_id from the args, and add a new argument to prepare_request_body like include_client=True/False to both add client_id and client_secret in the body, or not include both of them.

Thoughts?

poke @Diaoul @skion @thedrow

What about:

I'll suggest to remove client_id from the args, and add a new argument to prepare_request_body like include_client=True/False to both add client_id and client_secret in the body, or not include both of them.

Thanks

I actually triggered this same issue in one of my tests, but filed it against requests/oauthlib here: https://github.com/requests/requests-oauthlib/issues/330

I believe the issue is actually the fault of requests_oauthlib. Their docs - actually the first example in their entire docs when you load the page - support specifying the client_id in the OAuth2Session constructor. The logic at line 200 pulls the client_id from the kwargs, but does not have a fallback to pull it from the already constructed WebApplicationClient instance.

@jvanasco, the current issue #585 can be fixed in requests-oauthlib alone, or by reverting oauthlib's PR #505. However, none of the solution will fix the behavior mentioned by @skion in his comment at https://github.com/oauthlib/oauthlib/pull/505#issuecomment-351221107

According to the spec, the client_id parameter must be sent for unauthenticated clients, but is preferably NOT sent in the token request body for confidential clients, as in that case the preferred mechanism to authenticate the client is via HTTP Basic auth. However, the WebApplication class always includes it (which breaks some servers) plus offers no mechanism to remove it.

Oauthlib must provide an elegant and simple way for requests-oauthlib to solve the problem. If we can find a solution in this discussion it will be great; because that's a huge blocker.

Would allowing client_id=False in prepare_request_body() help, to indicate that a client_id isn't to be sent? Although even if so, that would lead to this ugliness near line 126:

client_id = None if client_id=False else self.client_id

Ah, I see.

Is there an existing unit-test for when client_id should be sent vs not? If not, does anyone have a listing that can be used to generate one? I'd be happy to take a stab at fixing this and requests-oauthlib, because it's blocking some of my work right now.

@JonathanHuot

I'll suggest to remove client_id from the args, and add a new argument to prepare_request_body like include_client=True/False to both add client_id and client_secret in the body, or not include both of them.

Reading this back, I actually quite like your suggestion. We can probably get away doing it in a non-breaking way, since the function already takes **kwargs.

One note:

to both add client_id and client_secret in the body

Since this is about public clients IIUC, I wouldn't think the client_secret is involved; it's just the client_id being added to the body? In that case I would also consider renaming the new parameter to include_client_id=True/False.

In that case I would also consider renaming the new parameter to include_client_id=True/False.

Indeed ! client_secret is not involved since it is not present in WebApplicationClient.

@jvanasco, if you want to do a PR I think the changes should be:
1) Revert https://github.com/oauthlib/oauthlib/pull/505
2) Change signature of prepare_request_body() to remove client_id and add include_client_id=True/False (True (default): it adds self.client_id)

Then, requests-oauthlib will have the choice in https://github.com/requests/requests-oauthlib/blob/master/requests_oauthlib/oauth2_session.py#L196-L211 to either:
A) Include client_id alone in the body

self._client.prepare_request_body(..)

B) Include client_id and client_secret in auth and not include them in the body (RFC preferred solution)

self._client.prepare_request_body(include_client_id=False, ..)
auth = requests.auth.HTTPBasicAuth(client_id, client_secret)

C) Include client_id and client_secret in the body (RFC alternative solution)

self._client.prepare_request_body(client_secret=client_secret, ..)

I'll generate PR's for both projects today.

I've pretty much got a PR and tests done for OAuthlib. I have a question though...

Should client_id still be allowed as a kwarg? This is in part for backwards compatibility, but also for edge cases. Because this method was somewhat broken, I think it may be worth either making it work as intended (as in allowing it in prepare_request_body to override the self.client_id) or to raise an Exception on incorrect usage (as in raising an error if client_id is provided but does not match self.client_id).

Wouldn't see how you'd ever want to override client_id with a different value so would vote for raising an exception if they differ.

Should we, in addition, log a DeprecationWarning if client_id was provided at all as a kwarg?

PR #593 submitted. It raises a DeprecationWarning when client_id is submitted, and a ValueError if it differs from self.client_id. There is also a new test that ensures compliance with the three scenarios @JonathanHuot detailed.

ran into my first problem with requests-oauthlib PR candidate as I write some tests

It ALWAYS invokes prepare_request_body with username=username, password=password. This seems wrong. I'm hoping someone here is more familiar with the RFC and knows the answers to the following:

1. should a username+password even ever be a param in the request.body ?
2. if the username+password appear in the HTTP Basic Auth header, should they be duplicated in the request body?
3. Is it possible to have both a username+password body params and a HTTPBasicAuth header with the client details ?

Thanks for running into this.

1. username and password must always be present in request's body. They must be used only for password grant aka legacy. Those must not be used for other grants (implicit, code, client credentials).
2. HTTP Basic Auth is optional for client credentials (recommended for confidential clients i.e. with a client_secret). User credentials must never be in HTTP Basic Auth.
3. Yes

Thanks. Just to clarify two things, and please feel free to talk to me like I'm five. I want to ensure I get this and the tests right:

1. Username and password are only used in a certain type of grant which require them. If used they may only be present in the request body.

Unless they are explicitly specified they should not be present, correct?

1. If Http basic auth is only for client credentials, then the existing requests-oauthlib should not have the block which generates a basic auth from the username&password combo.

Please forgive me being verbose and obsessing over these little details. I just want to make sure j get the right behavior and can write tests that ensure we don't get another regression.

@jvanasco, I can tell about the OAuth2 RFC, however I'm not sure how the requests-oauthlib and flask-oauthlib fit together.

1. Yes

Correct.

1. Yes, AFAIU it should not have this block https://github.com/requests/requests-oauthlib/blob/master/requests_oauthlib/oauth2_session.py#L207-L211.

It is my understanding, however it will be good to collate with the reality of the field; i.e. requests-oauthlib and differents public providers experiences. Multiple requests-oauthlib discussions https://github.com/requests/requests-oauthlib/issues/218, https://github.com/requests/requests-oauthlib/issues/211, https://github.com/requests/requests-oauthlib/issues/264, already happened.

I believe they had a confusion between client password and client secret which are actually two wordings for the exact same thing.
If we follow the rationale behind https://github.com/requests/requests-oauthlib/pull/206, the PR's content should never have been like adding HTTPAuth(username, password) but it should have been HTTPAuth(client_id, client_secret (the client's password).

Poke @Lukasa, @chaosct, @ibuchanan which participated in the requests-oauthlib's discussions.

Great! thanks so much. I thought that was what is going on but wanted to confirm.

I think I know how I want to structure the requests stuff now. I have a handful of commits on the main requests project, so I know what the maintainers like to see in PRs and functionality.

1. Username and password are only used in a certain type of grant which require them. If used they may only be present in the request body.

Yes, to the first part: only a certain type of grant requires them. But on the 2nd part about sending them in the request body, the spec says:

Including the client credentials in the request-body
using the two parameters is NOT RECOMMENDED
and SHOULD be limited to clients unable to directly utilize
the HTTP Basic authentication scheme...

But for servers, it reads:

The authorization server MAY support including
the client credentials in the request-body...

A compliant client, would not send the credentials in the request body.
But, for some partially implemented servers, they will only accept them in the request body.
If I recall correctly, my PR solved for this confusion by adding the auth header,
so the client sends both.

I believe @JonathanHuot is correct about the 2nd point.

Hi @ibuchanan, the quotes you are referring to are using the term client credentials. We have to be very careful to not mix the client with the resource owner (the actual user).

The roles are clearly explained here rfc6749#1.1 .
This client credentials refers to client_id and client_secret and the resource owner is refering to username and password. Those are not interchangeable.

So, by reading the RFC with those roles mean that a compliant client SHOULD send client credentials (client_id,client_secret) in HTTP Basic and MUST send user credentials (username,password) in the request body (never read an alternative here); see rfc6749#4.3.2.

Some server reject the request (400) if the client_id is in the request
body. I think the default should be what is recommended by the spec.

Ok. I think the current PR for oauthlib satisfies the above concerns - the include_client_id flag explicitly allows the client_id to be sent or not.

in terms of requests_oauthlib, this is what I am thinking:

1. username and password will only appear in the body (not as HTTP Basic). If that behavior is needed for a non-compliant server integration, the implementor can submit an auth or headers argument to fetch_token().

2. supplying the client_id in the correct place is a bit annoying, but I think I have the logic and use-cases down. this will definitely need some review.

A question I have for @JonathanHuot and @ibuchanan :

oauthlib's OAuth2 Client and requests_oauthlib's OAuth2Session do not keep the client_secret and must repeatedly invoke it. This was not the case in the OAuth1, and I think this was the actual issue behind #370. The RFC didn't mention a need for this, and I couldn't find any history.

To me, it makes sense to extend the Client with storing the client_secret, and start to deprecate the OAuth2Session reliance on passing in the client_secret on fetch_token and request in favor of using the one now in the Client itself. (this would also address some other inconsistencies reported in https://github.com/requests/requests-oauthlib/issues/264 )

I've slightly changed the PR for oauthlib (#593) to standardize the test variables for username/password and client_id/client_secret. i think that should prevent mistakes from confusion between the two in the future.

The proposed change for requests-oauthlib : https://github.com/requests/requests-oauthlib/compare/master...jvanasco:fix-oauthlib_client_id

This one is a bit more drastic, because looking at the code and tests, it seems the library was just trying to make all sorts of things work that shouldn't.

The fix does a few things:

1. Checking for username and password happens only for LegacyApplicationClient instances -- as that should be the only kind needed (correct?). There is a section under the check which merges the username/password into the kwargs dict. It is currently outdented, because the tests suggest other clients might want to pass this information over.

2. The logic for handling the client_id/auth headers was rewritten to ensure the right types of auth happen in the right place by default. If a user wants to force credentials in another way, it is still possible.

3. Question: Is there any situation where client_secret would not be passed? I can't think of any, but there are many oAuth flows.

So in requests-oauthlib version:

| include_client_id | auth | behavior |
| ------------------- | --------------- | -------- |
| None (Default) | None (Default) | create an Auth object with the client_id. Do not send the client_id in the body. This is the default behavior, because the RFC recommends it. |
| None (Default) | present | use the Auth object. invoke oauthlib's prepare_request_body with include_client_id=False |
| False | present | use the Auth object. invoke oauthlib's prepare_request_body with include_client_id=False |
| True | present | use the Auth object. invoke oauthlib's prepare_request_body with include_client_id=True |
| True | None (Default) | create an Auth object with the client_id. invoke oauthlib's prepare_request_body with include_client_id=True |
| False | None (Default) | create an Auth object with the client_id. invoke oauthlib's prepare_request_body with include_client_id=False |

or stated otherwise:

• create an auth object
  
  
  
  • do not send the client id in the body:
  
  
  • (include_client_id=None, auth=None)
  
  
  • (include_client_id=False, auth=None)
  
  
  • send the client id in the body:
  
  
  • (include_client_id=True, auth=None)
  
  
• use an explicit auth object
  
  
  
  • do not send the client_id in the body:
  
  
  • (include_client_id=None, auth=authObject)
  
  
  • (include_client_id=False, auth=authObject)
  
  
  • send the client_id in the body:
  
  
  • (include_client_id=True, auth=authObject)
  
  

@jvanasco : looks very good on oauthlib and requests-oauthlib side.

About your 3. question:

You can have public clients without client_secret (or empty, that's close on a point of view of the RFC); so the python API must support "no secret".

The real-world use-case is often for native applications where you prefer to use authorization code, but you cannot guarantee the security around keep the secret safe, so, either you accept a client_id with no client_secret (or empty client_secret which are identical for the RFC); or you have also another PKCE RFC available (see https://oauth.net/2/pkce/ ). But the latter is not implemented on oauthlib side, yet.

thanks. i'll add some test cases to ensure we can submit client_id without client_secret. Sorry for asking so many questions, there are just so many correct implementations possible of this spec (and even more broken implementations that need to work).

@JonathanHuot the existing implementation does not support sending an empty string for client_secret. It is removed in this logic https://github.com/oauthlib/oauthlib/blob/master/oauthlib/oauth2/rfc6749/parameters.py#L90-L125 -- specifically line 122

Supporting it can be adding something like this right after that routine:

if ('client_secret' in kwargs) and ('client_secret' not in params):
    if kwargs['client_secret'] == '':
        params.append((unicode_type('client_secret'), kwargs['client_secret']))

That would send an empty string for client_secret when the secret is an empty string, but not send the client_secret if the value is None.

I think it is worth supporting this, because if the RFC supports either of the two variants... there are likely to be many broken implementations that only support one variant.

This original issue is fixed in oauthlib. However the behavior is still there until https://github.com/requests/requests-oauthlib/pull/331 is fixed.