Pip: Index URLs containing unescaped "@" in username cause "Failed to parse" error in pip 19.2

My thoughts: the current behavior is more technically correct and people should start using the new keyring support for this stuff.

At the same time, it definitely isn't nice that this broke existing reasonable workflows. I think if someone can figure out how to re-introduce the legacy behavior here, that'd be great.

Based on pip's documented deprecation policy, we don't need to have a deprecation cycle for such undocumented behaviors. None the less, we should do a deprecation cycle, warning users to switch to the more "correct" syntax or tooling here before dropping support and switching to a nicer error message.

I'm not sure that the bisection found the right commit here but none the less, thanks for doing that!

@pradyunsg, maybe I misunderstand.. can you please direct me to the needed "current behavior"?
If we need to refactor the way we use 3rd party repos (in our case gemfury.com) then that's fine. I just don't understand what the right syntax should be :)

I'm not sure that the bisection found the right commit here but none the less, thanks for doing that!

you're totally right. I corrected my test script and now it points to c63ee61027fe which makes more sense now I guess

Getting the same issue

pip install --user --index-url https://[email protected]:[email protected]/foo/api/pypi/PyPi-dev/simple -v em-generate-config 
Created temporary directory: /tmp/pip-ephem-wheel-cache-1q06a68y 
Created temporary directory: /tmp/pip-req-tracker-1igksn7z 
Created requirements tracker '/tmp/pip-req-tracker-1igksn7z' 
Created temporary directory: /tmp/pip-install-six90fkc 
Looking in indexes: https://rnc_build%40foo.com:****@foo.jfrog.io/foo/api/pypi/PyPi-dev/simple 
Collecting em-generate-config 
  1 location(s) to search for versions of em-generate-config: 
  * https://rnc_build%40foo.com:****@bar.jfrog.io/foo/api/pypi/PyPi-dev/simple/em-generate-config/ 
  Getting page https://rnc_build%40foo.com:****@bar.jfrog.io/foo/api/pypi/PyPi-dev/simple/em-generate-config/ 
ERROR: Could not install packages due to an EnvironmentError. 
Traceback (most recent call last): 
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/models.py", line 379, in prepare_url 
    scheme, auth, host, port, path, query, fragment = parse_url(url) 
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/util/url.py", line 234, in parse_url 
    raise LocationParseError(url) 
pip._vendor.urllib3.exceptions.LocationParseError: Failed to parse: https://[email protected]:[email protected]/foo/api/pypi/PyPi-dev/simple/em-generate-config/

can you please direct me to the needed "current behavior"?

Whoops. Yes. (This is why I needed a nap)

The issue here is that the parts of a URL should be... urlencoded.

Thus, the @ in the email used, as a username, should be %40. The following is in fact not a valid URL based on the RFC that defines these things:

https://[email protected]:[email protected]/pypi

The correct way to represent this information would be:

https://awesome%40pradyunsg.me:[email protected]/pypi

If someone has an issue with that... They should probably take that up with the folks who wrote that RFC. It's mentioned in the urllib issue linked further up in this thread.

Aside: @markuszoeller had pointed out the workaround in a different issue that's been closed as a duplicate of this.

What I think I'm suggesting in that comment... is that pip could detect when the user does use such invalid URLs (since it accepted that in the past), warn the user, and work like 19.1.1 -- after some time, the warning and workaround would be removed.

If y'all do think that's necessary after this breakage, do let us know.

I'll be more than happy to just skip the additional work if it's not helpful to the folks affected by it -- there's no point in doing things that are not useful for the end users. 🙃

I corrected my test script and now it points to c63ee61 which makes more sense now I guess

Indeed. I can see there why we weren't catching these uses earlier -- pip was doing "home grown" processing of the index URL's auth information which obviously didn't consider such input invalid. What changed is we're now using a urllib utility function to get parts of the URL, which is stricter and RFC compliant.

pip was doing "home grown" processing of the index URL's auth information which obviously didn't consider such input invalid. What changed is we're now using a urllib3 utility function to get parts of the URL, which is stricter and RFC compliant.

right, now I get it, thanks @pradyunsg. For those watching at home ;) as a fix one can pass https://<my-access-token>:@github.com/myorganization/myprivaterepo/archive/myrelease.zip#egg=mypackage (notice the : after the access token) which makes the URL more RFC conformant (does it?) and pip will happily install the package.

EDIT:
or (even better) https://:<my-access-token>@github.com/...

makes the URL more RFC conformant (does it?)

No clue. I'm not a subject matter expert.

All that I've said above is basically a well informed guess that I've made by looking at all the relevant bits of code on my phone... because my laptop is too far away from the bed. 🙃

https://:@github.com/myorganization/myprivaterepo/

Noting the last thing that I'm going to say here based on (only) intuition + memory, this seems like pip incorrectly requiring both username / password components to be present.

If that's the case, it's almost certainly a bug, albeit a separate one.

I tried @slafs last comment and it seems to be working fine for me.
Adding ":" after the token works while adding it before the token does not

Adding ":" after the token works while adding it before the token does not

right, not sure why it works for me (i.e. with pip), but now I see that a simple curl -L won't download the package if the : is before the token.

If that's the case, it's almost certainly a bug, albeit a separate one.

yeah, I'm now inclined to believe that having no : in the auth section of the URL is totally fine too. I mean, curl doesn't complain. Neither does urllib3:

>>> import urllib3
>>> urllib3.__version__
'1.25.3'
>>> urllib3.util.parse_url('http://user@host')
Url(scheme='http', auth='user', host='host', port=None, path=None, query=None, fragment=None)

I think I'm going to file an issue on this.

I think I'm going to file an issue on this.

Please do. :)

For people that have already spent time looking into this, can you link to the line of code that is doing the parsing under discussion?

@cjerdonek I'm not sure if I have an exact line for you. From looking briefly at c63ee61027fe7dea I just can tell that https://github.com/pypa/pip/blob/81040b5d0644460a46a360d67c451c03bbbb03ab/src/pip/_internal/download.py#L333-L335 looks suspicious to me. Specially compared to the previous version of that file: https://github.com/pypa/pip/blob/e34769861ec5ed8aa5ce23670e111fe8db446c39/pip/download.py#L91-L96

oh, sorry @cjerdonek, you're probably asking for the issue with unencoded @. I was refering to my issue with the auth token.

Also, I think we should be open to the possibility of there being two issues here. In this comment (second reply to the original post), there is a parsing error in the logs (evidently coming from urllib3 based on the "Failed to parse" text):

ERROR: Could not install packages due to an EnvironmentError: Failed to parse: https://<email.com>:[secure]@<artifactory_host>/artifactory/api/pypi/pypi-virtual/simple/setuptools/

But in the original post, a "No matching distribution found for package_name" error was reported without a "Failed to parse" error in the logs (unless that was present in the logs but simply not included in the report).

It would be good to know the stacktrace at the point of the first error above.

Hmm... Seems like my intuition wasn't exactly correct. Hah!

@cjerdonek I was referring to the parsing happening in MultiDomainBasicAuth.__call__ but I see now that it was in fact using urllib's parsing utilities. The weird thing that threw me off was the concatenation but I guess that's... fine?

ISTM now that the conditional that @slafs pointed out might be the root cause of this -- changing it to have or instead of and, and having the or "" in the call, should fix the auth-needs-2-parts issue as well as this report, I think.

Here is the full traceback when I reproduced the "Failed to parse" error with a made-up extra-index-url (I believe verbose mode causes the traceback to display):

$ pip install twine --extra-index-url https://[email protected]:[email protected]/simple/ -vvv
...
  Getting page https://foo%40example.com:****@github.com/simple/twine/
ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/.../pip/src/pip/_vendor/requests/models.py", line 379, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/.../pip/src/pip/_vendor/urllib3/util/url.py", line 234, in parse_url
    raise LocationParseError(url)
pip._vendor.urllib3.exceptions.LocationParseError: Failed to parse: https://[email protected]:[email protected]/simple/twine/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../pip/src/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/.../pip/src/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/.../pip/src/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/.../pip/src/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/.../pip/src/pip/_internal/operations/prepare.py", line 134, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/.../pip/src/pip/_internal/req/req_install.py", line 211, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/.../pip/src/pip/_internal/index.py", line 1201, in find_requirement
    req.name, specifier=req.specifier, hashes=hashes,
  File "/.../pip/src/pip/_internal/index.py", line 1183, in find_candidates
    candidates = self.find_all_candidates(project_name)
  File "/.../pip/src/pip/_internal/index.py", line 1128, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/.../pip/src/pip/_internal/index.py", line 1282, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "/.../pip/src/pip/_internal/index.py", line 234, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/.../pip/src/pip/_internal/index.py", line 182, in _get_html_response
    "Cache-Control": "max-age=0",
  File "/.../pip/src/pip/_vendor/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/.../pip/src/pip/_internal/download.py", line 610, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/.../pip/src/pip/_vendor/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/.../pip/src/pip/_vendor/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/.../pip/src/pip/_vendor/requests/models.py", line 313, in prepare
    self.prepare_url(url, params)
  File "/.../pip/src/pip/_vendor/requests/models.py", line 381, in prepare_url
    raise InvalidURL(*e.args)
pip._vendor.requests.exceptions.InvalidURL: Failed to parse: https://[email protected]:[email protected]/simple/twine/

Interestingly, as you can see from the first log message above:

Getting page https://foo%40example.com:**@github.com/simple/twine/

our own logging code does manage to quote the character in the username that, if it were quoted in the real request, would I believe resolve (one of) the issues. That quoting for logging happens in redact_netloc(): https://github.com/pypa/pip/blob/81040b5d0644460a46a360d67c451c03bbbb03ab/src/pip/_internal/utils/misc.py#L1109-L1122

heh, about that last one, I also found that pip doesn't always redacts the password. If you change the credentials to "valid" ones, then after doing pip install you get e.g.

$ pip install https://foo:[email protected]/slafs
Collecting https://foo:****@github.com/slafs
  Downloading https://foo:[email protected]/slafs
...

😅

@pypa/pip-committers So for the error caused by the stricter URL parsing introduced by the urllib3 v1.25 vendor update (original PR https://github.com/urllib3/urllib3/pull/1487 with follow-on issue https://github.com/urllib3/urllib3/issues/1640 ), do we want to keep things as is and require users to properly quote things (it doesn't seem like urllib3 will change), or do we want to "fix" the URL provided by the user by doing something like what's done in redact_netloc() above? Going back to the previous logic without fixing the user-provided URL would mean rolling back the vendor update of urllib3, which I'm not sure we'd want to do.

heh, about that last one, I also found that pip doesn't always redacts the password.

You can file that as a separate issue. pip doesn't have a "global" way of doing this right now. There are a couple similar instances like that I believe that exist as pip issues.

do we want to keep things as is and require users to properly quote things

If the users who've filed this issue say that quoting the @ is complex for them, I'd say let's add a workaround for urllib3.

I am not a fan of the idea of working around urllib3, even though I've suggested it before.

I agree - I'd prefer not to "fix" the URL, it's likely to be fragile code that we'd have to maintain, but if quoting the @ is complex for the users actually affected by this issue, we should do what we can to ease their pain.

I'd be inclined to make any workaround temporary, though, and start a process of deprecating non-compliant URLs (so giving users time to address this, but not maintaining the workaround indefinitely).

For what it's worth, it looks like twine uploads do not work if you use a username with the %40 instead of a @. So in my case, I have a PyPI username credential stored in my secret storage engine. If I update that username to foo%40bar.com, my pip installs will work, however my twine upload (which uses the same credential), will fail:

twine upload --verbose --repository-url https://mypypi.com -u foo%40bar.com -p <redacted> dist/my-package-*
Uploading distributions to https://mypypi.com
Uploading my-package-1.2.3.tar.gz

100% 166k/166k [00:00<00:00, 430kB/s]  
Content received from server:
<html>
 <head>
  <title>401 Unauthorized</title>
 </head>
 <body>
  <h1>401 Unauthorized</h1>
  This server could not verify that you are authorized to access the document you requested.  Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br/><br/>

I am witnessing a separate, yet similar issue where the --extra-index-url fails if there is any sort of authorization. Not just being escaped by @.

Using:
python 3.7.4
pip 19.2.1

1. Removing the authorization and making the package public on our private repo lets us download it
2. Of course, downgrading to pip 19.1.1 also solves the issue

pip3 install package --extra-index-url https://[email protected]/pypi -vvv

Traceback:

Created temporary directory: /private/var/folders/rb/rg18vqd16lxg47cm1kvgh3yh0000gn/T/pip-ephem-wheel-cache-dkqjnspg
Created temporary directory: /private/var/folders/rb/rg18vqd16lxg47cm1kvgh3yh0000gn/T/pip-req-tracker-_5f9p1zv
Created requirements tracker '/private/var/folders/rb/rg18vqd16lxg47cm1kvgh3yh0000gn/T/pip-req-tracker-_5f9p1zv'
Created temporary directory: /private/var/folders/rb/rg18vqd16lxg47cm1kvgh3yh0000gn/T/pip-install-ajhlm6xm
Looking in indexes: https://pypi.org/simple, https://[email protected]/pypi
Collecting package
  2 location(s) to search for versions of package:
  * https://pypi.org/simple/package/
  * https://[email protected]/pypi/package/
  Getting page https://pypi.org/simple/package/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/package/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/package/ HTTP/1.1" 404 13
  Status code 404 not in (200, 203, 300, 301)
  Could not fetch URL https://pypi.org/simple/package/: 404 Client Error: Not Found for url: https://pypi.org/simple/package/ - skipping
  Getting page https://[email protected]/pypi/package/
  Found index url https://private-repo.io/pypi
  Looking up "https://private-repo.io/pypi/package/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): private-repo.io:443
  https://private-repo.io:443 "GET /pypi/package/ HTTP/1.1" 404 10
  Status code 404 not in (200, 203, 300, 301)
  Could not fetch URL https://[email protected]/pypi/package/: 404 Client Error: Not Found for url: https://private-repo.io/pypi/package/ - skipping
  Given no hashes to check 0 links for project 'package': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement package (from versions: none)
Cleaning up...
Removed build tracker '/private/var/folders/rb/rg18vqd16lxg47cm1kvgh3yh0000gn/T/pip-req-tracker-_5f9p1zv'
ERROR: No matching distribution found for package
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 134, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 211, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/index.py", line 1228, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for package

@bmckalla did you try with https://:[email protected] instead ? see: https://github.com/pypa/pip/issues/6775#issuecomment-514272956

@TTRh

Having the same issue, but I think the:should go after the token. At least, that is a workaround that works for me now : https://token:@private-repo.io

@TTRh @jgspiro Putting : after the token worked, thanks!

For what it's worth, it looks like twine uploads do not work if you use a username with the %40 instead of a @.

@leviable ~I would recommend filing an issue in twine's tracker if one isn't already filed and link to this issue.~ Looking more closely at your original post, when you pass a username to twine, you're not doing that as part of a URL but rather as a stand-alone value -u <username>, so URL-quoting won't work in that context. You'll need to do some extra processing of the username to use the same setting for both (see e.g. the comment following this one for ideas).

just FYI, url quote the password works for us.

password='passwordwith@$$!!'
encoded=$(python3 -c 'from urllib.parse import quote;print(quote("'${password}'"))')
export PIP_EXTRA_INDEX_URL='https://username:'${encoded}'@nexus.hostname.com/repository/pypi-all/simple'
pip install your_package

@TTRh

Having the same issue, but I think the:should go after the token. At least, that is a workaround that works for me now : https://token:@private-repo.io

Is this considered a bug or intended behavior (not saving credential when there's no : with token)?

Some modifications to download.py here seems to solve the problem.

It's a bug that still needs to be fixed I believe.

I landed at the same patch as https://github.com/booleand/pip/commit/9761f46520b94c027b0e6732223e391088f745a5#diff-34481ab2fefef95e8820a46e86004ae7

@booleand would you be willing to submit a patch?

@pradyunsg Yes I am

@booleand it'd be great if you'd file a PR. :)

thanks @pradyunsg. filed a PR #6795

Following https://github.com/pypa/pip/issues/6775#issuecomment-514284653, I've opened #6796 for the "one element" auth in the package URL.

Thanks @slafs. In that case I think we can close this one.

How about documenting this case in the docs (including with an example) since it seems common and can trip people up?

@cjerdonek, added #6797 to track that action.

I've posted a PR (#6818), that fixes both of our URL handling issues (this and #6796).

I'd appreciate if folks from this thread could take #6818 for a spin and let me know if it works.

To install pip from that branch, into your local environment, run:

pip install https://github.com/pradyunsg/pip/archive/fix/URL-authentication-handling.zip

I don’t think that PR fixes this issue? I thought this issue was about usernames that include an @ when a password is present (it doesn’t have to do with None / non-None), and the other issue is about no password being present. But I don’t see the former among the test cases. (There would need to be two unescaped @ symbols in the test cases.) Also, I thought fixing the former would mean we’d have to “fix” / rewrite the URL, which I don’t see being done in the PR.

I don’t see the former among the test cases

Ah, yes. My bad, I omitted the test for it. I've gone ahead and added that case to the test suite -- running it locally confirms that it extracted the email address correctly.

"http://[email protected]:[email protected]/path" -> ("http://example.com/path", "[email protected]", "password")

I thought fixing the former would mean we’d have to “fix” / rewrite the URL, which I don’t see being done in the PR.

Ah, yes. Me too initially.

However, I looked at the diff of requests and urllib3 vendors and didn't see anything in a code path that affected us directly here. I decided to try seeing if reverting to the changes from the original structure worked and it seems like it did.

My understanding now, is that it's got something to do with requests' default handling of URLs that contain auth information and (somehow) we're circumventing those issues in our Session subclass.

My understanding now, is that it's got something to do with requests' default handling of URLs that contain auth information and (somehow) we're circumventing those issues in our Session subclass.

No, if you look at my comment above where I included the traceback for this issue, you'll see that we pass the original URL: https://github.com/pypa/pip/issues/6775#issuecomment-514313192
And your PR doesn't "fix" / rewrite the original URL, so that I wouldn't expect that behavior to change in your PR.

Just to be sure, I retried that command in your PR and confirmed that the same error still happens.

Here is my comment above where I talk about one approach to fixing the current issue: https://github.com/pypa/pip/issues/6775#issuecomment-514322734

Right, makes sense to me. I think I was mixing issues in my head.

How about documenting this case in the docs (including with an example) since it seems common and can trip people up?

It would also make sense to abort in this situation, and print a warning if there's an unescaped "@" in the URL. It's probably more discoverable than the documentation and is friendlier than having it in the documentation.

Summarizing the state of affairs in pip 19.2, for my stupid self:

• Authentication with only a token (i.e. username, no password) is broken -- pip's current logic assumes both parts would be present. That's what #6796 is for.
• Authentication with a username that's not URL-encoded does not work anymore -- pip 19.2 fails with a traceback, pip 19.1.1 accepts such URLs. The source of this issue is that urllib3 has become more strict in parsing such URLs.

Hi, we are facing the same issue with pip 18.1, urllib3(1.24.1) & Python2.7 but in the password section.
https://pypi.org/simple --extra-index-url http://user:password@@124.4.56:8086 --trusted-host 124.4.56
https://pypi.org/project/urllib3/ doesn't list any change related to the RFC 3986 specification above. Also @cjerdonek @pradyunsg could you please recommend stable versions of both the packages so that we can rather install them.

Released pip 19.2.2 containing a fix for the tokens needing a :.

For folks facing issues with @ in their passwordds, the suggested path forward is to escape the @ in the URLs.

Edit: Doesn't seem to be related to the '%', as I changed my password to "password" and got the same result.

Another kink in this: my old generated password (used to) end in "%".

On version 19.2.2

Replacing the % with %25 or leaving it alone, I see the same error:

(Assuming my password was: password%)

ERROR: Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/Library/Python/2.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/Library/Python/2.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Library/Python/2.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/Library/Python/2.7/site-packages/pip/_internal/operations/prepare.py", line 134, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/Library/Python/2.7/site-packages/pip/_internal/req/req_install.py", line 211, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Library/Python/2.7/site-packages/pip/_internal/index.py", line 1201, in find_requirement
    req.name, specifier=req.specifier, hashes=hashes,
  File "/Library/Python/2.7/site-packages/pip/_internal/index.py", line 1183, in find_candidates
    candidates = self.find_all_candidates(project_name)
  File "/Library/Python/2.7/site-packages/pip/_internal/index.py", line 1111, in find_all_candidates
    if self._validate_secure_origin(logger, link)
  File "/Library/Python/2.7/site-packages/pip/_internal/index.py", line 998, in _validate_secure_origin
    origin = (parsed.scheme, parsed.hostname, parsed.port)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urlparse.py", line 113, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'password%%40mydomain.com'

or

ValueError: invalid literal for int() with base 10: 'password%25%40mydomain.com'

Can you try encoding only the credential portion of the URL - e.g. password%[email protected]?

Switching to password%[email protected] does allow me to proceed.

There's nothing more actionable here. As a closing note, I'll summarize this issue and make a call-to-action.

What happened?

pip 19.2 now requires credentials to be URL quoted. Thus, characters like @ and % need to be URL quoted (like %40 and %25).

Thus, the following index-urls will no longer work:

https://[email protected]:[email protected]/
https://user:password%@mydomain.com/

The equivalents that do work, are:

https://pradyunsg%40email.com:[email protected]/
https://user:password%[email protected]/

Why?

pip (and requests) use urllib3 for a lot of the underlying URL handling. urllib3 has introduced stricter URL parsing in the version that pip upgraded to in 19.2. Since this strictness was only "caught" after the release, pip's developers decided to not to circumvent the stricter parsing. The workflows for affected users had already broken and none pointed out that it'd be particularly difficult for them to quote their credentials properly.

How can you help?

pip's developers do not want to be breaking their users' workflows. However, this is difficult since pip is a project maintained entirely by volunteers, with a legacy codebase that makes on-boarding additional contributors difficult.

There is a substantial lack of developer time on pip, especially given the impact that changes in it, can have on users. To help us avoid issues like this and make more substantial improvements to pip, consider funding Python Packaging improvements. Funding such projects related to pip, would allow current maintainers be able to work on pip for more time, and bring in additional experts (UX experts, project managers, technical writers etc) to make pip better.