Pip: Error with `import pip` in pip 9.0.2

I can confirm this as well. Just about to file same issue.

This is a dupe of #5079 -- importing pip is not a supported use case (and never has been).

Come on.. making a huge breaking change in a single x.x.PATCH version bump? For a top level method that's called "main"? I think that's pretty poor form..

This also broke Anaconda and I came up with the same solution as @Miserlou:

https://github.com/ContinuumIO/anaconda-issues/issues/8911

Errors are being reported across a lot of projects.

This error also bothered me when I was trying to use pip to update my anaconda-navigator.

Is there any chance of us getting a 9.0.3 that fixes this - ideally quite soon?

This is already affecting lots of major projects (Anaconda, SpaCy, Zappa), and there are more than 850k+ uses of this on GitHub alone that are now broken by this supposed "patch" version update.

Can you at least revert this massive change in 9.0.3 and then _announce_ to downstreams your intention to change this behavior for a future 10.x.x or at least 9.x.x release?

We don't want to use an older version either, but that's exactly what we ended up doing. 9.0.2 does not exist inside our CI environment, at least for now.

Also seeing this hit in some OpenStack projects.

Pip 10 is due out in about a month. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage.

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

Proposals for reintroducing some level of API support are certainly an option (see #5080 for example) but at this stage, it's too late for any such changes to make it into pip 10.

With no warnings being raised by the code for those using the old way, nothing denoting this as "internal" with starting with _, and this just being a bugfix bump, I actually think it's quite easy to see this as a sudden, unexpected breakage.

Yes, this is the type of change that people could expect from a MAJOR version update. That would be fine.

Unfortunately, this massive change came in a PATCH update, which is _supposed to fix things, not break them_. This is the exact opposite of semantic versioning. And, as a result, we are seeing massive damage across the Python ecosystem.

You should revert this change ASAP in another PATCH update, then have the major breaking change come with the MAJOR version update. Now that you've already broken everything for everybody prematurely, I think they will be more aware that the actual major change is coming.

I also think it's a cop out to say that this was documented and announced already, considering that it was _not_ documented in the stable documentation, and that the announcement said that the break would happen _for the major version_, not for in a patch a month prior.

Please, just save everybody a massive headache, revert this problem and start actually adhering to semantic versioning.

@Miserlou OK, I take your point - we targeted the internal renaming change for pip 10 because it's a major version. I don't really know the drivers for the patch release - @dstufft pinged me about it and it's apparently to avoid significant breakage for Mac OS users when an imminent deadline for TLS support hits us, which is before the pip 10 release. We expected the problems to be significant enough to warrant a short-term patch release. There was certainly no intention of breaking existing usage.

I have to leave the decision of a follow-up 9.0.3 to @dstufft - I don't have the details of what went into 9.0.2 or know whether it's even possible to identify how to fix this issue. And I can't judge whether pulling 9.0.2 altogether will be an overall benefit, as I have no idea how many people are affected by the Mac OS issue. I understand that (at least for some people) pinning to 9.0.1 is a solution, so that may end up being the least bad option.

The macOS TLS issue will affect everyone using a system Python on macOS<10.13

I have to leave the decision of a follow-up 9.0.3 to @dstufft

I'm of the same position as @pfmoore.

Workaround, if you have it available to you, is to check the order of imports and test moving importing pip above other package imports (specifically, importing pip _before_ importing requests seems to solve some cases). (Note that this still implies use of pip's internals, which isn't officially supported.)

Same issue here with pip 9.0.2 in a gitlab-ci with docker python 3.4: KeyError

  File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'

FYI, 9.0.2 was released with broken builts:

Travis-CI Reference: https://travis-ci.org/pypa/pip/builds/354616774?utm_source=github_status&utm_medium=notification

Granted the errors might be unrelated, though this just smells like a "rushed release"...

@pfmoore

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

pip always was documented as NOT having a consumable python api, i'm disappointed in people that even at that point try to turn around the "told you so since quite some years" karma

@RonnyPfannschmidt This is like saying "We've told you a 100 times you are not allowed to drive over the speed limit" and then enforcing the speed limit by replacing all engine-cars with flintstone-cars, so people can not go over the speed limit anymore.

you just demonstrated perfectly that turning around of words that so bad - the telling was "dont rely on it it will break" - now it broke and suddenly the project that told you before that it will break is at fault

that just putting blame because you dont like to be at fault

@RonnyPfannschmidt

that just putting blame because you dont like to be at fault

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip. Maybe, I am... I should have screened those 3rd-party packages for those errors and submitted an issue or even better a PR.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

If a third-party package you use breaks because it relies on undocumented, un-guaranteed internal APIs, it seems to me that you should file a bug against that package.

There's an easy-to-discover line between what parts of pip are and aren't intended for third-party consumption, and I don't support trying to force its developers to maintain internal APIs that never should have been consumed by third-party packages. I don't support treating this as the fault of the pip developers and sending angry users their way. If you want to go be angry at someone, go be angry at any packages you use which relied on internal APIs, and push for their maintainers to fix their own code.

@anx-ckreuzberger

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

While I understand the frustration here, I'm getting increasingly annoyed by the willingness of people to blame the pip maintainers for things we never promised.

If you want to make accusations like that, please back them up with

1. A pointer to a statement from any of the pip maintainers that we support use of the pip internal API in user code.
2. A pointer to a statement from any of the pip maintainers that pip uses semantic versioning.

I don't think you'll find that evidence...

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip.

No, but accusing the pip maintainers rather than those projects is not acceptable. We're trying to help you but can't be held responsible for what those projects do. Air your grievances with them.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

We tried to provide a warning. We sent announcements out months ago, we added docs explaining the situation, and we've consistently replied to issues on the tracker stating that use of internal APIs is unsupported. Adding deprecations is far from being as easy as you suggest, as pip itself would spew those warnings (or there'd be a way for pip to switch them off, which others would just copy - we're already hearing of people planning to just import pip._internal, so even that change won't stop them :disappointed:).

As to "fixing" this, I'll happily admit that 9.0.2 was released quickly to address a pressing issue, and we didn't anticipate this problem. Maybe releasing a 9.0.3 with a lifespan of 2-3 weeks is a reasonable thing to do, I don't think so myself, but I've stated that we'll consider that. I can't personally agree to do it, as I'm not the one involved in the 9.0.x releases. I'm working on pip 10, which will make all of this debate irrelevant, so this is likely my last word on this issue - I need to focus on things related to that release.

My personal advice:

1. Pin to 9.0.1 if you're affected by this and need a workaround now.
2. Be prepared for pip 10, when all the dependencies you currently have that are broken because they use pip internal APIs will break again. Push those dependencies to take action on the information we've been giving for months, and be glad you got advance warning that it would happen.
3. If pip 9.0.3 gets released, remove the pin.
4. Please, test the pip 10 beta when it comes out, and report any issues to the relevant parties (3rd party projects if they are calling pip internally, us if you are calling pip yourself).

I've experienced the same problem with pip 9.0.2 and Python 2.7.14 within a docker container.
I cannot reproduce the problem on my dev machine outside a docker container though.
I searched for a pip import (grep'd for import.pip, from.pip, \'pip, \"pip) but couldn't find anything.
We at Plone are using a mechanism to automatically include configurations from packages included in a setuptools setup.py file, which sounds suspicious - but again - this one is just using __import__ and nothign from pip AFAIcansee.

This the relevant part of the traceback:

  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/zcml.py", line 51, in includeDependenciesDirective
    info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml'])
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/dependency.py", line 26, in includableInfo
    module = resolve(dotted_name)
  File "/home/plone/.buildout/shared-eggs/zope.dottedname-4.2-py2.7.egg/zope/dottedname/resolve.py", line 39, in resolve
    found = __import__(used)
  File "/plone/buildout/lib/python2.7/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/plone/buildout/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/plone/buildout/lib/python2.7/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/site.zcml", line 15.2-15.55
    ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/package-includes/002-bda.aaf.site-configure.zcml", line 1.0-1.56
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure.zcml", line 11.2-11.48
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure-dependencies.zcml", line 10.2-10.39
    ZopeXMLConfigurationError: File "/plone/buildout/src-addons/plone.app.mosaic/src/plone/app/mosaic/configure.zcml", line 9.2-9.39
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.tiles-3.0.3-py2.7.egg/plone/app/tiles/configure.zcml", line 18.2-18.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.z3cform/plone/app/z3cform/configure.zcml", line 10.2-10.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.widgets/plone/app/widgets/configure.zcml", line 12.2-12.41
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/Products.CMFPlone-5.1.0.1-py2.7.egg/Products/CMFPlone/configure.zcml", line 15.2-15.46
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.contenttypes-1.4.9-py2.7.egg/plone/app/contenttypes/configure.zcml", line 10.2-10.37
    KeyError: 'pip._vendor.urllib3.contrib'

As far as I understand just importing pip - without using it - breaks already. This is not a good citizens behavior in Python land. Not supporting using it/ not providing a public API is one thing, just breaking on import another. This affects a lot of auto-inspecting code.

Think of it this way: pip is a command line utility, not a library. The fact that it's written in Python is irrelevant. That's the reality of the matter.

@pfmoore

While I understand the frustration here, I'm getting increasingly annoyed by the willingness of people to blame the pip maintainers for things we never promised.

Think of it this way: pip is a command line utility, not a library. The fact that it's written in Python is irrelevant. That's the reality of the matter.

The bottom line is it really doesn't matter what you promised. What matters is what you did and what the resulting situation is. Maybe you thought of it as a command line utility. But you released it as a library. That is the reality of the matter:

You've released a library with Feature X. People started using Feature X. Sure, you said "guys, please don't use Feature X". But you kept releasing it with Feature X. For years. And so people kept using it. Tens, maybe hundreds, of thousands of projects, big and small, use it now. Suddenly you remove it in a minor update without adequate warning.

What do you think happens next?

In the short term, people will just pin the old version and not unpin it unless there's a good reason.

In the long term... well, depends on how many people decide it would be more cost effective to replace you than to fix everything you've broken.

@pfmoore do you see something suspicious in the traceback I referenced above? There is no use of any pip internals AFAIK. It's interesting that most people have this problems within docker containers.

@ all, accusing the maintainers for broken code doesn't help with solving the problem.

Think of it this way: pip is a command line utility, not a library. The fact that it's written in Python is irrelevant. That's the reality of the matter.

Fact: It can not be imported, if imported even by auto-inspecting code for some reason: all breaks. I suppose this happens in @thet's case.
Conclusio: pip must isolate itself from the global or current virtualenv/venv Python namespace it installs packages into. That way it does not pollute it and not even import by accident ever happens.

First of all, sorry if I accused the maintainers of the repo for faulty code. Any accusation were made out of frustration, and, if at all, point to the fact that IMHO the 9.0.2 release is by the definition of semver a patch release (though @pfmoore made it clear that pip is not necessarily using semver - which is another issue for another day I guess).

@MikeHart85

In the short term, people will just pin the old version and not unpin it unless there's a good reason.
In the long term... well, depends on how many people decide it would be more cost effective to replace you than to fix everything you've broken.

Pretty much. I mean look at JavaScripts package managers: npm, bower, yarn, whatever is released next ~year~ week

No, but accusing the pip maintainers rather than those projects is not acceptable. We're trying to help you but can't be held responsible for what those projects do. Air your grievances with them.

I understand that my (our?) grievances within this issue might be misdirected. Though you have to admit that it is really hard to convince anyone that this is the fault of 3rd-party-maintainers.

We tried to provide a warning. We sent announcements out months ago, we added docs explaining the situation

Was this for pip 9.0.2 or for pip 10? If it was for pip 9.0.2, I would find it nice if there was a note in the Changelog. Anyway, thank you for being very proactive in the development of pip 10 and sending out announcements about not using import pip, that's good! Keep it up!

Adding deprecations is far from being as easy as you suggest, as pip itself would spew those warnings (or there'd be a way for pip to switch them off, which others would just copy - we're already hearing of people planning to just import pip._internal, so even that change won't stop them disappointed).

I don't think this would be too hard... You already have this in the __init__.py:

if __name__ == '__main__':
    sys.exit(main())

How about just adding an else?

if __name__ == '__main__':
    sys.exit(main())
else:
    logger.warning("You are importing PIP as a Python library. This behaviour is deprecated and should no longer be used. We will break the API with version 10.0!!!111eleven Please see https://pip.readthedocs.org/some_link_where_you_explain_this.html")

edit: You can even raise an exception here if you want to be "strict" about it, and add similar statements to the submodules.

The fact that people will work around this, is something else. If you want to hack a library, you can always reverse engineer it. You can always find a way around it. But you should not consider all the ways around it within your design decisions.

In the long term... well, depends on how many people decide it would be more cost effective to replace you than to fix everything you've broken.

I always find this "threat" amusing. It comes with it a fundamental misunderstanding about the amount of effort that is put into something like pip and how much that really costs (and how little people are willing to actually pay for it). If you feel you're able to do better, then by all means do so, I (and I assume the others) welcome it. A non trivial amount of effort in the past years has been on documenting and designing standards with one of the explicit goals being to make it possible for such a thing to happen.

It's important to keep a sense of perspective about these things. There are.. less than 15? 20? people commenting about how this broke them (though there is always a "tip of the iceberg" problem with these metrics), meanwhile pip 9.0.2 is already the second most used installer in the past week (counting days it hasn't even existed for) and has downloaded 17 million files from PyPI since it has been released. Looking at just the last day, it's 80% of the way to overtaking pip 9.0.1 as the most used installer on PyPI.

So while I recognize that this broke things for some people, it is a relatively small number of people doing either unsupported things, or in fairly edge case situations.

IF I can find the time, I may try to cut a 9.0.3, primarily to solve the case that @thet has run into where it is just an auto-importer trying to import things, and they're not actually attempting to use pip's internal APIs in any way. If that also ends up resolving the behavior for people using pip's internal APIs I'm not going to go out of my way to break them specifically, but if it doesn't I'm also not going to go out of my way to fix them specifically either.

Please do note that it takes many hours to release a 9.0.x at this point (things have diverged in master enough that it requires some finagling to do a release) and that doesn't count the time required to debug and fix the actual issue. If you want to increase the chance that this happens, doing that debugging and fixing and providing a patch (or a branch off of the 9.0.2 tag in your own fork) is the best way to do so.

First of all, sorry if I accused the maintainers of the repo for faulty code. Any accusation were made out of frustration

Thank you. Frustration is high here, and I understand that.

How about just adding an else?

Good idea - I wish we'd thought of this some time ago. Of course, there's no real point now, as the internal namespace will be reorganised in pip 10, so it's too late. I'll certainly remember this trick for the future.

Was this for pip 9.0.2 or for pip 10?

For 10.0. There was no original intention to release a 9.0.2, we only did so as an emergency release so that users of Mac OS wouldn't lose all access to PyPI when the TLS changes come into effect.

@dstufft has responded here much more fully, so I think this is as resolved as it's likely to be for now.

I always find this "threat" amusing.

It's not a threat at all, apologies if it came across that way.

It's an observation about something that happens all too often in the software world. People don't work with your promises or intentions. People work with the product you deliver. If your product has a feature that many people come to depend on, you can no longer just yank it out from under their feet. If you do, they start seeing your product as broken. If the only explanation you provide is "well, we never promised this anyway" then they start seeing your unwillingness to acknowledge their concerns as a liability.

The more this happens, the bigger the desire and demand for an alternative. Sure, people underestimate how much work that takes. But that only makes them more, not less, likely to start working on one. Once that ball is rolling, it has a life of its own.

Alternatives aren't always a good thing. They tend to make things messy. Personally, I'd prefer a single package manager. For everything. But I'll settle for a single package manager for Python.

When people report you've broken a feature, intended or otherwise, consider briefly explaining why you had to break it, or why keeping it wasn't viable, rather than dismissing concerns with "this wasn't supported all along".

Hey @dstufft , thanks for chiming in. This thread is getting pretty hot!

First of all, I thank you for your work! I know that maintenance is a endless, thankless task, and I imagine it only gets worse when you manage a project as large and popular as pip!

Now, to the issue at hand: Although twenty _maintainers_ have reported this problem so far, I know that is already affecting thousands of _people_ - some of the projects affected are massive in their own right: Anaconda, OpenStack, SpaCy, Zappa, and have many tens of thousands of users . I know our Slack channel is flooded with discussion about this. (Literally, as I was typing this, a new issue appeared.)

Clearly, there are many important Python projects which need the ability to programmatically inspect their environments. This is a totally reasonable thing to need to be able to do. Furthermore, the documentation never warned against doing this - and still doesn't! (Click the Docs link from this repo's README!)

I think the situation so far is:

• Given the format of the version string, we all believed that pip maintainers are following semantic versioning
• The pip maintainers released a "patch" which introduced a massively breaking change
• This change was unannounced and undocumented - and I assume, unexpected and unintended
• Now, just calling import pip immediately breaks a program, which is extremely developer-hostile
• This is causing major headaches across the Python ecosystem
• The documentation does not (and _still_ does not - click on the Docs link in this repo's README) warn against using pip programmatically.
• There was no announcement that import pip would cause this damage in a PATCH update - that announcement came for a version that _hasn't been released yet_.
• This change wasn't even mentioned in the Changelog
• We don't want to replace pip or the pip developers! We love you!
• ..but we don't want this type of thing to happen again!
• ..so we do want semver to be followed!
• We would really, really like another PATCH which undoes this major breaking change!

The need for a programmatic way to inspect an environment in a pip>=10.0.0 world is a topic for another discussion, but the fact of the matter at hand is that we were never told we shouldn't use pip programmatically in the pip<=9 world, and there was a massive, unannounced change, and we'd really really like to see it reversed because it is negatively affecting thousands of people.

Thanks again,
Rich

First of all: Thank you at the pip maintainers for their efforts and the insights within the project. I believe this really helps others understanding the issue (though it might be worth to write a summary blog article about it).

@dstufft

It's important to keep a sense of perspective about these things. There are.. less than 15? 20? people commenting about how this broke them (though there is always a "tip of the iceberg" problem with these metrics), meanwhile pip 9.0.2 is already the second most used installer in the past week (counting days it hasn't even existed for) and has downloaded 17 million files from PyPI since it has been released. Looking at just the last day, it's 80% of the way to overtaking pip 9.0.1 as the most used installer on PyPI.

I'm pretty sure that metric is heavily biased by all the automated pip install pip --upgrade commands from Continous Integration/Delivery systems (they should use caches, and therefore not need to re-install packages from pypi all the time; but at the same time, one should also not do import pip... this is how the IT world works).

The fact that (less than) 15 people have complained about this should show tell two things:

1. I don't think that many people use 9.0.2 (in production) yet, some may still be trying to debug this issue and will "temporarily" fix it by using pip 9.0.1 instead until it is resolved (or forever...) - also, some people might not have noticed that something is not working yet...
2. There are people already talking about it within 2 days of the release. More people will experience this issue. Wait for 2 weeks, and you could end up having 100 people complain about it (e.g., when they finish a sprint and deploy to QA/staging). At this point in time you really don't know how many people will be affected by this change.

Anyway, talking and arguing about this issue gives people a good example on why one should never rely on internal APIs, and hopefully some 3rd-party package maintainers will update their projects because of what has been talked here.

Clearly, there are many important Python projects which need the ability to programmatically inspect their environments. This is a totally reasonable thing to need to be able to do. Furthermore, the documentation never warned against doing this - and still doesn't! (Click the Docs link from this repo's README!)

It's not clear to me why you'd have to warn against this. The use of pip as anything other than a command line tool is completely undocumented. Searching the documentation I don't see any reference to importing pip at all. It's like complaining because you linked against some .so used by Chrome and they broke ABI compatibility.

The usual interpretation of SemVer is that it applies to the supported, public interface (in this case, the CLI), not the undocumented internals. Anyone using the internals should be pinning their pip versions anyway, since they are subject to arbitrary change.

There is nothing to reverse really. Backporting fixes to prevent a significant chunk of users on macOS from being unable to access PyPI in the near future has a bug when applied to the 9.0.x code base that appears to only express itself under unsupported conditions. The only path forward is to do additional work to resolve that bug in the 9.0.x series. Like I said if I can find time I’ll try to do it, if you want to increase the chance of it happening then provide a patch.

The documentation doesn’t warn against it because it is not possible to provide an exhaustive list of things you could possibly do with pip but which is not supported. Instead we rely on the fairly common approach of documenting what is supported and anything not documented should be assumed to be unsupported (and if you want to rely on something that isn’t documented, opening up an issue asking for it to documented and thus supported).

Longer term we are likely to move towards a date based release scheme to (hopefully) remove any confusion about whether we are semver or not.

Sent from my iPhone

On Mar 20, 2018, at 11:02 AM, Rich Jones notifications@github.com wrote:

Hey @dstufft , thanks for chiming in. This thread is getting pretty hot!

First of all, I thank you for your work! I know that maintenance is a endless, thankless task, and I imagine it only gets worse when you manage a project as large and popular as pip!

Now, to the issue at hand: Although twenty maintainers have reported this problem so far, I know that is already affecting thousands of people - some of the projects affected are massive in their own right: Anaconda, OpenStack, SpaCy, Zappa, and have many tens of thousands of users . I know our Slack channel is flooded with discussion about this. (Literally, as I was typing this, a new issue appeared.)

Clearly, there are many important Python projects which need the ability to programmatically inspect their environments. This is a totally reasonable thing to need to be able to do. Furthermore, the documentation never warned against doing this - and still doesn't!

I think the situation so far is:

Given the format of the version string, we all believed that pip maintainers are following semantic versioning
The pip maintainers released a "patch" which introduced a massively breaking change
This change was unannounced and undocumented - and I assume, unexpected and unintended
Now, just calling import pip immediately breaks a program, which is extremely developer-hostile
This is causing major headaches across the Python ecosystem
The documentation does not (and still does not - click on the Docs link in this repo's README) warn against using pip programmatically.
There was no announcement that import pip would cause this damage in a PATCH update - that announcement came for a version that hasn't been released yet.
This change wasn't even mentioned in the Changelog
We don't want to replace pip or the pip developers! We love you!
..but we don't want this type of thing to happen again!
..so we do want semver to be followed!
We would really, really like another PATCH which undoes this major breaking change!
The need for a programmatic way to inspect an environment in a pip>=10.0.0 world is a topic for another discussion, but the fact of the matter at hand is that we were never told we shouldn't use pip programmatically in the pip<=9 world, and there was a massive, unannounced change, and we'd really really like to see it reversed because it is negatively affecting thousands of people.

Thanks again,
Rich

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@Miserlou

Given the format of the version string, we all believed that pip maintainers are following semantic versioning

As someone who is strongly opposed to semver, can you let me know what format my version strings should be in to eliminate this confusion? "Three dotted numbers" does not imply "strictly follows semver" to me, so this assumption comes as a surprise.

In response to this comment: Whether or not pip adheres to semver, or promises to adhere to it, the use of a major.minor.micro versioning scheme still carries an implicit promise of some kind of restraint around micro releases. If a compatible release pin like ~= 9.0.1 isn't enough to shield users from unexpected breaks in backward compatibility, the only safe alternative left is plain version matching. If there's no intention to support anything more than version matching, then pip might as well switch to a Chrome-style version scheme that has only one monotonically increasing component.

Edit: I see that @dstufft already proposed moving in this direction by adopting date-based versions. I think that would be an unfortunate piece of collateral damage to result from this incident.

So yes, as users of a volunteer driven software project, we need to balance our expectations with an appreciation for the nature of the user/maintainer relationship. It's also clear that this release wasn't intended to make import pip suddenly start to fail. On the other hand, though, I think it's reasonable for people to be frustrated about this sort of regression happening in a micro release, and releasing a 9.0.3 version that fixes the issue seems like a reasonable remedy.

As an aside, I can reproduce this in a virtualenv (2 or 3, it doesn't matter) with the following steps:

virtualenv -p python3 /tmp/pip
/tmp/pip/bin/pip install -e path/to/clone/of/pip/9.0.2
/tmp/pip/bin/pip install requests
/tmp/pip/bin/python

Then, within the python shell:

>>> import requests
>>> import pip

If requests hasn't been imported already, then import pip succeeds.

unexpected breaks in backward compatibility

Please indicate to me where import pip was documented to be a stable API that falls under the backwards compatibility promises we make? I would like to make sure that I remove such documentation since we emphatically do not support that use.

Unless you're of the opinion that absolutely nothing can break, even unsupported, untested uses. In which case you probably want to pin == because it is entirely unknowable what all you may be using at that point, and every change potentially becomes a breaking change for someone (obligatory xkcd).

pip is Python's package manager. Python is a programming language. People need to inspect their environments programmatically. 1+1=2.

It was perfectly reasonable for people to assume that this was the correct way to do it. There wasn't - and there _still isn't_ - anything in the documentation saying _not_ to do this. As a reminder, there are over 700,000 applications which came to this same conclusion! Also - there isn't any other way to do it!

Just because something isn't documented in ReadTheDocs doesn't mean that it is forbidden - we all encounter and use projects every day that simply provide code this way.

If anything, the fact that pip even _has_ a command line interface at all seems to be a good indication that it can be used as a library, since it already has a Python client!

Furthermore, we're not talking about private APIs namespaced with a _, as is convention, or even any specific public method, we're only talking about _just calling import_ which causes this catastrophic failure.

There wasn't - and there still isn't - anything in the documentation saying not to do this.

https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

If you click on the "Docs" link of _this very repo_, you don't get to that page. Nobody is referencing latest. Your own links to documentation don't link to latest. There is no way to get to that page. Everything goes to stable, which does not include that section.

@Miserlou I think that note is a kindness for people who somehow think that they should be importing the internals of a command line tool just because it happens to be implemented in Python and those internals are importable.

I get that you have a different interpretation of the public interface of pip than me, the developers and most people who think of pip as a CLI program, but what is the actual harm of this? Just pin pip != 9.0.2 and be done with it.

It's obvious we're all on the same page at this point as to what is and is not part of the supported, public API, and there's nothing that can be done now to somehow retroactively put everyone on notice.

To be honest, the maintainers of pip already stated that IF time permits, they will try to fix this issue, though everyone is encouraged to file a pull request to speed things up.

I think for now all we can do is to make 3rd-party libraries aware of this issue, and point them to this issue. Long term 3rd-party libraries need to fix this issue anyway, and hopefully in a way where they do not import pip, but call pip via the commandline (with subprocess or similar python commands).

I believe that this discussion is unlikely to be productive nor is there really anything additional to be said. There is:

• An adequate description of the problem.
• Possible work arounds that someone could employ if they run into it.
• Rationale for why we don't consider this a breaking change as per our backwards compatibility guidelines.
• A statement that I will attempt to find time to fix it (though no promises about it).
• A Call to Action that one can be done if someone affected by this wants to make it more likely that a 9.0.3 exists with the fix.

At this point the discussion serves no real purpose except to further frustrate everyone involved, so I am bowing out of this issue for now. This issue will be closed either upon the release of 10.0, or 9.0.3. If someone does put effort into the Call To Action, they are welcome to either open a pull request, or submit a diff on this issue.

In order to make this do-not-import-pip mantra more visible, what about renaming the namespace to "_pip". This indicates that the whole is not for public use on a name level.
Also, it would be easier to tell auto-inspecting code not to look at stuff starting with underscore. Well, latter would need changes in the involvement inspection code too. But at least there's a chance to automate by it (by convention) without managing blacklists.

Ok, last thing I swear-- pip 10 already moved all relevant code to pip._internal (we didn't use _pip because that would break python -m pip, which is supported).

Can anyone verify if https://github.com/pypa/pip/pull/5100 solves this for them?

Scratch that, I think #5100 is wrong, could you verify https://github.com/pypa/pip/pull/5101 instead please.

@dstufft I can confirm, the fix in #5101 solves the problem for me.

Thanks @dstufft for your time in addressing this. I will work with the Anaconda teams to communicate this issue and help them move away from importing pip.

For the record in this thread, #5101 solved my issue as well.

Ditto, #5101 allows the import to work inside our tool. Caveat emptor: I haven't tested the patched pip nor our tool extensively.

This should be fixed in 9.0.3.

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

Hooray! Thanks for this!

I'd also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

pip list --format=json?

Firstly thank you all contributing to pip people for it totally covers all of my use cases with a few user-friendly options and arguments.

Due to this "surprisingly widely adopted and useful undefined behavior," do we need to make a piplib as libgit2 for git? Please note that libgit2 is not sharing any code with git, and is maintained by other group from git's. And it is good for git ecosystem. Maybe piplib will cover some interesting use cases which we didn't expect.

Here is a similar story: https://public-inbox.org/git/1267957655.3759.29.camel@mattotaupa/T/

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there's no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it's probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don't go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

@sruggier The packages metioned are all good, and WheelFile.install() also needs more promotion. But the one-stop service pip.main() provided is irreplaceable with all of the above. It's still worth trying.

The most important is that I hope these needs are held by another project, and pip10 will arrive smoothly without extra guarantees. The deprecation and minimizing the code base are the whole points for a major release.

Concrete implementation details for a permanent infrastructure "software" is ridiculous. You can't judge maintainers by the documented abuse case and hold back the wheel of pip.

If you insist to use pip as a lib, a lot of assumptions will need to be reconsidered.

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn't promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it's not hard to implement in a 3rd party module.

I appreciate the work you all do on pip, and know it's not easy. But for the record:

I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10

spaCy has moved away from import pip. But some people are still using spaCy 1, which did import pip --- and for obvious reasons, didn't pin pip down to a patch release. If the change had come in at v10, our old versions wouldn't have been affected. We've just issued a hotfix to address this.

proper instruction on how to programmatically inspect python environments in a pip10 world

What is PyPA's position on distlib? Pip is obviously using it in some capacity, but so too is it using packaging, which distlib purports to deprecate.

If there isn't an official position, then at least current thoughts and opinions from pip's core maintainers would be much appreciated. If there are relevant discussions on this topic elsewhere, a simple pointer to other discussions is good enough for me.

I'm not aware that distlib deprecates packaging. It mentions "distutils2/packaging" but distutils2 was something different.

My personal view is that both distlib and packaging are perfectly reasonable things to use. You should evaluate them yourself to confirm they meet your needs, obviously, just like any other dependency you rely on.

Maybe deprecates is too strong of word then. The source of my current understanding:

https://distlib.readthedocs.io/en/latest/overview.html#distlib-evolved-out-of-packaging

Ah, that's a different "packaging" - it was the proposed stdlib module that was intended to replace distutils. It's completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that :) See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.