Hey @KleinerNull!

I have no clue if this has something to do with the upgrade itself or with pyenv as environment.

I think that this is an environment issue. I suggest you open an issue over at pyenv as I think the folks there would be in a better position to comment on this/fix it.

@pradyunsg

I opened a issue at the pyenv repo.

We are suffering from this one as well, it's broken our pipeline. Operation being run:

 11 #upgrade pip and install uwsgi
 12 pip install --user --upgrade pip
 13 pip install uwsgi

Ubuntu 16.04
Python3.6

We are encountering the same trouble.

// in host
$ docker pull ubuntu:xenial
$ docker run --name pip-test --rm -it ubuntu:xenial bash

// in container
# apt update
# apt install -y python-dev python-pip
# pip install --upgrade pip
Collecting pip
  Downloading pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |################################| 1.3MB 865kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
# pip install requests
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

Same here .. Exact same output as @HayaoSuzuki and we don't use pyenv

If you're getting this outside of pyenv I suspect it's more likely to be related to #5221

Just to mention that pip install --user --upgrade pip on Ubuntu 16.04 also breaks.

Interestingly easy installing pip installs 10.0.0 but does not exhibit the issue
```
// in host
$ docker pull ubuntu:xenial
$ docker run --name pip-test --rm -it ubuntu:xenial bash

// in container

apt update

apt install -y python-setuptools

easy_install pip

Installed /usr/local/lib/python2.7/dist-packages/pip-10.0.0-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

pip install requests

Collecting requests
Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
100% |################################| 92kB 2.9MB/s
Collecting certifi>=2017.4.17 (from requests)
Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
100% |################################| 153kB 4.4MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |################################| 143kB 4.5MB/s
Collecting idna<2.7,>=2.5 (from requests)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |################################| 61kB 7.4MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
100% |################################| 133kB 4.4MB/s
Installing collected packages: certifi, chardet, idna, urllib3, requests
Successfully installed certifi-2018.1.18 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
```

which is not surprising, since that pip is egg-installed in a different place thus no longer affecting the global pip - it also means you now have a very interesting workingset for python

While I appreciate that the underlying issue reported here and in https://github.com/pypa/pip/issues/5221 is the environment, the cause is primarily that the import from pip import main was broken as the package pip.main was moved to pip._internal.main. It would be trivial to add a link from pip.main to pip._internal.main to fix this (whereas fixing the environment is a lot of work across many locations for many people). Is there a good reason to not do this?

@davidjlloyd

This comment gives some informations about not to do this, but I am not entirely sure if this is also important for the pip script itself. However, it solved the problem for me.

@davidjlloyd Because from pip import main was never supported, basically. And while it's easy to say "yes, but it's a simple API and it just worked", it really didn't - we've had multiple bug reports from people expecting certain behaviours from it, that we simply don't cater for (running pip.main in multiple threads, expecting pip.main to not change the configuration of the logging system, ...)

Rather than keep explaining to people that they shouldn't do this, and continually dealing with the fact that people are assuming "if I can find a function to call, it's supported" we moved everything to the _internal namespace to make it abundantly clear that you're not supposed to call it.

The bulk of the complaints have come from people using pip.main - which is ironic, as it's so easy to call pip in the supported command line way via subprocess. So of all the possible breakages, this is the easiest to fix - and yet people still haven't fixed it, even after months of warnings. (Although to be fair to the Linux distributions, they don't support people upgrading their system packages using pip, so reports like #5221 of cases where distro-supplied scripts break is fundamentally user error, not failure of the distros to address the pip 10 changes - something I'm sure they are handling perfectly well).

I can also confirm that this issue is absolutely destroying my previously very stable process of a building a docker image, here are example minimal things I'm doing inside my docker image build process:

+ pip install -U pip setuptools
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl (569kB)
Installing collected packages: pip, setuptools
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0 setuptools-39.0.1

...

+ pip install jupyter opencv-python plyfile pandas
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

Fix for us was pinning to pip 9.03, so:

pip install --upgrade pip==9.0.3

instead of

pip install -U pip

obvious fix but in case it helps somebody else!

@peteflorence So presumably when running docker, you're creating a base image, then running pip install -U pip setuptools as root in that image? Is there a reason you need the latest pip/setuptools if all you're doing is installing other packages with them? Could you not simply upgrade to the latest pip/setuptools available as a distro package?

I appreciate that this is a problem for you - it seems to be common for docker builds to be more inclined to do stuff as root than if you were in a normal OS (probably because a docker image is isolated). But it's still not a good idea to do this. The problem is that pip doesn't manage /usr/bin/pip, so we have no way of "fixing" that to work with pip 10.

What you could do is switch from using /usr/bin/pip to using python -m pip. It's still not supported, and may hit other issues (I don't know what changes your base OS vendor may have made to the system pip) but it will avoid the issue in /usr/bin/pip while you sort out a longer-term solution to your issue.

Pinning to pip 9 is also a solution, but this begs the question, why upgrade your OS pip at all if pip 9 is OK? Does your vendor not offer a packaged version of pip 9?

Rather than keep explaining to people that they shouldn't do this, and continually dealing with the fact that people are assuming "if I can find a function to call, it's supported" we moved everything to the _internal namespace to make it abundantly clear that you're not supposed to call it.

I'm not sure that aggressively breaking existing uses of an unsupported feature instead of deprecating the feature for a couple of major releases is the best approach. Our initial reaction was to pin pip back to 9.0.3, and more lazy developers would probably just call it a day at that point. This would leave a lot of users stubbornly clinging to old releases, which I doubt anyone wants. However your motivation makes sense, and the eventual result is the same.

For any users hitting this issue, I think the nicest solution for replacing system or pyenv installations was kindly provided by @standag here: https://github.com/pypa/pip/issues/5221#issuecomment-381568428

This solution should work for anyone building in Docker such as @peteflorence without pinning to stale releases or anything horrible like that.

Temporary Fix upgrade pip using.

curl https://bootstrap.pypa.io/get-pip.py | python3

Instead of pip install -U pip

For pip2 pip2 install --upgrade pip

I upgraded with pyenv, both python2 and python3, now pip2 not work and pip3 works
After comparing pip2 and pip3, the difference is the import line

pip2
from pip import main

pip3
from pip._internal import main

After substitute pip2 import line with pip3 version, it works

Having the same issue, heh.

Same issue, but solved with solution: https://github.com/pypa/pip/issues/5240#issuecomment-381673100

Same issue:

+ pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl
 (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
+ pip install awscli requests simplejson
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

Now it's weird, it installs pip 10 to /usr/local/bin, and that is first in the PATH before /usr/bin, but it doesn't use it, not until you go to a new shell. I saw this happen when I went to this box to try to install a newer awscli by hand...

$ python --version
Python 2.7.12
$ pip --version
pip 10.0.0 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
$ sudo pip install --upgrade awscli 
 <blah blah>
$ aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1049-aws botocore/1.4.70
$ /usr/local/bin/aws --version
aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-1049-aws botocore/1.10.4
$ which aws
/usr/local/bin/aws
$ echo $PATH
/home/ec2-user/bin:/home/ec2-user/.local/bin:/opt/bamboo-elastic-agent/bin:/opt/jdk-8/bin:/opt/maven-2.1/bin:/opt/maven-1.0.2/bin:/opt/ant-1.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/bin:/bin:/opt/puppetlabs/bin

@pfmoore thanks for comments. Yes our docker image apparently needed pip 9 and we inherit from an nvidia cuda 8.0 docker image and bunch of other stuff which must not have had it. Pinning is a good solution for us -- this is a research system, we want the code reasonably locked down. But yes obvsiously for many others you'll want a solution that brings you up to compatibility with pip 10

Now it's weird, it installs pip 10 to /usr/local/bin

I'm not a Unix user, but I recall seeing some people talk about "rehashing" in the shell? Could that be the issue? Does bash cache path lookups and need prompting to redo the path search?

Yes our docker image apparently needed pip 9 ...

Cool - certainly pinning to pip 9 is a reasonable solution in your case. I'm a little twitchy about letting suggestions to "just pin to pip 9" go unchallenged, because there's a risk people will see them and blindly copy them, just deferring the moment when they need to fix their environment. But certainly thinking about your requirements and deciding that pinning works for you is fine. So apologies for using your comment as a soapbox :-)

We pinned to pip 9 and that "fixed" us, but of course we're interested in being able to use pip 10 at some point.

try to use pip2 install xx @HayaoSuzuki

how to go back to pip 9.0.0..
It still shows same error to me. Please write all the steps

@swtt123 you can try pip install pip==9.0.1

pip 10.0, I also face

AttributeError: 'module' object has no attribute 'main'

while trying to use pip.main(['install', '-r', 'requirements.txt'])

@p00j4 Importing pip from within your program is not supported - see the docs

oh! I get it however, now forced to change a lot of places.
Thanks @pfmoore for the head.

Watch out for bash caching the executable location:

$ which pip
/usr/bin/pip

$ pip install --user pip
Collecting pip
(...)
Successfully installed pip-10.0.0

$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

$ which pip
/usr/bin/pip

$ hash -d pip  # this clears the 'pip' entry from bash's executables locations hash table

$ which pip
/home/zwinny/.local/bin/pip

$ pip --version
pip 10.0.0 from /home/zwinny/.local/lib/python2.7/site-packages/pip (python 2.7)

I've created a new Raspbian install on a Raspberry Pi 3B+. Nothing special - pretty vanilla configuration - and so far, everything has gone fine.

I've just reached the very last step in my install process:

pip install --upgrade pip

Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 101kB/s 
Installing collected packages: pip
Successfully installed pip-10.0.0

...and now pip is completely borked:

$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

As per the comment above, I've confirmed that ~i/.local/bin/pip runs fine. The error occurs with /usr/bin/pip. Running hash -d pip did not update the cached location, though.

Force-downgrading to 9.0.1 (via options line -Iv and --force-reinstall) also did not resolve the issue. Apparently, the only solution, besides not upgrading pip, is to run pip from ~/.local/bin/pip.

python -m pip install --upgrade pip 

Worked for me. I hope that helps someone.

I installed a fresh Ubuntu distro today and tried to get some basic Python packages up and running. I was prompted to upgrade pip, and so I ran the command it told me to. This upgraded pip to version 10, which is apparently broken with the same error at the top of this thread. I haven't done anything other than what a normal user would do to upgrade pip and install their favourite packages. Not even using pyenv.

None of the solutions here fix my problem. If I run python -m pip install --upgrade pip I get "Requirement already up-to-date". If I try to downgrade to version 9 I get the same initial error:

Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

After re-reading this thread, @sfsdfd's solution to use ~/.local/bin/pip is what finally worked:

~/.local/bin/pip install my-favourite-package

Apparently this is working by (successfully) reverting back to my older version of pip. Adding this to ~/.bashrc is a nicer fix for me:

export PATH=$PATH:~/.local/bin

Simply restarting my terminal fixed it for me.

I've fixed the issue by update the code in /usr/bin/pip, change from pip import main to from pip._internal import main.

Here details:

$ dpkg -S /usr/bin/pip
python-pip: /usr/bin/pip
$ dpkg -S /usr/bin/pip2
python-pip: /usr/bin/pip2
$ apt-file search /usr/bin/pip
colorized-logs: /usr/bin/pipetty
pipebench: /usr/bin/pipebench
pipemeter: /usr/bin/pipemeter
pipexec: /usr/bin/pipexec
python-pip: /usr/bin/pip
python-pip: /usr/bin/pip2
python3-pip: /usr/bin/pip3
rt-tests: /usr/bin/pip_stress

After pip install --upgrade pip:

$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
$ pip2
Traceback (most recent call last):
  File "/usr/bin/pip2", line 9, in <module>
    from pip import main
ImportError: cannot import name main

$ cat /usr/bin/pip
#!/usr/bin/python
# GENERATED BY DEBIAN

import sys

# Run the main entry point, similarly to how setuptools does it, but because
# we didn't install the actual entry point from setup.py, don't use the
# pkg_resources API.
from pip import main
if __name__ == '__main__':
    sys.exit(main())

After updated the /usr/bin/pip:

$ cat /usr/bin/pip
#!/usr/bin/python
# GENERATED BY DEBIAN

import sys

# Run the main entry point, similarly to how setuptools does it, but because
# we didn't install the actual entry point from setup.py, don't use the
# pkg_resources API.
# from pip import main
from pip._internal import main
if __name__ == '__main__':
    sys.exit(main())

$ pip --version
pip 10.0.0 from /home/devops/.local/lib/python2.7/site-packages/pip (python 2.7)

$ pip2
Traceback (most recent call last):
  File "/usr/bin/pip2", line 9, in <module>
    from pip import main
ImportError: cannot import name main

My system information:

$ uname -a
Linux devops-kubernetes-master 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.10
DISTRIB_CODENAME=artful
DISTRIB_DESCRIPTION="Ubuntu 17.10"
NAME="Ubuntu"
VERSION="17.10 (Artful Aardvark)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.10"
VERSION_ID="17.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=artful
UBUNTU_CODENAME=artful

$ python --version
Python 2.7.14

$ apt list --installed | grep python-

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libpython-all-dev/artful,now 2.7.14-2ubuntu1 amd64 [installed]
libpython-dev/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
libpython-stdlib/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
python-all/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
python-all-dev/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
python-apt-common/artful,artful,now 1.4.0~beta3build2 all [installed]
python-asn1crypto/artful,artful,now 0.22.0-1 all [installed,automatic]
python-cffi-backend/artful,now 1.9.1-2build2 amd64 [installed,automatic]
python-crypto/artful-updates,artful-security,now 2.6.1-7ubuntu0.1 amd64 [installed,automatic]
python-cryptography/artful,now 1.9-1 amd64 [installed,automatic]
python-dbus/artful,now 1.2.4-1build3 amd64 [installed,automatic]
python-dev/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
python-enum34/artful,artful,now 1.1.6-1 all [installed,automatic]
python-gi/artful,now 3.24.1-2build1 amd64 [installed,automatic]
python-idna/artful,artful,now 2.5-1 all [installed,automatic]
python-ipaddress/artful,artful,now 1.0.17-1 all [installed,automatic]
python-keyring/artful,artful,now 10.4.0-1 all [installed,automatic]
python-keyrings.alt/artful,artful,now 2.2-2 all [installed,automatic]
python-minimal/artful,now 2.7.14-2ubuntu1 amd64 [installed,automatic]
python-pip/artful,artful,now 9.0.1-2 all [installed]
python-pip-whl/artful,artful,now 9.0.1-2 all [installed,automatic]
python-pkg-resources/artful,artful,now 36.2.7-2 all [installed,automatic]
python-secretstorage/artful,artful,now 2.3.1-2 all [installed,automatic]
python-setuptools/artful,artful,now 36.2.7-2 all [installed,automatic]
python-setuptools-doc/artful,artful,now 36.2.7-2 all [installed]
python-six/artful,artful,now 1.10.0-4 all [installed,automatic]
python-talloc/artful,now 2.1.9-2ubuntu1 amd64 [installed]
python-wheel/artful,artful,now 0.29.0-2 all [installed,automatic]
python-xdg/artful,artful,now 0.25-4 all [installed,automatic]

I fixed this way:

$ pip --version
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in
from pip import main
ImportError: cannot import name main
$ sudo apt-get remove python-pip
$ sudo apt-get install python-pip
$ pip --version
pip 10.0.0 from /home/user/.local/lib/python2.7/site-packages/pip (python 2.7)

I hope that helps someone.

Even after following everything I am not able to downgrade pip. Keep getting error.
Here's how i fixed this:

$ sudo apt-get remove python-pip
$ pip -v
bash: /usr/bin/pip: No such file or directory

but when i try this
$ python -m pip -V
pip 10.0.0 from /home/user/.local/lib/python2.7/site-packages/pip (python 2.7)

so i removed complete folder of pip
$ sudo rm -r /home/user/.local/lib/python2.7/site-packages/pip*

then again i try this
$ python -m pip -V
/usr/bin/pip: No module named pip

then install pip again
$ sudo apt install python-pip
$ python -m pip -V
pip 8.1.1 from usr/lib/python2.7/dist-packages (python 2.7)

Hope this solve's the problem

AFTER WASTING 3 HOURS THIS IS WHAT I GOT

TO UNINSTALL pip 10.0.0:
sudo apt-get remove python-pip

but then u wont be able to install the correct required version of pip by the direct methods.

TO DOWNGRADE TO A KNOWN WORKING VERSION AFTER INSTALLING
when pip10 throws the error: ImportError: cannot import name main
you will have to run:
python -m pip install pip==KNOW_WORKING_VERSION>

TO INSTALL ANY PACKAGE USING PIP10:
sudo python -m pip install PACKAGE_NAME
this worked.

Cheers :)

Same problem, fix issue: full reinstall: python-pip

just relogin shell as mentioned here
https://github.com/pypa/pip/issues/5240#issuecomment-382262586

pip3 install --upgrade pip==9.0.3
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

heh... what did I except...

I just unrolled my pip installation.
python -m pip install --user --upgrade pip==9.0.3

I wanted to report that this issue also happens on Windows build when using AppVeyor for CI:
The result is the following:

Running Install scripts
SET PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
pip install --disable-pip-version-check --user --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Successfully installed pip-10.0.1
pip install -U setuptools
Traceback (most recent call last):
  File "c:\python27-x64\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27-x64\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27-x64\Scripts\pip.exe\__main__.py", line 5, in <module>
ImportError: cannot import name main
Command exited with code 1

I have: pip install --disable-pip-version-check --user --upgrade pip in several appveyor.yml scripts, based on the recommendation from this sample: https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor.yml#L111

This has worked well until pip 10.x, and I suspect if others have created appveyor.yml based on this will also run into this issue.

Switching to easy_install -U pip worked, but I do have several repos that worked before that now need to be updated to work with pip 10.x.

Seems like the recommended approach for CI setup should be sticking with 9.0.3 or using easy_install.

Seems like the recommended approach for CI setup should be sticking with 9.0.3 or using easy_install.

The recommended approach is to remove the dependence of your library/application/tool from pip's internal implementation details. easy_install is no longer being actively improved AFAIK and likely doesn't support a lot of the things that pip does.

As for your specific issue, you can probably fix it by running python -m pip install --force-reinstall pip, to fix the broken script. If not, please file a new issue.

The recommended approach is to remove the dependence of your library/application/tool from pip's internal implementation details. easy_install is no longer being actively improved AFAIK and likely doesn't support a lot of the things that pip does.

I'm guessing you didn't read the rest of his post. He was referring to using easy_install to install pip, not instead of pip:

Switching to easy_install -U pip worked...

Also he is not relying on pip's internal implementation details. He's using a fresh spinup of AppVeyor and upgrading pip using the command he specified, and nothing more:

pip install --disable-pip-version-check --user --upgrade pip

Many other people in this thread are having the same problem. You can reproduce this with a basic installation of pip without doing anything special or dependent on pip's internals. This has been documented by many people in the posts above.

@ikreymer The issue you seem to be referring to is on Appveyor, which runs Windows. The issue being discussed here is in relation to pyenv which is an exclusively Unix utility.

I don't know about @pradyunsg, but I'm getting increasingly confused as to what the various problems being discussed are. The root cause is easy - pip (deliberately) moved the internal APIs, and this is causing problems for people who were relying on tools that were using them, either directly or indirectly. But that's not something that's going to be changed. So if we're to assist users with finding a resolution to their specific issues, we need to keep the discussion focused on one problem at a time.

So can I ask - if you need help with a problem on Appveyor, please could you open a new issue that describes the problem, and we'll relocate the discussion over there. (Most of the detail is likely in your comment, so that will do as a starting point for the issue, but a complete description of how to reproduce the problem in a brand new Appveyor project would be immensely useful in helping us to diagnose the issue.

@arvoelke:

He was referring to using easy_install to install pip, not instead of pip

Installing pip using easy_install could well have issues, and if it does, we won't be able to support you, because (as @pradyunsg said) easy_install is old, not actively maintained, and missing recent features. But if it works for you, and you don't need us to help, then fine - no-one is stopping you.

Many other people in this thread are having the same problem

Define "the same". If you mean "seeing errors from code that imports pip.main then yes, but that's not a problem with pip, it's a deliberate, backward incompatible, change to the internal implementation of pip. If you want us to just say "tough, you shouldn't be using the internal APIs of pip", then fine - a single issue is enough, and we'll just close it as "not a bug". But if you want us to help you to work out what part of your environment has failed to follow the advice published 6 months ago, and find a workaround for you that lets you carry on using pip while your environment gets fixed by the vendor, then you'll have to help us to help you. And saying "I have the same problem" on a report about a purely Unix utility, when you're running on Windows, is definitely not helping us to help you...

The issue being discussed here is in relation to pyenv which is an exclusively Unix utility.

One of the first replies to this thread was:

Same here .. Exact same output as @HayaoSuzuki and we don't use pyenv

The pyenv aspect seems more or less secondary based on the above comments and linked issues and commits, including those occurring on fresh installations of Ubuntu and RaspberryPi. Again all of this is already a part of this issue. The root cause is "the same" (i.e., upgrading pip and then trying to launch it) for everyone, including the OP, and so we can create more issues but there isn't really anything different going on between them -- only in how the issue manifests.

And saying "I have the same problem" on a report about a purely Unix utility, when you're running on Windows, is definitely not helping us to help you...

I'm not running on Windows. I'm running on Ubuntu as I said before. The other poster is using Windows, but they are simply upgrading pip and running pip, just like the rest of us in this issue. There is a pretty common consensus that use of pyenv is not fundamental to the issue.

All those issues seem to stem from the same problem: upgrading pip, but keeping on using an old launcher that still use the old entrypoint. A good way to avoid this kind of issue is to use python -m pip.

All those issues seem to stem from the same problem: upgrading pip, but keeping on using an old launcher that still use the old entrypoint. A good way to avoid this kind of issue is to use python -m pip.

Yep, I think that seems to be the root of the issue, and happens across all different platforms mentioned in this thread, including Windows.

In case this helps anyone else, I can confirm that switching appveyor.yml pip upgrade from:

pip install --disable-pip-version-check --user --upgrade pip

to:

python -m pip install --upgrade pip

does fix the issue. Now to update several more repos!

I encountered the same symptom discussed here in a different situation.

I was trying to use a locally installed pip, on a Ubuntu 16.0.4 system:

curl -O https://bootstrap.pypa.io/get-pip.py
export PYTHONUSERBASE=$(pwd)
python ./get-pip.py --user
export PYTHONPATH=$(pwd)/lib/python2.7/site-packages

python ./bin/pip --version

Traceback (most recent call last):
  File "/path/to/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal

It turned out that Ubuntu prepends a pip-specific path to sys.path via site.py, which was overriding my PYTHONPATH, shown below:

import sys
print(sys.path)
['', '/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg', '/path/to/lib/python2.7/site-packages`, ...]

I tried avoiding the prepend with -S flag for python, documented in the man page as:

-S Disable the import of the module site and the site-dependent manipulations of sys.path that it entails.

and it worked:

python -S ./bin/pip --version

pip 10.0.1 from path/to/bin/pip (python 2.7)

Sharing in case this might help others - In my docker image (base is ubuntu:xenial) I received the following error:

Step 8/12 : RUN pip install -U pip  && pip install -r /tmp/requirements.txt
 ---> Running in e4ff51b013f0
Collecting pip
  Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.1
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

I changed pip install -U pip && pip install -r /tmp/requirements.txt to pip2 install -U pip && pip2 install -r /tmp/requirements.txt. This resolved the issue.

I'm not sure if I saw an answer / response to @davidjlloyd's comment:

I'm not sure that aggressively breaking existing uses of an unsupported feature instead of deprecating the feature for a couple of major releases is the best approach.

Can I please ask why there was not a deprecation process in place for this?
It seems like it would have been quite easy, on the import of pip, check sys.argv[0]; if it's not pip or pipX[.Y], spew some DeprecationWarnings that this will fail in release X+, with a link to what you mentioned: https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

Is there a process in place to try and ensure this kind of thing is hopefully avoided in the future?

I'm not sure if I saw an answer / response to @davidjlloyd's comment:

It's been answered repeatedly, Maybe not on this specific issue, but a search of the issue tracker should find plenty of discussion on the matter.

Can I please ask why there was not a deprecation process in place for this?

Because it was never supported. Why would we warn that we're desupporting something we never supported? People assumed it was OK to read pip's source code and use functions they found in there in their own code. It never was. We said it could break, and in pip 10 it did.

It seems like it would have been quite easy, on the import of pip, check sys.argv[0]; if it's not pip or pipX[.Y], spew some DeprecationWarnings

I'm not sure it's as easy as you think. And I say that from the perspective of someone who has had to deal with issues where warnings added in pip 10 were being triggered when we hadn't expected them to be...

And again, there's no need to deprecate something that's not supported in the first place.

Is there a process in place to try and ensure this kind of thing is hopefully avoided in the future?

What sort of thing? Breakage caused by us changing things that we don't guarantee backward compatibility for? No. There's no need for us to avoid that. Although in fact, we do try to manage the process, as a courtesy to our users (not an obligation!). In this case, we publicised the change 6 months in advance, offered suggestions for people who needed to change their code, and have been spending a lot of time since the release helping users who have had problems because software they rely on hasn't heeded those warnings. That's a lot of work that a very small volunteer group put into trying to mitigate a situation caused by people expecting support that was never offered or promised. You're welcome.

We do have processes (deprecation warnings, etc) in place for changing things that we do guarantee compatibility for - but importing pip into your own program isn't one of them.

I had a bash script that installed flask and requests the upgrade broke my script, but I do understand the reasons stated above. My work around to keep my script working was to simply launch a new bash process, maybe that is wrong but it worked for me. Hopefully it might help others with similar scripts.

pip install --upgrade pip
echo "pip install Flask" | bash
echo "pip install requests" | bash

@OneLogicalMyth what you are looking for may be hash -d pip, see https://github.com/pypa/pip/issues/5221#issuecomment-381568428.

completely missed that even after reading it carefully, thank you @austinbutler this has resolved my issue.

The only reason this issue was kept open and not closed as a duplicate immediately was because I was concerned if pyenv does something with shims and if pip would need to help out in some way. That wasn't the case so, this issue is closed now.

I've listed the workarounds before (all of them trivial) for almost all the these problems mentioned here - take a look at https://github.com/pypa/pip/issues/5221#issuecomment-382069604. Hopefully, that addresses all the concerns raised in this issue. If it doesn't, please open a new issue.

@benoit-pierre I added your suggestion of using python -m pip to the linked comment. Thanks for that. :)

Installing pip using easy_install could well have issues, and if it does, we won't be able to support you, because (as @pradyunsg said) easy_install is old, not actively maintained, and missing recent features.

Thanks for clarifying my position @pfmoore. That's what I was talking about.

I'm getting increasingly confused as to what the various problems being discussed are.

Me too.

Just some general tips (take it or leave it), about things I saw in this issue:

• If the pip command doesn't work for some reason, try running python -m pip. Chances are python -m pip will work for you even if your wrapper scripts break. If it doesn't work, open an new issue.
• Don't run pip as root or do sudo pip. You'll likely break your system and if that's not bad enough, you're doing remote code execution as root.
• Don't use easy_install for the reasons are quoted at top of this comment. Here's more context (that's slightly out of date).
• Please please don't pin to pip 9. I strongly believe that doing that is actively acting to slow down the progress of the Python Packaging in general.
  
  
  
  • pip 9 is never going to support PEP 517/518 and staying on pip 9 means you'll have to take the pains to switch over later, when a package breaks last minute because they switch to the newer packaging standards.
  
  

To summarize, this is likely not a single project/person's fault and everyone is well reasoned for taking their actions. Yes, your environment broke. We understand. Do not throw blame at people who're volunteering their free time, to help you out right now and to develop pip.

Want to help us? https://donate.pypi.org is a thing and if not that's not your type of thing, there's lot of open issues in this very issue tracker that you can help us out with.

I'm gonna step back from this issue now.

Okay, one last thing, if someone still wants to discuss regarding our decision to move all of our codebase to pip._internal, open a new issue and gimme a mention. I'll spend some time to write up something with links and all, about why pip did this. That way, @pypa/pip-committers would have a single place to people link to on this topic.

For the time being, this worked on python3 / pip3, to roll it back

python3 -m pip install --user --upgrade pip==9.0.3

@freckletonj Please do not tell people to revert to pip 9. That is not how you should resolve this issue. There are much better ways.

I have linked to a comment just above yours, listing out ways to resolve this issue.

sorry @pradyunsg , but that still doesn't work:

$ pip3 install --upgrade --user pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.1
$ pip3
Traceback (most recent call last):
  File "/usr/local/bin/pip3", line 7, in <module>
    from pip import main
ImportError: cannot import name 'main'

From https://github.com/pypa/pip/issues/5221#issuecomment-382069604, which I've linked to above:

hash -r pip # or hash -d pip

If anyone has any other queries, please open a new issue.

5599 provides information and provides a single location to seek help toward resolving this issue for end users.

The comments section of that issue are open for users to discuss specific problems and solutions. :)