Se #4805 and #3389 for the history. Basically, pip cannot properly uninstall packages installed by "pure" distutils, because distutils doesn't record enough metadata for us to do so. We've previously removed the installation metadata, so that it looks like we did the install, but we've had to leave files behind. That causes issues. Since pip 8, we've been trying to stop doing this, because it's a cause of problems, but our initial attempt was reverted because it affected too many people at the time. With pip 10, we've finally stopped trying to uninstall distutils packages, and now we report the problem to the user as you see here.

Basically, if you (or some infrastructure you use) installed a package using distutils, you need to manage (and in particular) uninstall it "using distutils". Unfortunately distutils doesn't include an uninstall command, so "uninstall using distutils" means manually removing the package.

@pfmoore Thanks for the quick answer!

This is rather inconvenient because most of the packages come installed as dependencies and we do not manage it ourselves. Automating the removal will be interesting.

I see there is some movement to only upgrade packages, without uninstalling them first. It would be great if this would eventually happen.

We could close this issue since it is widely discussed somewhere else. Thanks!

Try removing package manually from "site-packages". This works perfect!

I seem to get round this problem by providing the version of pip with the command
pip install -I==9.0.3 -r requirements.txt

I used follow steps and resolved this problem

1. Reduced version:
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package:
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip:
   pip install --upgrade pip

This seems to fix the issue for me: https://github.com/blockstack/blockstack-core/issues/504

In the orignal case above, that'd be:

pip install docker-py --ignore-installed PyYAML

Don't know who you are, but I love you.

So @pfmoore what would you recommend for a team that automates their pip installations? We set up our server using terraform, so we automatically call pip install for smart-open and boto3 and then we have numpy, scipy, boto, sklearn, and datadog in a requirements.txt. We get the distutils error for urllib3 for multiple installs (because smart-open installed it). The installs work when I use --ignore-installed urllib3, but is there a more proper way to go about this?

Also, is there any chance that distutils will add an uninstall feature or more metadata? Have you guys talked to that team about this?

@ruby-is-pretty-cool I have no real opinion on that. You say of urllib3 "because smart-open installed it" but I don't see anything in smart-open that declares a dependency on urllib3, so I don't know what you mean by that. If smart-open is somehow installing urllib3 in a way that triggers this issue, that sounds like a smart-open problem.

Also, is there any chance that distutils will add an uninstall feature or more metadata? Have you guys talked to that team about this?

Not that I know of, and no we haven't talked to them about it. I doubt they would, as distutils doesn't really implement new features these days (but feel free to ask if you want to). The official packaging documentation doesn't recommend using distutils directly anyway, so if projects are doing so, it's really up to them to manage the implications.

I just tried installing smart-open. It depends on requests, which in turn depends on urllib3. That is installed properly by pip when I do the install in a virtualenv. Are you doing this installation in a system environment? In which case, are you seeing (and trying to upgrade) the system installed urllib3? If so, the normal advice "don't use pip on system packages" applies - use a virtualenv or if you need to install to your system environment use distro packages.

I am pretty new to Python packaging, but I have a feeling that either a virtualenv or distro packages will fix our issue - I'll have to look into them more. Thank you for your help!

This is insane. Installations are being broken because of some religion-like believing. if some setup.py entry of some package touches urllib3, then there is no way to tells it to ignore it. In our case, we must upgrade urllib3 because in Ubuntu 14 the installed version of pip (v1.5.x) refuses to install packages from https URLs.

The point is that nobody would upgrade "system" packages if they would not stop working. In linux kernel project this behavior is "breaking userspace" and Torvalds does not react nicely to it. It is incredible python people takes it so naturally fine.

At least you should warn people in pip messages with something like "this version of pip will refuse to work if it means to upgrade system packages, do this and do that. Use pip version 9.x if you are working with old installers or old systems." or similar.

Ok, I wasn't paying attention as mine's still hosed, but, my point is more simple.
The entire reason for just aggressively upgrading userland was: ATT&T and berkley were still sliming on who built the trashcan, persay... And If you got the shit to install on aha1542, You could ditch a joilet's that was dosed due to licensing; and as that has come up recently as well, they reason why we do this shit in the first place; for CHRIST sake, is to have the choices AND co-operate. I am still somehow baffled why slackeware and the *bsd path went into the deepend, but anyway, even in two camps, 1 will drive the other, I'd be bored of kernel, go back to 2.x, 16+ screens, (I have yet to refind the promix driver in the menu, but I digress, & now back into my regularly schedule, WTF can't we get a .app so this shit works day. (of screwing off of course.)

I agree with @pabloa - This has been going on forever now, a vanilla install of CentOS 7 cannot install docker-compose because of this.

And yes, @JohnBDonner, that little flag just saved me a LOT of time. THANK YOU

Initial Error

"Cannot uninstall 'pywin32'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall"

How Do I Get there?

I've pywin32==221 installed into my machine, and now i'm attempting to uninstall an upgrade version of it (pywin==224), i've gain link to its ".whl" and i facing challenges to install them as i received following error "Cannot uninstall 'pywin32'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall
.". I'm on Window 10 64bit, using py3.4 32bit.

Reason i'm doing this?

is because it seems like "win32.client" in my machine only has the "optimize" python file ".pyo file extension". I found out this when I receive compilation error saying i dont have "Dispatch" module available for "from win32com.client import Dispatch". Appreciate if anyone could assist me on this.

YOU MAD BRO??

Nope, but thanks for the random abuse. It makes doing this in my spare time so rewarding :smile:

It wasn't intended to solve the problem. It was intended to explain why things are the way they are, and why pip can't solve the problem. Just because people are sad about the situation doesn't alter it.

Is it possible to change the installer package used by disyutils such that
it includes the required meta data?
On Thu, Nov 22, 2018 at 12:56 PM Paul Moore notifications@github.com
wrote:

It wasn't intended to solve the problem. It was intended to explain why
things are the way they are, and why pip can't solve the problem. Just
because people are sad about the situation doesn't alter it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/5247#issuecomment-441095536, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADrjCZvIpDr4X1sxcSZe3rRH7MBnn7Tsks5uxuVagaJpZM4TWMU4
.

>

Nick Artman
+1 (330) 558-1230

I don't know, I don't work on distutils. But even if it were, why would people use distutils these days anyway? This thread is about packages that were already installed using distutils. The solution for those was already mentioned earlier in this thread.

@AddoSolutions the very change you ask for is called setuptools

So the reason I am on this thread is because I often work with vanilla
CentOS 7 boxes and specifically install docker compose on them. CentOS
comes stock with pip and the problem is I get these errors as described
above.

I am personally not much of a Python user, I just use docker compose and
sshuttle on it. I personally have no idea the difference between setuptools
and distutils, I’m guessing that the CentOS python is basically installed
using yum upon install?

Two things with that: can the yum package be updated such that it includes
the correct metadata? Alternatively, could upon getting this error, could
the message give more helpful info on what to do? Like instead of the
relatively cryptic message about distutils (again for people like me using
pip I don’t even know what that is) could it say something like “the way
that pip was installed is not recommended. We recommend uninstallling by
doing x and then reinstalling using x”. I feel like that would
significantly reduce the number of people with issues with this.

Thoughts?

Note: I fully realize I could dig into the difference, and probably will,
but asking this for future users and it’s also thanksgiving and I want
turkey :)
On Thu, Nov 22, 2018 at 1:19 PM Ronny Pfannschmidt notifications@github.com
wrote:

@AddoSolutions https://github.com/AddoSolutions the very change you ask
for is called setuptools

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/5247#issuecomment-441099032, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADrjCTBfoKjBuYdceDeBM0bunByQ2bxJks5uxuq2gaJpZM4TWMU4
.

>

Nick Artman
+1 (330) 558-1230

wrt the updates of the yum package - that's pretty much the job of the distribution - and it hasn't been doing it, but neither have the packages themselves in some cases

when it comes to enterprise distros - things are so painfully outdated - sort it out with your vendor - open source upstreams have no power over that and should never have to bear the support burden for your choice in enterprise distros - so go to your vendor

Gotcha. So the pip package in this case would be maintained by the CentOS
group, NOT by the pip group?
On Thu, Nov 22, 2018 at 5:38 PM Ronny Pfannschmidt notifications@github.com
wrote:

wrt the updates of the yum package - that's pretty much the job of the
distribution - and it hasn't been doing it, but neither have the packages
themselves in some cases

when it comes to enterprise distros - things are so painfully outdated -
sort it out with your vendor - open source upstreams have no power over
that and should never have to bear the support burden for your choice in
enterprise distros - so go to your vendor

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/5247#issuecomment-441129627, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADrjCZAQ-ZBmpkm7izyKkE0WymSoT6M1ks5uxyd2gaJpZM4TWMU4
.

>

Nick Artman
+1 (330) 558-1230

@AddoSolutions the problem there is not pip, but other packages - pip simply stopped pretenses and dropped support for really bad things, and now stuff falls apart on bad actors like enterprise distros - so you have to sort out the package in terms of the enterprise distros and with your vendor

just a FYI - you will likely be told to use pip from the system (managed by the vendor) and not a random latest version

I was getting this error for wxPython

Cannot uninstall 'wxPython'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

It was not located in sitepackages but rather in distpackages (which makes sense), but it didn't feel right juist deleting the file from the folder.

Turns out I had installed it via apt, so doing apt remove --autoremove python-wxgtk3.0 removed the package from my system.

This "not our problem, someone else fix it" is to be expected from pip. The software doesn't even pretend to manage package conflicts. I got this message as part of an ansible deployment of containers to many systems.

Thank you for those of you who supplied workarounds. I've changed the ordering - "pip install docker" prior to "pip install --upgrade pip" - worked this time. Hopefully this won't cause problems (such as data corruption) in future.

My reading of this is that we should be using virtual environments to protect ourselves from oddness in our environments... however, I was able to get past this for now by adding the --user flag:

pip3 install -r requirements.txt --user

https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-user

You could also try working with the exact version of the transient dependency that was installed by your environment / distutils as it could be compatible. In my example, Pipenv will resolve the transient PyYAML dependency of awscli==1.15.85 and apache-airflow==1.10.1 to 3.13, but the system already has 3.11 installed. Looking at the declared dependencies on my local development machine, 3.11 would be fine for both:

$ pipenv graph | egrep 'airflow|awscli|PyYAML'
apache-airflow==1.10.1
    - PyYAML [required: >=3.0, installed: 3.13]
awscli==1.15.85
  - PyYAML [required: >=3.10,<=3.13, installed: 3.13]

A simple pipenv install PyYAML==3.11 will pin PyYAML at 3.11 and make both packages happy:

$ pipenv install PyYAML==3.11
Installing PyYAML==3.11…
...
Locking [packages] dependencies…
✔ Success!

$ pipenv graph | egrep 'airflow|awscli|PyYAML'
apache-airflow==1.10.1
    - PyYAML [required: >=3.0, installed: 3.11]
awscli==1.15.85
  - PyYAML [required: >=3.10,<=3.13, installed: 3.11]

My Pipfile / Pipfile.lock subsequently installs cleanly on Ubuntu 14.04LTS using pipenv install --deploy --system.

I also checked and PyYAML==3.11 is installed through python3-yaml 3.11-3build1 and it is a direct dependency of cloud-init 18.4-0ubuntu1~16.04.2, which we extensively use as we run on EC2. Both 18.04 LTS and 18.10 come with PyYAML 3.12, only 19.04 will come with PyYAML, which happens to be the latest version as of today.

That being said, avoid system dependencies and environmental issues at all costs. Use pipenv and/or virtualenv.

I used follow steps and resolved this problem:

1. Reduced version,
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip
   pip install --upgrade pip

This worked well for me. But does upgrading pip after this give any problems to the installed packages??

@s-eswar So far, we are not found problems about the package, but maybe one situation that if use the high version of the pip install package, then use the lower version re-install or check may be occur dependency problem. I suggest that always use lower version. For instance, pip 9.0.3.

@s-eswar So far, we are not found problem about for packages, but maybe one situation that if use the high version of pip install package, and use the lower version re-install or check may be occur dependency problem. I suggest that always use lower version. For instance, pip 9.0.3.

@wangxf1987 but for using ML libraries with the same config, don't we get errors regarding upgradation/use of lower version.

@s-eswar I am not sure ML libraries whether prefect works under lower version. I using pip=9.0.3 works will with the latest version of the Kubernetes. It's should be check the requirements file whether need to the latest version of the pip or testing in the dev env.

Since pip 8, we've been trying to stop doing this, because it's a cause of problems, but our initial attempt was reverted because it affected too many people at the time. With pip 10, we've finally stopped trying to uninstall distutils packages, and now we report the problem to the user as you see here.

I believe the part "now we report the problem to the user" is called "pushing your technological debt onto someone else". I'd like to thank everyone for bring me into the fiasco. Especially since I am not a Python dev and I have no idea how to fix it. I get to wates hours on this problem now. Hooray!

Remove Dist Package and RUN

sudo rm -rf /usr/lib/python3/dist-packages/yaml

sudo rm -rf /usr/lib/python3/dist-packages/PyYAML-*

Removing folder from distutils works

by upgrading to newer version it should solve the problems but pip introduces problems what a irony.

The solution is to uninstall it manually, so go to.../anaconda3/lib/python3.*/site-packages/ and delete all files and folders of the package

@ramonamis sudo pip install --ignore-installed PyYAML

This successfully upgraded it for me.

This also worked for imutils:

pip install --upgrade imutils --ignore-installed imutils

I used follow steps and resolved this problem

1. Reduced version:
   pip install --upgrade --force-reinstall pip==9.0.3
2. Tried to re-install package:
   pip install xxx --disable-pip-version-check
3. At last, recover the latest version for pip:
   pip install --upgrade pip

it works for me， Thanks!

I used follow steps and resolved this problem

1. Reduced version:
   `pip install --upgrade --force-reinstall pip==9.0.3`

2. Tried to re-install package:
   `pip install xxx --disable-pip-version-check`

3. At last, recover the latest version for pip:
   `pip install --upgrade pip`

this is a bad solution. I cannot do anything now.

Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3-pip : Depends: python-pip-whl (= 9.0.1-2.3~ubuntu1) but 9.0.1-2.3~ubuntu1.18.04.1 is to be installed
               Recommends: python3-dev (>= 3.2) but it is not going to be installed
               Recommends: python3-setuptools but it is not going to be installed
               Recommends: python3-wheel but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
E: Could not read response to hello message from hook [ ! -f /usr/bin/snap ] || /usr/bin/snap advise-snap --from-apt 2>/dev/null || true: Success

I'm locking this thread now. @pfmoore's initial comment covers everything here. The rest of this thread has divulged into support requests or straight up abuse directed at the maintainers.