Pip: assertion when using --no-cache-dir in 19.0

Same thing happening with:
Python v3.6.8
pip version 18.1
on
Ubuntu:latest image.

@snstanton what base image are you using? I'm seeing a similar issue on pip v18.1 as well

I've got the exact same issue.
on my side it seems it doesn't matter which package/distribution I try to install

I'm seeing this even without --no-cache-dir set. It happens for all packages I try to install, even if they're already installed.

• pip version: 19.0
• Python version: 3.6.0
• OS: Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

I should note that in my case I'm seeing it when running pip with a combination of sudo -H and bash -l -c.

$ sudo -H bash -l -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Looking in indexes: https://pypi.org/simple, http://pypi.lan.cogtree.com/cogtree/simple/
Collecting hypothesis
  Downloading http://pypi.lan.cogtree.com/cogtree/simple/hypothesis/hypothesis-4.1.0-py3-none-any.whl (238kB)
    100% |████████████████████████████████| 245kB 120.5MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Exception:
Traceback (most recent call last):
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Running the same commands without -l on my bash -c, or without bash -l -c involved at all, it all works fine.

$ sudo -H bash -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Collecting hypothesis
  Downloading https://files.pythonhosted.org/packages/89/7b/d6206dcde963139daa03a1d85b0c3428cb3ebf2ae8de3244b14a63e22680/hypothesis-4.1.0.tar.gz (180kB)
    100% |████████████████████████████████| 184kB 33.7MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Building wheels for collected packages: hypothesis
  Building wheel for hypothesis (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/10/12/eb/4ab734432e8466d545c8501f531458845b45e8c4427d5367f9
Successfully built hypothesis
Installing collected packages: hypothesis
Successfully installed hypothesis-4.1.0

Fascinatingly, if I run the same command without sudo or bash involved at all, it still fails with the same error, so it seems like it's some weird permissions issue maybe.

Another workaround for some situations

For people who hit this bug because virtualenv is automatically installing the latest version of pip, you can work around it by giving virtualenv the --no-download option, or setting VIRTUALENV_NO_DOWNLOAD=1.

But be aware that this may give you a very old version of pip, depending on the last time you upgraded virtualenv.

This also works with tox: VIRTUALENV_NO_DOWNLOAD=1 tox.

for what it's worth : it also fails with same error if package is already installed:

gregory.starck@canon:~/tmp$ ./venv/bin/pip install --no-cache-dir six ; echo $?
Looking in indexes: http://pypi:3141/root/ax/+simple/
Requirement already satisfied: six in ./venv/lib/python3.6/site-packages (1.12.0)
Exception:
Traceback (most recent call last):
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
2
gregory.starck@canon:~/tmp$

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

Issue is with failing assert, so setting env PYTHONOPTIMIZE=1 (or with parameter -O) makes this error go away.
Just tested it.
This workaround works because python optimizes code removing all asserts as one of the things.
Do not go for =2 (or -OO), as this removes docstrings and other tracebacks will appear - some code wants to operate on them.

It looks like someone knew this might end up being an issue (source):

        # TODO: This check fails if --no-cache-dir is set. And yet we
        #       might be able to build into the ephemeral cache, surely?
        building_is_possible = self._wheel_dir or (
            autobuilding and self.wheel_cache.cache_dir
        )
        assert building_is_possible

https://github.com/pypa/pip/pull/5884 looks like this is a related change that could have caused this?

Seems like the pip maintainers should rollback the recent 19 release to address this breaking change?
19.0 release notes: https://github.com/pypa/pip/blob/master/NEWS.rst#190-2019-01-22

UPDATE: not trying to cast aspersions here, was just suggesting as one way to quickly unblock people affected by this seeing as the release had _just_ happened. Rolling forward with a hotfix works too. I appreciate the hard work of the community that supports this mission critical tooling, and agree with the sentiments below about postmortems to learn from mistakes and prevent future issues. Meanwhile, we are doing the same internally, which means a liberal amount of hardpinning pip versions in all the places :)

The PR adding the TODO comment also has this comment in reply: https://github.com/pypa/pip/pull/5743/files#r215832743

Based on that comment and also the commenter above saying that passing PYTHONOPTIMIZE=1 makes the error go away, it seems like simply removing the assertion might be the correct fix (independent of the question of rolling back).

Yeah, when I delete that assert, packages do install fine with --no-cache-dir. In that case, it says it's Running setup.py install instead of Building wheel for sdist packages.

This is also happening to my projects. I can reproduce this in Docker images built FROM ubuntu:bionic and FROM centos:centos7 where I am installing Python 3 from source (here is a Gist that demonstrates a failing example for both of those Docker images in the event that it might be helpful). For the requirements.txt in the example in the Gist and

$ pip3 install --upgrade pip setuptools wheel
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages (19.0)
Requirement already up-to-date: setuptools in /usr/lib/python3.6/site-packages (40.6.3)
Requirement already up-to-date: wheel in /usr/lib/python3.6/site-packages (0.32.3)

then

$ pip3 install --upgrade --no-cache-dir -r requirements.txt

fails with

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/usr/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

but

$ pip3 install --upgrade -r requirements.txt

works fine as expected.

I'm particularly hitting this with tox + docker + ENV PIP_NO_CACHE_DIR=off

My workaround is to use tox-virtualenv-no-download plugin to prevent pip from auto-updating

We also have --no-cache-dir in our installs inside Docker to keep the images small. Our workaround has been --cache-dir=/pipcache and then rm -rf /pipcache in the same RUN step so it never ends up in the image.

Software development is hard and bugs like this are always going to happen. Certainly nobody should blame the pip maintainers or contributors for this incident.

However, I would suggest that this bug merits some sort of post-mortem analysis on the part of the pip team, due to the number of (missed) opportunities for this bug to have been caught before slipping into a general release. For example:

• automated testing of core functionality like --no-cache-dir
• pre-commit, pre-merge, or pre-release checks that flag (or prohibit) TODOs
• a (human) pre-merge review of all unresolved review comments in the PR (Github autominimizes most review comment threads when their associated code has been changed, and as of recently lets you manually mark review comment threads as resolved)
• changes in the release process — why not release a beta first and then wait several weeks before a general release?
• etc

A postmortem could result in a number of helpful betterments to ensure that software that is as core to the Python project as pip doesn't ship with bugs of this magnitude in future.

I can replicate this bug. Removing --no-cache-dir fixes it. Since I don't want it in my docker image I'm using the solution @coderanger proposed. Cheers 🌈🍰🌈

This looks like a duplicate of Issue #6166

Quick and easy reproduction Dockerfile:

FROM buildpack-deps:buster
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
RUN apt-get update && apt-get install -y --no-install-recommends python3-dev && rm -rf /var/lib/apt/lists/*
RUN curl https://bootstrap.pypa.io/get-pip.py | python3 - --no-cache-dir

simply removing the assertion might be the correct fix

Not exactly -- I think we should keep this there for the non ephem builds. I'll file a bugfix PR, once I'm done with breakfast. :)

@pradyunsg check my PR for a failing test

For me, pip v19.0 will fail to install anything if the --no-cache (or --no-cache-dir) option is used.

I've filed #6171 as a bugfix for this issue. Could folks from this thread try out that PR and verify that it indeed fixes this issue?

PS: Thanks @tgs for filing a PR to aid fixing this issue quickly! :)

wfm, thanks for the fix!

$ pip install pip --upgrade
Requirement already up-to-date: pip in ./venv/lib/python3.6/site-packages (19.0)
$ pip install --no-cache-dir pip
Requirement already satisfied: pip in ./venv/lib/python3.6/site-packages (19.0)
Exception:
Traceback (most recent call last):
  File "/tmp/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/tmp/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/tmp/venv/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
$ pip install git+https://github.com/pradyunsg/pip@fix/pep-517-building-assertion
Collecting git+https://github.com/pradyunsg/pip@fix/pep-517-building-assertion
  Cloning https://github.com/pradyunsg/pip (to revision fix/pep-517-building-assertion) to ./pip-req-build-g_3qep31
Branch 'fix/pep-517-building-assertion' set up to track remote branch 'fix/pep-517-building-assertion' from 'origin'.
Switched to a new branch 'fix/pep-517-building-assertion'
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-sb1_muik/wheels/bd/86/cd/7688dba746eabc598fb37d4a93e2ff9bd05a6d9f907ee7b6cd
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 19.0
    Uninstalling pip-19.0:
      Successfully uninstalled pip-19.0
Successfully installed pip-19.1.dev0
$ pip install --no-cache-dir astpretty  # downloads a wheel
Collecting astpretty
  Downloading https://files.pythonhosted.org/packages/9d/10/cb0c3a3edb16f45be05bdba7f37798fcddb8cf085def8cb6e62b2ad7c711/astpretty-1.4.1-py2.py3-none-any.whl
Installing collected packages: astpretty
Successfully installed astpretty-1.4.1
$ pip install --no-cache-dir simplejson  # requires building
Collecting simplejson
  Downloading https://files.pythonhosted.org/packages/e3/24/c35fb1c1c315fc0fffe61ea00d3f88e85469004713dab488dee4f35b0aff/simplejson-3.16.0.tar.gz (81kB)
    100% |████████████████████████████████| 81kB 1.0MB/s 
Installing collected packages: simplejson
  Running setup.py install for simplejson ... done
Successfully installed simplejson-3.16.0

I hope merge PR6171 soon and release version 19.0.1

People should really pin pip in CI like you would do for any other package or dependency, IMO. Otherwise, you don't have reproducibility and risk sudden breakages. By pinning, you can check things in advance and upgrade at your own pace.

People should really pin pip in CI like you would do for any other package or dependency, IMO. Otherwise, you don't have reproducibility and risk sudden breakages. By pinning, you can check things in advance and upgrade at your own pace.

@cjerdonek: I agree with you that — from a pip user perspective — it is a good idea to pin pip in many (perhaps most) contexts. At the very least, if you don't pin you should know that you're risking exactly this sort of thing, and you can't complain to the pip maintainers if it happens!

However... from a pip maintainer perspective (and more broadly from a PyPA or Python core team perspective) I think it would be prudent to view the fact that a great many people don't pin pip as an asset. It's intangible, but demonstrates great trust by the userbase. (As an aside, counterintuitively perhaps, in my experience It's often the case that the most core tools are not pinned like most dependencies. For example, where I work, python itself is usually effectively pinned to a minor version — new patch versions automatically get picked up by new builds. I think this shows the elevated level of trust that users have in these core tools and their maintainership.)

Incidents like this erode that trust. The broken CI builds are not the actual issue (as you say, you either should pin pip in CI builds or be aware what you're risking), but are a symptom — or rather, a correlated warning — of that eroded trust.

That's why I proposed that this incident merits some sort of (blameless) postmortem process. No pip maintainer should be feeling bad right now, but, this is a serious issue and should be examined for betterments.

Yeah, incidents like this don't help build trust. We will have a post mortem to figure out how to avoid these (we do that after every release because there's always scope for improvement).

Thanks for (mostly) maintaining proper discourse and for the constructive comments y'all! Usually, things are a lot more corrosive for us when there's an issue like this. :)

None the less, there's a couple other bug reports to look into and we'll have a 19.0.1 soon enough.

I think the key thing here is that this has exposed that we don't have sufficient tests for building under --no-cache-dir. Additional tests in that area would be a huge help in avoiding regressions like this, and more generally a review of what "key" functionality is under-tested would be helpful.

As a pip maintainer, I'd say that one problem I have is knowing what people consider "key" functionality. Personally, I would have assumed --no-cache-dir was fairly niche, so obviously my intuition isn't reliable in this case :-) Hence feedback like this is particularly valuable.

I think it can be released 19.0.1 soon only for this bug , after all, it is significant and urgent . Another bug reports can solve in 19.0.2 after day.

As a pip maintainer, I'd say that one problem I have is knowing what people consider "key" functionality. Personally, I would have assumed --no-cache-dir was fairly niche, so obviously my intuition isn't reliable in this case :-) Hence feedback like this is particularly valuable.

The sole reason why I use --no-cache-dir, is for installing mpi4py.
This way, I can enforce that it is redownloaded and rebuilt before installing, making sure that any changes made to my MPI distribution are taken into account.

Same issue here, able to reproduce it outside our CI system. As a workaround we've downgraded to pip 18.1.0 and everything works:

pip install pip==18.1.0

Hope and upgrade soon.

I will use:

pip install "pip!=19.0"

Hope 19.1 is fixed :)

I suspect we'll have a 19.0.1 relatively soon with fixes for the emergent issues.

I'm curious if including --no-use-pep517 along with --no-cache-dir is another work-around for this issue, as it is for another PEP 517 related issue: https://github.com/pypa/pip/issues/6163#issuecomment-456772043

As a pip maintainer, I'd say that one problem I have is knowing what people consider "key" functionality. Personally, I would have assumed --no-cache-dir was fairly niche, so obviously my intuition isn't reliable in this case :-) Hence feedback like this is particularly valuable.

FWIW: I use --no-cache-dir pretty often when building Docker images, to prevent the chances of any cache cruft getting left around in an environment where it won't be useful.

People should really pin pip in CI like you would do for any other package or dependency, IMO. Otherwise, you don't have reproducibility and risk sudden breakages. By pinning, you can check things in advance and upgrade at your own pace.

In Many environments pip is not a dependency. It's installed when creating the virtualenv.

And by the way isn't testing there to see if your product works with the most recent versions. Pinning everything just leads to old versions used. And updating will soon be a task nobody dares to start. Been there, done that. So my opinion is to pin only if really necessary. And try to fix problem as soon as they are occur.

pip 19.0.1 is out with the fix for this issue.

I was excited to see the new 19.0.1 version fix, but still having an issue. I'm also creating a Docker image with --no-cache-dir that works fine with pip < 19.0. Anyone else getting this?

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/usr/lib/python3.6/site-packages/pip/_internal/wheel.py", line 886, in build
    assert have_directory_for_build
AssertionError

I was excited to see the new 19.0.1 version fix, but still having an issue. I'm also creating a Docker image with --no-cache-dir that works fine with pip < 19.0. Anyone else getting this?

fix is working for me with 19.0.1 -- I suspect you've got a docker layer cache that's confusing you? -- try pip --version to check what version you're on

I have a Python and pip version check in all my Docker files, and it reports 19.0.1 correctly.

@dmulter I rebuilt my Docker images in my Gist from scratch this morning and things are working fine there with v19.0.1. Can you share your Dockerfile in a Gist also so we can all look at it?

Just cleaned everything again just to be sure. Here's the Dockerfile and my build output.

See my note on the build output for the docker commands that were used.

Is there a fix for pip3? Heres the error I got...

> pip3 install --upgrade 'pip>=19.01' setuptools

  Could not find a version that satisfies the requirement pip>=19.01 (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0)
No matching distribution found for pip>=19.01
You are using pip version 10.0.1, however version 19.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@MrAtheist You have a small typo/missing a decimal. The patch release is 19.0.1 but you have 19.01 written.

oops my mistake, but either way, the possible versions dont have 19.0.1 listed... ¯_(ツ)_/¯

Like @dmulter I am finding the issue still unresolved. Extract from build output:

. venv/bin/activate;  python -m pip install --upgrade pip; python -m pip install ndg_httpsclient; python -m pip install . -i https://xxxx.yyyy.com/simple --upgrade --no-cache-dir flask
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in ./venv/lib/python2.7/site-packages (19.0.1)
...
Requirement already satisfied, skipping upgrade: pycparser in ./venv/lib/python2.7/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko<3.0,>=1.10->Fabric==1.14.0->conference-gll-load-test===0.0.1-SNAPSHOT) (2.19)
Exception:
Traceback (most recent call last):
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 886, in build
    assert have_directory_for_build
AssertionError
make: *** [install] Error 2

Earlier in the thread I had asked if including --no-use-pep517 along with --no-cache-dir makes things work for people, but I didn't see a reply. For people that are still experiencing the option, can you try that?

Adding the --no-use-pep517 option fixed the problem for me. Hope that helps narrow things down.

pip 19.0.1 working for me in a virtualenv. But inside Jenkins (Shining Panda) it still fails. Adding --no-use-pep517 fixes the problem

I'm reopening since some people are still experiencing the same issue.

I can also confirm that --no-use-pep517 fixed the issue for me after upgrading to pip 19.0.1 did not.

But why all those projects have to adapt whenever pip gets new version?

At @pradyunsg's request, I've opened a new issue (https://github.com/pypa/pip/issues/6197) specific to the AssertionError in the 19.0.1 release, as it's narrower in scope and will need new investigation. So I'm reclosing this issue.

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

or your FROM python:3.6-alpine can change to FROM python:3.6.7-alpine

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.