Virtualenv: pipx.y missing, instead pip-x.y present

Created on 15 Mar 2020 · 4Comments · Source: pypa/virtualenv

Issue

In virtualenv 20.0.10 at least when a virtualenv environment is created the naming convention of the pip major version file has changed.

In 16.7.x there was pip3.7 in 20.0.2 this has changed to pip-3.7 although there is still pip3 in both, in terms of automation patterns that provision multiple environments and builds there may be many build dependencies that use the pip${MAJOR_VERSION} pattern.

Perhaps adding a symlink to honour the older naming convention may prevent many people spending many hours debugging.

Environment

OS: CentOS 8
OS Python: Python 3.6.8
python interpretor being used for the virtualenv is Python 3.7.6 compiled from source.
System pip list

pip list --format=columns

Package             Version
------------------- -------
appdirs             1.4.3
asn1crypto          0.24.0
cffi                1.11.5
configobj           5.0.6
cryptography        2.3
dbus-python         1.2.4
decorator           4.2.1
distlib             0.3.0
filelock            3.0.12
gpg                 1.10.0
idna                2.5
importlib-metadata  1.5.0
importlib-resources 1.3.1
iniparse            0.4
isc                 2.0
netifaces           0.10.6
pciutils            2.3.6
perf                0.1
pip                 9.0.3
ply                 3.9
pycparser           2.14
pygobject           3.28.3
pyOpenSSL           18.0.0
pyparsing           2.1.10
python-dateutil     2.6.1
python-dmidecode    3.12.2
python-linux-procfs 0.6
pyudev              0.21.0
rhnlib              2.8.6
rpm                 4.14.2
schedutils          0.6
selinux             2.9
sepolicy            1.1
setools             4.2.2
setuptools          39.2.0
six                 1.11.0
slip                0.6.4
slip.dbus           0.6.4
syspurpose          1.25.17
zipp                3.1.0

Reproduce

cd /opt
pip install virtualenv==20.0.10

pip list --format=columns | grep virtualenv
virtualenv          20.0.10

virtualenv --python=/opt/python_virtualenv/versions/3.7.6/bin/python3.7 venv_20.0.10

ls -al /opt/venv_20.0.10/bin | grep pip
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip3
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip-3.7

pip uninstall virtualenv
pip install virtualenv==16.7.10
virtualenv --python=/opt/python_virtualenv/versions/3.7.6/bin/python3.7 venv_16.7.10

ls -al /opt/venv_16.7.10/bin | grep pip
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip3
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip3.7

ls -al /opt/venv_20.0.10/bin
total 68
drwxr-xr-x. 2 root root 4096 Mar 15 14:08 .
drwxr-xr-x. 4 root root 4096 Mar 15 14:08 ..
-rw-r--r--. 1 root root 2206 Mar 15 14:08 activate
-rw-r--r--. 1 root root 1425 Mar 15 14:08 activate.csh
-rw-r--r--. 1 root root 3056 Mar 15 14:08 activate.fish
-rw-r--r--. 1 root root 1751 Mar 15 14:08 activate.ps1
-rw-r--r--. 1 root root 1199 Mar 15 14:08 activate_this.py
-rw-r--r--. 1 root root 1147 Mar 15 14:08 activate.xsh
-rwxr-xr-x. 1 root root  242 Mar 15 14:08 easy_install
-rwxr-xr-x. 1 root root  242 Mar 15 14:08 easy_install3
-rwxr-xr-x. 1 root root  242 Mar 15 14:08 easy_install-3.7
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip3
-rwxr-xr-x. 1 root root  233 Mar 15 14:08 pip-3.7
lrwxrwxrwx. 1 root root   51 Mar 15 14:08 python -> /opt/python_virtualenv/versions/3.7.6/bin/python3.7
lrwxrwxrwx. 1 root root    6 Mar 15 14:08 python3 -> python
lrwxrwxrwx. 1 root root    6 Mar 15 14:08 python3.7 -> python
-rwxr-xr-x. 1 root root  220 Mar 15 14:08 wheel
-rwxr-xr-x. 1 root root  220 Mar 15 14:08 wheel3
-rwxr-xr-x. 1 root root  220 Mar 15 14:08 wheel-3.7


ls -al /opt/venv_16.7.10/bin
total 15360
drwxr-xr-x. 2 root root     4096 Mar 15 14:10 .
drwxr-xr-x. 5 root root     4096 Mar 15 14:10 ..
-rw-r--r--. 1 root root     2206 Mar 15 14:10 activate
-rw-r--r--. 1 root root     1425 Mar 15 14:10 activate.csh
-rw-r--r--. 1 root root     3090 Mar 15 14:10 activate.fish
-rw-r--r--. 1 root root     1751 Mar 15 14:10 activate.ps1
-rw-r--r--. 1 root root     1517 Mar 15 14:10 activate_this.py
-rw-r--r--. 1 root root     1147 Mar 15 14:10 activate.xsh
-rwxr-xr-x. 1 root root      245 Mar 15 14:10 easy_install
-rwxr-xr-x. 1 root root      245 Mar 15 14:10 easy_install-3.7
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip3
-rwxr-xr-x. 1 root root      236 Mar 15 14:10 pip3.7
lrwxrwxrwx. 1 root root        9 Mar 15 14:10 python -> python3.7
lrwxrwxrwx. 1 root root        9 Mar 15 14:10 python3 -> python3.7
-rwxr-xr-x. 1 root root 15664944 Mar 15 14:10 python3.7
-rwxr-xr-x. 1 root root     2335 Mar 15 14:10 python-config
-rwxr-xr-x. 1 root root      223 Mar 15 14:10 wheel

bug

Source

earthgecko

All 4 comments

This is a side effect of adopting the standard library that handles script creation, script variant added here https://bitbucket.org/pypa/distlib/commits/eedb3ae9233562e970dab2212892969bd87be01c. Not totally sure why 16.x was adding them without the dash... might need to check if pip escapes distlibs behaviour, but AFAIK pip should also generate similar styles.

gaborbernat on 15 Mar 2020

👍1

So seems here pip does something odd, compared to rest, though maybe it's in line with how Linux distributions handle things, and how python itself versions:

see here https://github.com/pypa/pip/blob/master/setup.py#L79-L81, pip seems to hardcode a pipx.y format; it's a bit off as the console script is written at build time, so in case of the universal wheel if you install on different python (e.g. wheel built with 3.8, install on 2.7) pip has to rewrite the version number to target python version, somehow not respecting what's defined within the console script mapping, and altering the name; notably, it also uses a pipx.y format.
distlib prefers the pip-x.y format, per https://bitbucket.org/pypa/distlib/commits/eedb3ae9233562e970dab2212892969bd87be01c
setuptools prefers the pip-x.y format too, see https://github.com/pypa/setuptools/blob/master/setup.py#L46-L47
wheels do not define versioned executables at all, see https://github.com/pypa/wheel/blob/master/setup.cfg#L49

Seems to me like we never standardized how tools should define versioned scripts, and how the frontend can communicate to the backend that it wants such. Or should be a frontend option? @pfmoore @pradyunsg thoughts feels like should be part of PEP-517 somewhere 🤷‍♂

For now, I think it's a workaround to also generate pipx.y and pip-x.y...

gaborbernat on 18 Mar 2020

Seems to me like we never standardized how tools should define versioned scripts, and how the frontend can communicate to the backend that it wants such.

This is basically the situation.

Pip generates versions that don't match distlib because that was the format pip had used before we switched to distlib, and backward compatibility was a concern (IIRC).

For a longer-term solution, this probably should be standardised (actually, the whole console-scripts mechanism isn't standardised, so it probably needs a new PEP covering the whole of this area).

pfmoore on 18 Mar 2020

👍1

Hello, a fix for this issue has been released via virtualenv 20.0.11; see https://pypi.org/project/virtualenv/20.0.11/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-11-2020-03-18). Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks