Virtualenv: lib64 directory is no longer symlinked to lib?

Created on 26 Mar 2020  ·  4Comments  ·  Source: pypa/virtualenv

Issue

Came across this issue while debugging tox environment creation issues. It would seem that when using virtualenv==16.7.10 or simply python -m venv, the lib64 directory was a symlink to the lib directory as expected. However, in virtualenv==20.0.14, this is no longer the case. Here, lib64 is created as a directory.

For example, the following will reproduce the issue.

docker run --rm -i fedora:31 <<EOF
set -x
dnf --quiet install -y pip
for v in 16.7.10 20.0.14; do
  pip install virtualenv==\$v >/dev/null 2>&1
  virtualenv venv-\$v
  ls -all venv-\$v
done
EOF

The following, is a snippet from the output.

+ pip install virtualenv==16.7.10
+ virtualenv venv-16.7.10
Using base prefix '/usr'
New python executable in /venv-16.7.10/bin/python3
Also creating executable in /venv-16.7.10/bin/python
Installing setuptools, pip, wheel...
done.
+ ls -all venv-16.7.10
total 28
drwxr-xr-x. 5 root root 4096 Mar 26 03:35 .
drwxr-xr-x. 1 root root 4096 Mar 26 03:35 ..
drwxr-xr-x. 2 root root 4096 Mar 26 03:36 bin
drwxr-xr-x. 2 root root 4096 Mar 26 03:35 include
drwxr-xr-x. 3 root root 4096 Mar 26 03:35 lib
lrwxrwxrwx. 1 root root    3 Mar 26 03:35 lib64 -> lib
...
+ pip install virtualenv==20.0.14
+ virtualenv venv-20.0.14
created virtual environment CPython3.7.6.final.0-64 in 249ms
  creator CPython3Posix(dest=/venv-20.0.14, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
+ ls -all venv-20.0.14
total 32
drwxr-xr-x. 5 root root 4096 Mar 26 03:36 .
drwxr-xr-x. 1 root root 4096 Mar 26 03:36 ..
drwxr-xr-x. 2 root root 4096 Mar 26 03:36 bin
drwxr-xr-x. 3 root root 4096 Mar 26 03:36 lib
drwxr-xr-x. 3 root root 4096 Mar 26 03:36 lib64
-rw-r--r--. 1 root root  202 Mar 26 03:36 pyvenv.cfg

Environment

Provide at least:

  • OS: Fedora release 31 (Thirty One), Linux 7ab2b964bf3a 5.5.10-200.fc31.x86_64 #1 SMP Wed Mar 18 14:21:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • pip list of the host python where virtualenv is installed:

```console
Package Version


appdirs 1.4.3
distlib 0.3.0
filelock 3.0.12
gpg 1.13.1
importlib-metadata 1.5.2
libcomps 0.1.14
pip 19.1.1
rpm 4.15.1
setuptools 41.6.0
six 1.14.0
virtualenv 20.0.14
zipp 3.1.0
```

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

75 setup logging to NOTSET [DEBUG report:43]
75 created app data folder /root/.local/share/virtualenv [DEBUG app_data:52]
84 find interpreter for spec PythonSpec(path=/usr/bin/python3) [INFO builtin:44]
84 proposed PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.7.6 (default, Jan 30 2020, 09:44:41) \n[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
84 accepted PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.7.6 (default, Jan 30 2020, 09:44:41) \n[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:52]
91 filesystem is case-sensitive [DEBUG info:28]
160 create virtual environment via CPython3Posix(dest=/venv-20.0.14, clear=False, global=False) [INFO session:52]
160 create folder /venv-20.0.14/bin [DEBUG _sync:21]
160 create folder /venv-20.0.14/lib/python3.7/site-packages [DEBUG _sync:21]
160 create folder /venv-20.0.14/lib64/python3.7/site-packages [DEBUG _sync:21]
161 write /venv-20.0.14/pyvenv.cfg [DEBUG pyenv_cfg:34]
161     home = /usr [DEBUG pyenv_cfg:38]
161     implementation = CPython [DEBUG pyenv_cfg:38]
161     version_info = 3.7.6.final.0 [DEBUG pyenv_cfg:38]
161     virtualenv = 20.0.14 [DEBUG pyenv_cfg:38]
161     include-system-site-packages = false [DEBUG pyenv_cfg:38]
161     base-prefix = /usr [DEBUG pyenv_cfg:38]
161     base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
161     base-executable = /usr/bin/python3 [DEBUG pyenv_cfg:38]
161 symlink /usr/bin/python3 to /venv-20.0.14/bin/python [DEBUG _sync:40]
162 create virtualenv import hook file /venv-20.0.14/lib/python3.7/site-packages/_virtualenv.pth [DEBUG api:76]
163 create /venv-20.0.14/lib/python3.7/site-packages/_virtualenv.py [DEBUG api:79]
163 ============================== target debug ============================== [DEBUG session:54]
163 debug via /venv-20.0.14/bin/python /usr/local/lib/python3.7/site-packages/virtualenv/create/debug.py [DEBUG creator:187]
163 {
  "sys": {
    "executable": "/venv-20.0.14/bin/python",
    "_base_executable": null,
    "prefix": "/venv-20.0.14",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/venv-20.0.14",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib64/python37.zip",
      "/usr/lib64/python3.7",
      "/usr/lib64/python3.7/lib-dynload",
      "/venv-20.0.14/lib64/python3.7/site-packages",
      "/venv-20.0.14/lib/python3.7/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.7.6 (default, Jan 30 2020, 09:44:41) \n[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]",
  "os": "<module 'os' from '/usr/lib64/python3.7/os.py'>",
  "site": "<module 'site' from '/usr/lib64/python3.7/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib64/python3.7/datetime.py'>",
  "math": "<module 'math' from '/usr/lib64/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/usr/lib64/python3.7/json/__init__.py'>"
} [DEBUG session:55]
201 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1) [INFO session:59]
202 Attempting to acquire lock 140367118139024 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels.lock [DEBUG filelock:270]
202 Lock 140367118139024 acquired on /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels.lock [INFO filelock:274]
203 get bundled wheel /usr/local/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/pip-20.0.2-py2.py3-none-any.whl [DEBUG acquire:51]
204 get bundled wheel /usr/local/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/setuptools-46.1.1-py3-none-any.whl [DEBUG acquire:51]
204 get bundled wheel /usr/local/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/wheel-0.34.2-py2.py3-none-any.whl [DEBUG acquire:51]
209 install wheel from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
210 install setuptools from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels/setuptools-46.1.1-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
210 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any of wheel-0.34.2-py2.py3-none-any.whl [DEBUG base:57]
210 install pip from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
210 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any of setuptools-46.1.1-py3-none-any.whl [DEBUG base:57]
211 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any of pip-20.0.2-py2.py3-none-any.whl [DEBUG base:57]
236 changing mode of /tmp/tmppzoh5m0f/wheel-3.7 to 755 [INFO util:566]
237 changing mode of /tmp/tmppzoh5m0f/wheel to 755 [INFO util:566]
238 changing mode of /tmp/tmppzoh5m0f/wheel3 to 755 [INFO util:566]
243 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /venv-20.0.14/lib/python3.7/site-packages/wheel [DEBUG _sync:48]
256 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /venv-20.0.14/lib/python3.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:48]
262 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /venv-20.0.14/lib/python3.7/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:48]
263 changing mode of /venv-20.0.14/bin/wheel-3.7 to 755 [INFO util:566]
264 changing mode of /venv-20.0.14/bin/wheel to 755 [INFO util:566]
264 changing mode of /venv-20.0.14/bin/wheel3 to 755 [INFO util:566]
265 generated console scripts wheel-3.7 wheel wheel3 [DEBUG base:53]
296 changing mode of /tmp/tmp1h0cn9gf/easy_install to 755 [INFO util:566]
297 changing mode of /tmp/tmp1h0cn9gf/easy_install3 to 755 [INFO util:566]
297 changing mode of /tmp/tmp1h0cn9gf/easy_install-3.7 to 755 [INFO util:566]
306 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any/setuptools-46.1.1.virtualenv to /venv-20.0.14/lib/python3.7/site-packages/setuptools-46.1.1.virtualenv [DEBUG _sync:48]
307 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any/pkg_resources to /venv-20.0.14/lib/python3.7/site-packages/pkg_resources [DEBUG _sync:48]
313 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any/easy_install.py to /venv-20.0.14/lib/python3.7/site-packages/easy_install.py [DEBUG _sync:48]
313 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any/setuptools-46.1.1.dist-info to /venv-20.0.14/lib/python3.7/site-packages/setuptools-46.1.1.dist-info [DEBUG _sync:48]
317 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.1-py3-none-any/setuptools to /venv-20.0.14/lib/python3.7/site-packages/setuptools [DEBUG _sync:48]
344 changing mode of /venv-20.0.14/bin/easy_install to 755 [INFO util:566]
345 changing mode of /venv-20.0.14/bin/easy_install3 to 755 [INFO util:566]
345 changing mode of /venv-20.0.14/bin/easy_install-3.7 to 755 [INFO util:566]
345 generated console scripts easy_install easy_install-3.7 easy_install3 [DEBUG base:53]
383 changing mode of /tmp/tmp483fsq8a/pip-3.7 to 755 [INFO util:566]
383 changing mode of /tmp/tmp483fsq8a/pip3.7 to 755 [INFO util:566]
384 changing mode of /tmp/tmp483fsq8a/pip to 755 [INFO util:566]
384 changing mode of /tmp/tmp483fsq8a/pip3 to 755 [INFO util:566]
407 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.virtualenv to /venv-20.0.14/lib/python3.7/site-packages/pip-20.0.2.virtualenv [DEBUG _sync:48]
408 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /venv-20.0.14/lib/python3.7/site-packages/pip [DEBUG _sync:48]
460 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /venv-20.0.14/lib/python3.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:48]
461 changing mode of /venv-20.0.14/bin/pip-3.7 to 755 [INFO util:566]
461 changing mode of /venv-20.0.14/bin/pip3.7 to 755 [INFO util:566]
461 changing mode of /venv-20.0.14/bin/pip to 755 [INFO util:566]
462 changing mode of /venv-20.0.14/bin/pip3 to 755 [INFO util:566]
462 generated console scripts pip-3.7 pip3.7 pip3 pip [DEBUG base:53]
462 Attempting to release lock 140367118139024 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels.lock [DEBUG filelock:315]
462 Lock 140367118139024 released on /root/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels.lock [INFO filelock:318]
462 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:65]
464 write /venv-20.0.14/pyvenv.cfg [DEBUG pyenv_cfg:34]
465     home = /usr [DEBUG pyenv_cfg:38]
465     implementation = CPython [DEBUG pyenv_cfg:38]
465     version_info = 3.7.6.final.0 [DEBUG pyenv_cfg:38]
465     virtualenv = 20.0.14 [DEBUG pyenv_cfg:38]
465     include-system-site-packages = false [DEBUG pyenv_cfg:38]
465     base-prefix = /usr [DEBUG pyenv_cfg:38]
465     base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
465     base-executable = /usr/bin/python3 [DEBUG pyenv_cfg:38]
465 created virtual environment CPython3.7.6.final.0-64 in 392ms
  creator CPython3Posix(dest=/venv-20.0.14, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:21]
question

Most helpful comment

Not sure who added the downstream changes; is it an OS patch or upstream CPython? I think the intention here is to keep them separate; that being said merging them can also have some merit 🤷‍♂ though I consider it more error-prone.

All 4 comments

@abn this is the intention and by design. The lib does not have to be linked to the lib64 directory. There are two separate library paths, called platlib and purelib. This might differ and virtualenv < 20 solution was to link them, and as such merge them into the same; I consider this a bad solution as you're mixing those two together, so you might end up with conflicts where tools expect them to be separate; so with virtualenv 20 we handle them separately, as most tools also do.

@gaborbernat thanks for the clarification. I was unsure if this was by design. The confusion arose as the python3.7 -m venv behaviour was the same as with < 20. Will deal with both paths downstream.

Not sure who added the downstream changes; is it an OS patch or upstream CPython? I think the intention here is to keep them separate; that being said merging them can also have some merit 🤷‍♂ though I consider it more error-prone.

I am not entirely sure on that one. But I do agree that keeping them separate is probably the better choice here.

Was this page helpful?
0 / 5 - 0 ratings