Pip: Every single command from pip runs super slow

What are you referring by “it used to”? Does this slowdown happen because of a pip upgrade or a system upgrade? If it happens out of nowhere, it is very likely not a pip problem, but something happening on your particular machine, which pip does not have control over.

No, I didn't do any pip upgrade. Windows 10 does make frequent updates but I cannot fathom how this impacts performance with pip inside WSL2.

I'm hoping someone could point me a direction on how I can even trace this problem. Right now pip doesn't output any error messages so no way to know what's going on.

Could you share how many packages are already installed (i.e. the output of pip list)? It might be related to pip's internal logic [1] that tried to look at all the installed packages before any output processing.

[1] pkg_resources's WorkingSet construction on load, for those wondering what I'm talking about

Same issue with me too. I ran pip3 list command and it took around 10+ seconds to list out packages. For now I create virtual environment using pipenv which removes the problem. I think it might be interfering with the sharing of executables between wsl2 linux and windows. I'm not sure what is the source of problem though!

I think it might be interfering with the sharing of executables between wsl2 linux and windows

This sounds plausible. WSL2’s filesystem performance is terrible if you access the Windows filesystem at the Linux side. What is the Python associated to your pip3 command? Can you provide its sys.path? Does this happen if you run pip3 at a different location? Does it matter if the location is in the Linux filesystem or the Windows side?

If we run same pip3 list on windows powershell, it's instant and the issue doesn't happen.

Sys path without pipenv environment activated

['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/<user>/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

Sys path with pipenv environment activated

['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/<user>/.local/share/virtualenvs/myproj-SiazyaGz/lib/python3.8/site-packages']

It could be possible if sys path is changed for global, then pip3 won't interfere with windows one. I haven't tried it out!

Hmm, none of the directories seem out of ordinary. Do you happen to mount any of the Windows directories? For example, do you symlink your home directory (or anything listed in sys.path) to a Windows directory? Or do you run the command in a directory under /mnt?

I would probably try to temporarily move some of the entries listed in sys.path away (espacially the ~/.local/lib one) and see if that changes anything. Or maybe you can just insert a few profilers into the Python runtime and see what exactly are slowing things down. There are many tricks you can do to isolate the problem. That would be immensely more useful than me (or anyone without physical access to your machine) trying to troubleshoot against thin air.

Edit: This may somehow be linked to the display? The slowdown largely goes away (takes 0.5 seconds) when I start up an X11 server (using MobaXterm). The reason I found this issue was that matplotlib was super slow so I tried to use pip to reinstall. I forgot that I needed to run an Xterm to use matplotlib.

I have also encountered this problem and have the same output from my python's sys.path as piyushchauhan2011. I have a symlink in my home directory to a Windows directory like so
test -> /mnt/c/Users/<user>/Documents/<git_project_folder>/

I'm developing a python package whose results need to be bench marked against a program that can only run in a unix environment. I use sublime test to edit the files in my windows directory and use the WSL2 to run the benchmark code on those files through the symlink.

I can run the following commands without any slowdown: check, show, config
I'm using pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

Running pip3 list does return a result but it takes ~ 30 seconds.
I have about 100 packages installed.

I tried the following without any success
python3 -m pip --retries 2 --timeout 5 --no-cache-dir --isolated --verbose list
I have tried running pip both inside the WSL2 filesystem and at the Windows side each of them suffering the same slowdown.

I'm not 100% what the best way of changing the sys.path would be but here is my attempt:
I launched ipython3 which starts with a sys.path of:

'/usr/bin',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '',
 '/home/<user>/.local/lib/python3.8/site-packages',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/usr/lib/python3/dist-packages/IPython/extensions',
 '/home/<user>/.ipython']

Which I backup as follows backup = sys.path.copy()
At this point I confirmed that if I use run '/usr/bin/pip3' list I still encounter the slowdown.
However, when after that I set sys.path = [] and run again I get a ModuleNotFoundError: No module named 'pyparsing'. This result repeats each time I run again. BUT! Once I set sys.path = backup now run '/usr/bin/pip3' list miraculously works!
The output of using time:

CPU times: user 12.2 ms, sys: 426 µs, total: 12.6 ms
Wall time: 11.8 ms

So clearly there is something wrong with the paths.
Afterwards I can again set sys.path =[] and run '/usr/bin/pip3' list still works for some reason.

I'm not sure if this is relevant but I figured I'd mention it:
After using the run command my sys.path gets populated as follows (after having been set to an empty list)

['/usr/share/python-wheels/idna-2.8-py2.py3-none-any.whl',
 '/usr/share/python-wheels/distlib-0.3.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/msgpack-0.6.2-py2.py3-none-any.whl',
 '/usr/share/python-wheels/lockfile-0.12.2-py2.py3-none-any.whl',
 '/usr/share/python-wheels/pytoml-0.1.21-py2.py3-none-any.whl',
 '/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl',
 '/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/setuptools-44.0.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/pep517-0.8.2-py2.py3-none-any.whl',
 '/usr/share/python-wheels/chardet-3.0.4-py2.py3-none-any.whl',
 '/usr/share/python-wheels/webencodings-0.5.1-py2.py3-none-any.whl',
 '/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl',
 '/usr/share/python-wheels/ipaddr-2.2.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/certifi-2019.11.28-py2.py3-none-any.whl',
 '/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl',
 '/usr/share/python-wheels/wheel-0.34.2-py2.py3-none-any.whl',
 '/usr/share/python-wheels/appdirs-1.4.3-py2.py3-none-any.whl',
 '/usr/share/python-wheels/packaging-20.3-py2.py3-none-any.whl',
 '/usr/share/python-wheels/html5lib-1.0.1-py2.py3-none-any.whl',
 '/usr/share/python-wheels/six-1.14.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/pip-20.0.2-py2.py3-none-any.whl',
 '/usr/share/python-wheels/colorama-0.4.3-py2.py3-none-any.whl',
 '/usr/share/python-wheels/progress-1.5-py2.py3-none-any.whl',
 '/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/pyparsing-2.4.6-py2.py3-none-any.whl',
 '/usr/share/python-wheels/contextlib2-0.6.0-py2.py3-none-any.whl',
 '/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl',
 '/usr/bin',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '',
 '/home/<user>/.local/lib/python3.8/site-packages',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/usr/lib/python3/dist-packages/IPython/extensions',
 '/home/<user>/.ipython']

Which still suffers the slowdown until the sys.path is set to an empty list and then reset back to the original list or this list.

This may somehow be linked to the display? The slowdown largely goes away (takes 0.5 seconds) when I start up an X11 server (using MobaXterm). The reason I found this issue was that matplotlib was super slow so I tried to use pip to reinstall. I forgot that I needed to run an Xterm to use matplotlib.

Maybe…? The entirely issue is very weird to me. If this is a sys.path issue, wouldn’t the same slowdown happen for all Python imports, not just pip? I am quite stumped 😞

Hello, my environment is:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

With Python 3.6.9 and pip 9.0.1 and I can confirm that pip is extremely slow with every command (especially with install)

My pip3 list is

asn1crypto (0.24.0)
attrs (17.4.0)
Automat (0.6.0)
chardet (3.0.4)
configobj (5.0.6)
constantly (15.1.0)
cryptography (2.1.4)
distro-info (0.18ubuntu0.18.04.1)
hyperlink (17.3.1)
idna (2.6)
incremental (16.10.1)
keyring (10.6.0)
keyrings.alt (3.0)
netifaces (0.10.4)
pip (9.0.1)
pyasn1 (0.4.2)
pyasn1-modules (0.2.1)
pycrypto (2.6.1)
pygobject (3.26.1)
pyOpenSSL (17.5.0)
python-apt (1.6.5+ubuntu0.3)
python-debian (0.1.32)
pyxdg (0.25)
PyYAML (3.12)
SecretStorage (2.3.1)
service-identity (16.0.0)
setuptools (39.0.1)
six (1.11.0)
Twisted (17.9.0)
ufw (0.36)
unattended-upgrades (0.1)
wheel (0.30.0)

@MattiaFailla If you want a _temporary_ fix I recommend running an X11 server such as MobaXterm(or some other equivalent). I don't understand why, but it did fix my slowdown issue for all commands.

@ngraymon that’s odd but I will try this temp-fix.
Thanks!
I'll update this issue once tried.

Just to confirm that the behavior is still the same I checked just now:

Running time pip3 list inside a Windows Terminal on WSL2:

After starting MobaXterm and running time pip3 list in the same terminal:

@ngraymon Hello,
I have resolved the issue, please try the following steps:

• Do not run pip commands with sudo
• apt-update && apt-upgrade
• Reboot the server/computer
• Keep close attention to docker, last night I noticed that the python3 process was being intensely used by the swarm

@MattiaFailla
I'm glad you resolved your issue.
I tried your suggestions but it did not solve the issue.
I don't run pip with sudo, but I did install pip3 using sudo apt install python3-pip, maybe that is relevant?
I'm happy with the way things are for myself as I need an X server anyways since I'm plotting using matplotlib.

@ngraymon could you run python -m pip, and see if that is also slow?

If it is, and you have a new-enough Python version, please provide us the output of python -X importtime -m pip -v. If the slowdown is in the imports, this will help us know.

@pradyunsg
Hello,

I ran time python3 -m pip without any command for pip which replies with the help message in this time

However if I run time python3 -m pip list

Running time python3 -m pip check which has not/ still isn't affected by the slow down

I ran the following:

• python3 -X importtime -m pip -v and attached that asout_1.txt

• python3 -X importtime -m pip -v list and attached that as out_list.txt

• python3 -X importtime -m pip -v check and attached that as out_check.txt

It appears the culprit for the list command is keyring.core?
import time: 96023197 | 96029594 | keyring.core

Hopefully that is helpful :)

It appears the culprit for the list command is keyring.core?

Combined with the weird thing around having an X server helping, I wonder if keyring depends on a GUI, and there's some code in there that tries to find X and causes a delay until it concludes there isn't one?

@pfmoore
Based on their page at pypi it seems it requires either D-Bus or an X11 server?

I tried python3 -m keyring --disable and export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring but neither seemed to fix the issue.

jaraco/keyring#434 seems related, but the advice there on disabling points to the docs which states what you've already tried.

Experiencing this issue as well, believe it started after running a routine sudo apt-get update && sudo apt-get upgrade yesterday. It is definitely related to keyring and the display somehow. In addition to the fix of having an X-Server running, I was able to fix the issue by removing the line I had in my .bashrc file pointing the Display to WSL2's IP Address. The line in question is:
export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0

After removing this line from .bashrc and restarting WSL2, pip behaves as expected. Not quite sure if this points anyone in the right direction, as I'm not sure how to proceed from here.

It appears as if I was able to permanently fix the issue by updating keyring with pip3 install -U keyring and ensuring that
[backend]
default-keyring=keyring.backends.null.Keyring
is set in the keyring config file at ~/.config/python_keyring/keyringrc.cfg

Wonderful, cjpellicci's suggestion of pip3 install -U keyring did the trick.
I also had to move ~/.local/share/python_keyring/keyringrc.cfg to ~/.config/share/python_keyring/keyringrc.cfg.
Running pip3 list takes 1/2 a second now instead of 90 seconds.
This is without any X server running.

The above doesn't seem to work for me. There's no keyringrc.cfg in either ~/.local/share/python_keyring/ or ~/.config/share/python_keyring/.

Could this be a different behaviour between WSL Ubuntu and Ubuntu?

@peidaqi, after updating keyring you may want to try creating the config file with that exact name in the ~/.config/python_keyring/ directory, and then adding the following into the config file:
[backend]
default-keyring=keyring.backends.null.Keyring

I believe that keyringrc.cfg may be made after following the steps above to disable keyring by setting an environment variable, but I could be wrong.

BTW: I am running Ubuntu on WSL2 as well.

Hello! I was facing more or less the same issue with pip list running slow (~1 min) including any pip install command I was trying to run (on wsl2). However, I'm not sure if the behavior was exactly the same (pip list _did_ output the packages but the command would return after ~1 min).

What finally solved my issue was this: https://askubuntu.com/a/38468/938540 - I'm not sure the issues are related, but the symptoms were very similar. Hope this helps!

Adding another possible data point:
pip list took around 90 seconds under WSL2.
I was setting the DISPLAY environment variable to an X server running under the windows desktop. Clearing the DISPLAY environment variable or launching my X server changed the time to 0.343s.

It appears as if I was able to permanently fix the issue by updating keyring with pip3 install -U keyring and ensuring that
[backend]
default-keyring=keyring.backends.null.Keyring
is set in the keyring config file at ~/.config/python_keyring/keyringrc.cfg

That works for me.

Ubuntu 18.04
Python 3.6.9
pip 20.0.2

It appears as if I was able to permanently fix the issue by updating keyring with pip3 install -U keyring and ensuring that
[backend]
default-keyring=keyring.backends.null.Keyring
is set in the keyring config file at ~/.config/python_keyring/keyringrc.cfg

That works for me.

Ubuntu 18.04
Python 3.6.9
pip 20.0.2

Confirming this works for me too.

Ubuntu 18.04
Python 3.6.8
pip 20.2.3

Hi folks -- we don't need more reports confirming that disabling keyring would give users a speedup. We'd appreciate if someone would come forward to help with https://github.com/pypa/pip/issues/8719. :)

This occurs for me when running in wayland on Fedora 33! Hoping this is a useful addition even as a me too comment, since all others seem to be on WSL.

Environment

• pip 20.2.2 from /usr/lib/python3.9/site-packages/pip (python 3.9)
• Python 3.9.0
• OS: Fedora 33
• desktop environment: swaywm (wayland tiling wm), started via gdm

timed execution of pip and pip list:

pip  0.11s user 0.01s system 99% cpu 0.122 total
pip list  0.24s user 0.03s system 1% cpu 25.285 total

$ python -m pip uninstall jrnl
^CTraceback (most recent call last):
  File "/usr/lib64/python3.9/site-packages/dbus/bus.py", line 177, in activate_name_owner
    return self.get_name_owner(bus_name)
  File "/usr/lib64/python3.9/site-packages/dbus/bus.py", line 361, in get_name_owner
    return self.call_blocking(BUS_DAEMON_NAME, BUS_DAEMON_PATH,
  File "/usr/lib64/python3.9/site-packages/dbus/connection.py", line 652, in call_blocking
    reply_message = self.send_message_with_reply_and_block(
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NameHasNoOwner: The name does not have an owner

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.9/site-packages/pip/__main__.py", line 26, in <module>
    sys.exit(_main())
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 73, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
    module = importlib.import_module(module_path)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/uninstall.py", line 6, in <module>
    from pip._internal.cli.req_command import SessionCommandMixin
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 20, in <module>
    from pip._internal.network.session import PipSession
  File "/usr/lib/python3.9/site-packages/pip/_internal/network/session.py", line 26, in <module>
    from pip._internal.network.auth import MultiDomainBasicAuth
  File "/usr/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 34, in <module>
    import keyring  # noqa
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/__init__.py", line 1, in <module>
    from .core import (
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/core.py", line 186, in <module>
    init_backend()
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/core.py", line 90, in init_backend
    filter(limit, backend.get_all_keyring()),
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/util/__init__.py", line 22, in wrapper
    func.always_returns = func(*args, **kwargs)
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/backend.py", line 214, in get_all_keyring
    return list(rings)
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/util/__init__.py", line 33, in suppress_exceptions
    for callable in callables:
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/util/properties.py", line 26, in __get__
    return self.fget.__get__(None, owner)()
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/backend.py", line 68, in viable
    cls.priority
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/util/properties.py", line 26, in __get__
    return self.fget.__get__(None, owner)()
  File "/home/daboross/.local/lib/python3.9/site-packages/keyring/backends/kwallet.py", line 50, in priority
    bus.get_object(cls.bus_name, cls.object_path)
  File "/usr/lib64/python3.9/site-packages/dbus/bus.py", line 241, in get_object
    return self.ProxyObjectClass(self, bus_name, object_path,
  File "/usr/lib64/python3.9/site-packages/dbus/proxies.py", line 250, in __init__
    self._named_service = conn.activate_name_owner(bus_name)
  File "/usr/lib64/python3.9/site-packages/dbus/bus.py", line 182, in activate_name_owner
    self.start_service_by_name(bus_name)
  File "/usr/lib64/python3.9/site-packages/dbus/bus.py", line 277, in start_service_by_name
    return (True, self.call_blocking(BUS_DAEMON_NAME, BUS_DAEMON_PATH,
  File "/usr/lib64/python3.9/site-packages/dbus/connection.py", line 652, in call_blocking
    reply_message = self.send_message_with_reply_and_block(
  File "/usr/lib64/python3.9/site-packages/dbus/exceptions.py", line 47, in __init__
    def __init__(self, *args, **kwargs):
KeyboardInterrupt

Thanks for the reports here. It's clear now that this happens due to pip's keyring integration. #8687 would make pip significantly less likely to lookup stuff in the keyring and #8719 would make that an opt-in.

For people's information, importing the keyring module into anything can cause long stalls in the right environment, whether or not code ever explicitly calls keyring functions. This can be seen by timing an import: time python3 -c "import keyring". For me this takes 25 seconds or so on a Fedora 32 machine that I am remotely logged in to and that doesn't have a graphical login session.

For me, the direct cause of this is that keyring runs code on import that eventually attempts to make a DBus connection to org.kde.kwalletd5 and this fails very slowly. You can duplicate the fundamental code (and reproduce the stall) with:

>>> import dbus
>>> from dbus.mainloop.glib import DBusGMainLoop
>>> bus = dbus.SessionBus(mainloop=DBusGMainLoop())
>>> bus.get_object('org.kde.kwalletd5', '/modules/kwalletd5')

In keyring itself, this code is in keyring/backends/kwallet.py, in the priority() method. If kwalletd isn't running already and can't be started, this appears to require a long timeout inside DBus itself or the Python DBus libraries.

This implies that the entire import of the keyring module must be conditional and gated on any command line flag, not just use of it.

For people's information, importing the keyring module into anything can cause long stalls in the right environment

Thanks for this information. It sounds like a major issue with the keyring module itself - imports are meant to be cheap. Has it been raised as a bug there? I think our plan for pip is sufficient to mitigate the worst of this behaviour, but ultimately I think that fixing this is down to the maintainers of keyring.

If someone can link to a bug report against keyring, that would be great so we can monitor what they do and provide advice to pip users getting hit by this.

FWIW, a workaround for users hitting this is disabling keyring, as documented here: https://github.com/jaraco/keyring#disabling-keyring

Has it been raised as a bug there? I

Yes: https://github.com/jaraco/keyring/issues/403