Pip: Default to --user

What does this mean for the pypa install instructions?

currently the mention of root or Administrator access is in a footnote, so the instructions would fundamentally stay the same (assuming get-pip automatically defaulted to --user as well). the change would be to explain that they will end up with installs that only apply to their current user.

regardless of the --user change, the current instructions need to explain that some distributions like debian, are disabling ensurepip, so they shouldn't go looking for it as a way to get pip.

Long term I'm pretty sure they are going to keep it enabled, they are just figuring out how to do it.

although this issue is described as being about where pip should manage packages by default, almost more important (to me at least) is that this is indirectly about where get-pip will place pip by default.

How does this interact with altinstall'd Pythons?
Specially such as are installed with tools like https://github.com/yyuu/pyenv

the user scheme is .local/lib/pythonX.Y/site-packages, so if you're managing multiple pythons with the same major/minor version, you could end up with a mess I guess.

for tools like that, that manage real pythons, pinning pip to global mode would make sense for those.

A relevant issue on the redhat/fedora bug tracker https://bugzilla.redhat.com/show_bug.cgi?id=662034#c10

It looks like for Fedora/RedHat PATH=$PATH:$HOME/.local/bin:$HOME/bin is already in their /etc/skel/.bash_profile. This would make stuff installed with --user still show up by default, at least for bash users (default shell). Maybe other distros already have this?

There's nothing specially different about Windows that I know of. But we've only just got the standard Python directory onto people's PATH, I don't expect that getting a per-user scripts directory on is going to be particularly easy - and there may be technical difficulties as well, I'm not sure putting a user-level environment variable like %LOCALAPPDATA% into the system-level %PATH% even works :-(

I'm against rushing this. I would prefer waiting till we've had more experience with user installs and have managed to iron out the issues first. I know that's a chicken and egg problem, but I don't see rushing into a change as helping.

Also, with my Windows hat on, I'd have to say that making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification...

I'm assuming that this would only apply when running pip from a system Python.

Well the different thing on Windows is there isn't a system provided Python like there is on *nix that also comes with system provided Python packages :)

I'm not particularly wanting to rush this either. I just wanted to start the discussion.

This doesn't really affect people who type sudo pip install as I think installing something as root should still go to the system site packages by default. This would be to guide people towards using --user when running as an unprivileged user. Right now what you get is that pip install as a regular user just bombs out with a permission error. Even if we improve that message to bomb out and suggest --user that's still not the greatest UX.

In talking to one of the Fedora people, I think the way it may make to do this is to implement a permissions check. If I have permission to write to the site-packages directory then I am a privledged user and pip installs to the global Python, If I do not then I am an unprivileged user and it installs to --user.

As far as Windows goes, we usually duck the problem because we install Python to a non-privileged directory by default, so pip doesn't trigger UAC when installing globally. If someone changes that to install into Program Files instead, then UAC will trigger when they run pip (which is also OK). I'm not sure what happens if they select a per-user install in the installer.

As Paul notes, PATH on Windows doesn't include the _user_ scripts directory yet (just the global one), so that's definitely worth taking into account.

I don't see any major barriers on the POSIX side though. How does this UX idea sound: for wheel based installs, check for permissions on the installation target directory, and if the user doesn't have write permissions, put up a prompt asking if they would like to do a --user install instead? A new "--global" or "--system" flag would force the "no" answer.

And then at some point in the future, we could skip the prompt and simply assume --user if the permissions weren't right for a global install.

A prompt (except in the case of --no-input) probably makes sense for the transition step. I think checking permissions should make things work for Windows too since by default the location is in a spot where people have permissions on and so the only time they'd hit this would be if they are using a systems admin provided Python that explicitly didn't give them permission.

Essentially this makes our failure mode much better.

making it harder for Unix users who haven't yet worked out that careless use of sudo
is bad to do stupid things, isn't really that good a justification..

sudo and security is really a secondary issue IMO.
this is primarily about preventing the conflict between OS packaging and pip in the system python.
Currently, linux users are often placed in an impossible situation.

1. OS Mandate: 'Use the OS pkg mgr; Don't infect your system with roque pip-installed packages (including a get-pip'd pip)'
2. Reality: "The OS packages (including pip) are too old, and I can't upgrade to the experimental release of the distro, just to get upgrades of some package. I'm going rogue...."

another baby step is to document (assuming we test it) that you can get-pip.py --user, and have the PUG mention --user as a possibility for the install of virtualenv (and wheel, and twine too, if not in a virtualenv)

FWIW, distro vendors also consider the status quo a problem and are looking at various ways to improve the tools for doing selective upgrades without impacting core OS components. Still worth us tackling from the upstream side though, since those efforts are at various stages of maturity and we want a consistent cross-platform solution for end users.

I have no problem with changing things to help co-operate with distros on Unix; if defaulting to --user on Unix and not on Windows is acceptable, that's fine with me. I just don't see switching from a bad experience on Unix to a bad experience on Windows as a good trade-off. And I'm yet to be convinced that --user on Windows is ready for prime time.

Note that triggering UAC when running pip _is_ bad on Windows, because what it actually means is that pip won't run except in an elevated console window. It does not mean that pip users get a nice prompt which they can say "OK" to, unfortunately, that's only how GUI apps work...

@pfmoore What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

FWIW I'm totally OK with making this optional depending on Windows or not. I'm just wondering if the check of "Do I have permissions to write to the site-packages folder" check won't achieve that anyways in 99% of installs, and if installing to --user would be a better experience in the fraction of cases where you don't have write permissions to the site-packages directory.

In other words, if we check permissions the proposal isn't replacing global install with a user install, it's replacing a permission error with a --user install.

the proposal isn't replacing global install with a user install
it's replacing a permission error with a --user install

ok, but recall that the permissions check idea (#1048) won't the be the easiest thing

It'll be pretty easy in the Wheel case, and we can get the 99.9% case covered with sdists, it should only be a problem in setup.py's that do really funky things.

What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

Well, being able to write to the site-packages directory is so rare on Windows that I'm not sure how relevant the question is in practice. But if it happened, I would say that an error saying "system site-packages is not writeable - did you mean to use --user?" would be the sensible approach.

Actually, I think that would be better on Unix as well. Switching the behaviour depending on writeability is bizarre, and it's not clear to me if using --user for a personally-built Python in a writeable directory is the right approach (but I have precisely zero experience of such a setup, so I don't have anything to back that opinion up).

Suggestion: Start with an error suggesting --user as above. Once people are using --user more commonly, and we have more experience with it, then let's consider switching the default.

The approach Paul suggests is roughly what I had in mind, but with a yes/no prompt rather than bailing out with an error message. There are pros & cons to those two approaches (mostly relating to how they fail when invoked from another script).

Would it be possible to add to Python something like sys.localprefix (I see python-dev here)?
Sys.localprefix would be default to sys.prefix. Distros could define it same way as they define prefix (e.g. prefix = '/usr', localprefix='/usr/local'). Pip and easy_install would use sys.localprefix as a location to install to if used as sudo, if they don't have sufficient rights fall back to --user.

We've done something like this with "gem install" in Fedora 17 and Ruby devels were generally very satisfied about it, so I thought I'd share:

• "gem install foo" installs "foo" into ~/somedir if uid != 0
• "gem install foo" installs "foo" into /usr/local/somedir if uid == 0
• "yum install rubygem-foo" installs RPM-packaged "foo" into /usr/somedir

I think that choosing install dir based on user privileges would be very confusing for people; "pip install foo" should work the same way for all non-root users. If you consider majority of users, they'll want to "pip install" into their home and "sudo pip install" into a system-wide dir. Therefore I think it's the best approach to do this based on uid as shown above - and for those users who are not root but want to install into a system-wide dir, there is always the --target option.

@bkabrda that's an easy choice to make as long as you're not worried about adding scripts to $PATH. What have you done there?

@Ivoz as noted by @dstufft, Fedora already has $HOME/.local/bin on PATH, so everything worked out of the box (speaking of scripts).

Slavek's suggestion sounds good to me for POSIX systems. It should also work nicely with containers, since stuff running in a container can be told to think it is uid 0, even though it's something else entirely from outside the container.

For Windows, I'm less sure what the right answer is. We don't have quite the same problem with getting into an argument with the system package manager, although it does exist to some degree (pip install vs MSI installers), and have the additional complication that even in Python 3.4, the installer's optional PATH modifications only add the global Scripts directory, not the per-user one.

Windows also has weirdness based on whether Python is installed to the default location (uncontrolled) or into Program Files (UAC privilege escalation needed to make changes).

The "localprefix" idea suggested by @rkuska looks nice to me, simple and easy. Also I really like the approach @bkabrda suggests, especially for distros like Arch that make python 3.x the default python.

One more note is, Arch has disabled ensurepip in the 3.4.0 release just like Debian (mostly because we want to provide latest versions of setuptools and pip, while the bundled versions may be outdated for ages), so it would be nice to have a tip for it.

@lvoz
For Arch we didn't add any path under $HOME to PATH, but I still think it fine. We already have wiki entry for Ruby that tells the user to add it.

It would be nice if rather than just disabling it, the Arch and Debian developers would help Slavek work on his patch to have ensurepip reconstruct a wheel from the system versions and install that into virtual environments.

Just FYI, we already have working Python 3.4 RPM builds in Fedora's Copr build system (testing, not advisable to install them if you don't want to break anything). If you want to know more about how we approach this, have a look at the code etc, see my discussion with Barry Warsaw [2] at debian-python ML.
(Our patches still need some love before we propose them upstream, but everything works ok for us downstream ATM)

[1] http://copr-fe.cloud.fedoraproject.org/coprs/bkabrda/python-3.4/
[2] https://lists.debian.org/debian-python/2014/03/msg00046.html

Sorry, but AFAIK Arch never supported wheel, nor do we want to make ensurepip invoke package manager (correct me if I understand this approach wrong, though).

For ensurepip, it would be a nice idea to warn the user not to install pip using ensurepip though, and that's the most I can think of for now.

Our ensurepip will never invoke package manager. In short, Fedora's ensurepip will rely on system packaged setuptools and pip to be always present (our python3 package will depend on them, which will create a dependency loop, but we can handle that) and when user will create new virtualenv, it will repack system setuptools and pip into wheels and install them into the new virtualenv.
We're starting to get a bit off-topic, so I guess we should continue this discussion someplace else...

@felixonmars Does that mean on Arch venv is broken the same as it is on Debian?

OK, thanks for the explanation, I think I got the idea. I'll see what I can do and follow up on debian-python for this.

@dstufft Yes, the bundled version got installed instead of the system version.

@bkabrda So that solves the problem if you're invoking pip with a system Python, we have to special case virtualenv/venv in that case (which isn't the end of the world). But it also means that we're assuming all Pythons are system Pythons, even ones installed by the user into their homedir such as with https://github.com/yyuu/pyenv (which is how I run Python actually). So I'm not sure, that's why I thought of the permission check.

ditto what @dstufft said, in using a tool like pyenv to juggle non-root pythons, it would be pretty odd for root to be the key to do a global install (where "global" != "system")

We've done something like this with "gem install" in Fedora 17

you mean the rpm of gem for fedora 17 has this as a distro hack?, or gem itself has this logic?

one thought is that pip should provide a "system friendly mode" that can be turned on by distros that are packaging pip, or when users know they are installing pip themselves into a system-managed python. otherwise, pip does it's normal global-by-default logic.

Ah @felixonmars I just saw https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python , if all you're doing is calling --without-ensurepip that's fine. That leaves ensurepip module intact for the venv module. The patches Ubuntu (Debian?) currently has in place actually calls rm -rf on the ensurepip package itself, making it unavailable to be called within a venv.

I think defaulting to user installs was proposed by Nick two years ago, or maybe someone else longer ago, but the thread went into many directions and was not fruitful. Main obstacle was backward compat IIRC, but these days sysadmins and pip users may be more used to be careful with virtualenv/pip/setuptools updates, for better or worse. I don’t remember if that discussion happened on distutils-sig or pypa-dev.

it was catalog-sig: https://mail.python.org/pipermail/catalog-sig/2013-February/004773.html

@dstufft Oh, I see... But I still feel wrong to have a bundled (maybe old) version installed while there are packages installed system-wise :)

@felixonmars well the problem is you can't install a system package into a virtualenv afaik :) (And even if you could there are different concerns with what you want in a system package vs inside a virtualenv)

@dstufft That's where Slavek's rewheel kicks in, but that's a bit off-topic :)

Yea, check the last bit of https://lists.fedoraproject.org/pipermail/python-devel/2014-March/000583.html :(

@dstufft Thanks!

Not sure if I should continue the discussion there, because that shouldn't be a problem since we didn't unbundle pip in Arch, and rewheel should generate usable wheel :)

@dstufft you're right that we would probably need to specialcase virtualenv and possibly even pyenv, which seems ugly. I have to admit I'm not that familiar with pyenv, so I can't think of any "proper" solution for that right now. I'll try to have a closer look at it and think of something.

@qwcode for "gem install" on Fedora, it's a downstream patch. TBH I didn't touch Fedora's Rubygems for long, but the last time I checked, it was downstream only.

I have posted link to this issue to related bug in python [1]. I would like to move this discussion to python core, to have a configurable location for local installs of not only pip but also easy_install and python setup.py install, after that pip could use (e.g.) sys.localprefix dir if uid==0 and install to user dir if uid!=0 as @bkabrda mentioned.

For the time being I am fine with pip installing to user dir by default.

[1] http://bugs.python.org/issue1298835

As Nick pointed out I moved the discussion from issue1298835 to pypa-dev [1].
[1] https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0

Default to --user

Good idea!

pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python

It's worse than that. Many people (such as users of a university computer network) don't have sudo privileges. When pip fails with a permission error, they will either:

1. Email their administrator and ask them to install the package. That's slow and tedious for everyone.
2. Give up and do without the package.

Only if they are particularly experienced or well-informed will they try pip install --user.

If we end up deciding against 'default to --user', we should consider the less disruptive alternatives:

1. Fallback to --user (if install fails due to perimissions errors)
2. (If install fails due to perimissions errors), encourage the user (in large friendly letters) to try --user

I actually have been using --user in all my installations for a while now, but generally people are not aware of this option. I agree it's annoying to see the permissions error every single time you forget the option. I noticed user installations are not compatible with conda though (conda/conda#448).

I'd like to revive this, as people are going to run into it on Windows far more often now that 3.5 defaults to installing into Program Files (which requires administrator privileges to modify).

I like the "fallback to --user on permission error" option best for installs, with "default to --user and fallback to system" on uninstalls and maybe add a --system or --system-only option to prevent the fallback. Users deliberately running pip with admin permissions will get a system install as expected, and currently users without permissions would get an error, so I don't believe there's any back-compat issues here.

My aim is for an uninformed user to be able to do pip install spam; python -c "import spam" without errors (though possibly a warning when retrying with --user).

I've been thinking about this lately especially since #2403 and #2401 were opened recently that made me think about the use of sudo pip install and how that can often break systems outside of just the permissions problems.

To this end I wonder if the better end goal isn't just that --user is the default, no magical permissions checks, and add a new flag like --global, --system, or --no-user which will restore the previous behavior. This is easier to explain to people since it'll have the same behavior no matter how/where things are installed.

Now, if we decide that's the way we want to go, there is still the problem of how do we get to that point from where we are today. Obviously at whatever point we switch the default is going to be a fairly big breaking change so we're going to need a fairly long lead time. I think that if we decide to go that way the best way to do it would be to continue to keep the --user option around, add a new option like --global, and then if one or the other option isn't configured do the permission check magic that we've talked about and if the magic has decided to use the --global behavior we'll raise a very loud deprecation warning. This deprecation warning would stick around for a lot longer than our other deprecations that we've issued since this would be a fairly major change in behavior. I think that we would also want to add an option which would turn off the magic fallback and implement the future behavior so that people can get that behavior today.

Another important question is what should we do if we detect that the Python we're installing into does not have user sites enabled. In this case I would simply state that --user is an error and for those Pythons we'll simply fallback to defaulting to --global. This would cover the virtualenv/venv I believe automatically, and if someone wants a "heavier" weight isolated environment using fully compiled Pythons they can simply patch site.py so that they set ENABLE_USER_SITE to False at the top of the file.

+1, and I'd vote for --system.

I'll note, and I'm sure you'll love this, the Ubuntu default has changed:

https://launchpad.net/ubuntu/+source/python-pip/1.5.6-4ubuntu1

On Feb 09, 2015, at 04:41 PM, Donald Stufft wrote:

To this end I wonder if the better end goal isn't just that --user is the
default, no magical permissions checks, and add a new flag like --global,
--system, or --no-user which will restore the previous behavior. This
is easier to explain to people since it'll have the same behavior no matter
how/where things are installed.

I think that --system is wrong because that really only applies to some Pythons. It doesn't apply to:

• Python on Windows
• pyenv installed Pythons
• Conda installed Pythons
• Nix Installed Pythons
• Custom Compiled Pythons.

It _only_ applies on *nix systems where the Python happens to be shipped by the system. I feel like --global is a much better name for that flag.

And FML, why does Debian/Ubuntu feel the need to constantly add random patches that break expected use cases?

So if the Ubuntu patch gets reverted, what's the timeline on this getting fixed here? A week/a month/3 months/6 months/a year...?

It depends on what you mean by fixed.

Assuming for a moment we're using the idea I had 2 posts ago (which still needs sign off from the other pip developers) we could likely pretty quickly get the transition plan where the following holds true:

• When --user is passed we _always_ install into the user site packages.
• When --global (or whatever) is passed we _always_ install into the global site packages.
• When neither is passed we implement a fallback method which:
  
  
  
  • Detects if we're in a virtual environment and if so acts as if --global was passed.
  
  
  • Detects if user site-packages is disabled and if so acts as if --global was passed.
  
  
  • Detects if the effective user does not have write permissions to sys.path, and if they do not act as if --user was passed.
  
  
  • Finally, if all other detection methods failed, act as if --global was passed and print a deprecation warning saying that at some point in the future the behavior of this will change and it will instead install into --user.
  
  

This won't stop someone from doing sudo pip install foo into their system Python, however it will print a warning that at some point this is going to mean --user and tell them to explicitly start using --global. Once the above has been released for a long enough time we could then remove the deprecated code path and simplify this so that it's simply:

• When --user is passed we _always_ install into the user site packages.
• When --global (or whatever) is passed we _always_ install into the global site packages.
• When neither option is passed in we implement a fallback method which:
  
  
  
  • Detects if we're in a virtual environment and if so acts as if --global was passed.
  
  
  • Detects if user site-packages is disabled and if so acts as if --global was passed.
  
  
  • Finally uses --user as the default.
  
  

However, moving from the first behavior to the second behavior is going to have a fairly long timeline. That is going to be a really major breaking change which is going to break things for any software (such as salt, chef, puppet) or deployment script (like custom one off Fabric scripts baked into thousands of projects across the world) which expects that sudo pip install <foo> is going to install into the global Python. We'll need to give a long lead time for people to see the warning, and modify their software and scripts to pass an explicit --global flag if they actually need that behavior.

I think Donald's suggested plan is a good one, although if the pip team decides to adopt that approach, we should also file a 3.5 release blocker issue with CPython to get the per-user scripts directory added to the path on Windows along with the global one. If 3.5 adds that along with the switch to installing into Program Files by default, then we should get the desired behaviour of installing without admin permissions continuing to "just work".

Being able to change the default install location via the pip config files may also be useful, allowing folks to opt in to "per-user by default" on their own systems, even if the default hasn't changed yet. (e.g. "default-install=user" vs "default-install=global")

@dstufft: I'm happy to give a hand to implement your proposal as told on the ubuntu bug if you need me. Once we get that behavior in trunk, I'll backport and replace the current ubuntu patch with this of course.

Keep me posted if you need any help.

I'm +1 on @dstufft's plan. I'd go for --global as the name, for the reasons stated. We've been sitting on the fence long enough now, let's go for it.

O'n and +1 on @ncoghlan's suggestion that Python 3.5 add the user site directory to PATH on Windows. Let's avoid any further stumbling blocks for this transition.

I'm also +1 for @dstufft's plan and +1 for --global, although I'm still wondering about the sudo pip install foo needing --global. Is there in fact any need for root to have user site packages? Could we special case root to have site packages off by default? (Which would make sudo pip install foo work just as it does now)

I have never seen anyone using --user for root with the current state, but if someone actually knows such a use case, I'd like to know.

One thought that occurred to me is how will the user site directory work with multiple versions of Python installed?

If I do pip install pytest --user in Python 2.7 and 3.4 (with Python 3.4 my default Python), which version will the py.test command execute? I have a suspicion that the two installs will overwrite each other so it's a "last one wins" situation, which is _not_ good. Also, if it is "last one wins", and I install the 2.7 version after the 3.4 one, how would I fix that to make 3.4 the default again?

And what about uninstalls? Consider the following scenario (untested, because I'd need to set up a clean machine if I didn't want to risk breaking my main laptop):

    C:\Apps\Python34\python.exe -m pip install pytest --user
    C:\Apps\Python27\python.exe -m pip install pytest --user
    C:\Apps\Python34\python.exe -m pip uninstall -y pytest

First, why didn't the second command issue a warning that it was overwriting an existing file? Or would it? Whether it did or not, do I have any options on what to do?

Second, would the uninstall not fail because the checksum of py.test.exe wouldn't match the value in the RECORD file? If it does fail, how would I uninstall? If it doesn't fail, will it break my Python 2.7 copy of pytest (by deleting the exe)?

And finally (for now!) would the user scripts directory go before or after the system scripts directory in PATH? IMO, it should go after, as otherwise a user install of a package in a Python that was _not_ requested by the user to be "the default Python" could override the same package installed in the "default" Python's system site-packages.

Hmm, maybe there are some open questions that need resolving before we make this change...

I'd expect the name clashes in the shared Windows user script directory to be handled the same way they're handled on POSIX systems with their shared bin directories: Python 3 wouldn't install the unqualified script name for installed packages, so the unqualified name would refer to the Python 2 version.

Same goes for whether the system install or the per-user install takes precedence by default: put the user script directory after the system one.

"Python 3 wouldn't install the unqualified script name for installed packages" - presumably you mean that pip and setuptools (under Python 3) wouldn't, here? So that's a change to those tools?

Oh, and bleh. My muscle memory is 100% trained to type "pip install foo" to install into my default Python (3.4). That's going to be hell to retrain.

Also, Windows users get to choose whether "python" means Python 2 or 3 (via "make this the default Python" and "Add to PATH") - unlike Unix users where the system Python makes the rules. So if I say I want to make Python 3.5 my default, why shouldn't "pip" (or any other unqualified name) refer to that version? It seems bizarre that a user with only Python 3.5 installed on Windows can't use unqualified names simply because of a Unix issue around how the system Python needs things named. (Apologies if my description of the Unix constraints is inaccurate - I don't really understand the issue there). People installing multiple Python versions need a solution, but that solution shouldn't affect the majority who only need one version.

Maybe the simplest answer is that the user scripts directory should be versioned, just like the user site packages directory is?

Oh, and should this debate be on python-dev rather than here, as it's more generally about the user site directory?

Reopening the PEP 370 design discussion for the Windows per-user scripts directory on python-dev would certainly be reasonable. It's odd that global script installations on Windows are scoped by Python version, but per-user scripts installations are not.

That contrasts with the situation on POSIX, which consistently uses a shared namespace for executables at both levels (global or per-user), so name conflicts are more likely to show up at installation time, rather than through name shadowing at runtime.

As far as the general solution to handling parallel installation goes, that would be the "-m" switch - that way you always know which Python you're invoking, rather than guessing about the contents of a separate script's shebang line.

Understood re -m, but last time that was brought up in the context of how to document things, the strong preference was that we should treat pip install foo as the canonical way to install things (with -m and versioned commands being treated as the exceptions for specialist cases.

The pip manual doesn't even _mention_ alternatives (except in the one case of using -m to upgrade pip itself on Windows).

I recommend -m in the stdlib package docs for handling pip invocation when
dealing with parallel Python installs - it's the only invocation style that
works across all platforms, for global installs, user installs and virtual
environments.

On Feb 09, 2015, at 07:58 PM, ncoghlan wrote:

Being able to change the default install location via the pip config files
may also be useful, allowing folks to opt in to "per-user by default" on
their own systems, even if the default hasn't changed
yet. (e.g. "default-install=user" vs "default-install=global")

I like this. One of the things we're struggling with is how to resolve
upstream's more conservative approach to migrating to a --user default, with
distros competing goals of making pip system-safe and consistent with other
tools. E.g.

https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1419695
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725848

If we had a system-wide configuration file to select the default, then at
least all the machinery involved would be identical, and the distros wouldn't
have to carry ugly deltas. All we would have to do would be to change the
config file and document that to our users. Problems that caused would be
ours to handle, but at least it would be easy to explain.

I don't think Debuntu currently installs a site-wide pip.conf file (I'll have
to double check).

https://pip.pypa.io/en/latest/user_guide.html#configuration

One complication: we install different pip scripts for Python 2 and Python 3,
so I think we'd need separate config files for the different versions of
Python, e.g. /etc/{xdg/pip/}/pip{2,3}.conf or some such.

On Feb 10, 2015, at 01:24 AM, Paul Moore wrote:

One thought that occurred to me is how will the user site directory work with
multiple versions of Python installed?

On Debian/Ubuntu (maybe all *nixes?) we have no collision because the
libraries get installed into ~/.local/lib/pythonX.Y/site-packages

We _do_ get collisions on ~/.local/bin though. Installing a package with a
script 'foo' using pip install --user foo and pip3 install --user foo will
end up clobbering the ~/.local/bin/foo script.

One downside of this change is that I think Fedora is the only downstream distro that adds ~/.local/bin to $PATH by default (and it should come before /usr/bin and /usr/local/bin just like the user site packages comes before site-packages). Ideally downstreams will be able to modify their systems so that ~/.local/bin is on $PATH by default.

On Feb 10, 2015, at 07:24 AM, Donald Stufft wrote:

One downside of this change is that I think Fedora is the only downstream
distro that adds ~/.local/bin to $PATH by default (and it should come
before /usr/bin and /usr/local/bin just like the user site packages comes
before site-packages). Ideally downstreams will be able to modify their
systems so that ~/.local/bin is on $PATH by default.

Yep, but we can make a separate decision about that. (Or rather, downstreams
distros can do that.)

(and it should come before /usr/bin and /usr/local/bin just like the user
site packages comes before site-packages)

Hmm, @ncoghlan just said the opposite, that (on Windows, in that particular
context) the user scripts directory should be _after_ the system one. Which
I agree with as long as the user scripts directory is unversioned. If it's
versioned, I'm less concerned. But that probably reflects the fact that
Windows users have never needed to endure unversioned scriipts clashing and
monstrosities like pip2/pip3, the way Unix users have ;-)

Oh, and the script clobbering thing is a more general problem that isn't really related to --user exactly. The same problem exists everywhere except Windows --global installs. I think the solution to that problem is a secondary concern/problem that we should solve separately to this issue.

So current thoughts:

• I think there is agreement on the general idea I posted earlier, so I think we'll want to move forward with something like that.
• We're going to use the --global flag.
• We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH.
• We'll want some sort of default-install option that'll essentially disable the fallback behavior and just hardcode to either --global or --user.
• As of pip 6.0 we have machine specific config files located in /etc/ however these are not specific to the version of Python that is executing pip. I think that's a good feature though somewhat separate as well (and might go hand in hand with solving the script clobbering thing as well).

I see one major open question still: Long term what do we want to happen when someone does sudo pip install foo or rather, if someone has permission to write to the site-packages directory?

The options I can think of are:

1. Install into --global. This is the most backwards compatible option and likely represents what users expect when they type that. However it has the downside that it'll (silently) mess with the global Python, which on many systems can cause broken systems.
2. Install into --user, this will protect from broken systems (I think?) but I don't think anyone would expect the root user to get a /root/.local/ install.
3. Just error out and require the user to select either --user or --global manually.
4. Pick one of the first two options, but provide a switch that distros can turn in in /etc/ that will enable the third option.

The fourth option might be the best one but I feel like we should try to select one of the other options first if we can come to some sort of solution that works the same on all platforms/scenarios and makes sense on all of them as I would prefer to have consistent behavior cross platform if possible.

I feel like the third option is likely a bad one, since "I have write permission" is going to be the scenario that anyone on Windows is on pre Python 3.5, or anyone who uses Conda, or anyone who uses pyenv. Erroring out seems like the wrong thing to do in all of those cases and is soemwhat user unfriendly to boot.

So I guess between 1 and 2 it really comes down to how bad of a practice do we think it is for people to (conceptually) do sudo pip install --global foo. On Windows, Conda, pyenv, etc I feel like the answer to that is "we don't think it's a bad practice at all". On *nix I feel like maybe the answer is "users should be allowed to do what they want to their system but we should provide rails to lead them away from "bad" things".

So thinking about it more, I think I'd have to go with the first option being the correct option for us to go with. That solves the backwards compatibility issues more or less since the behavior we'd really be changing is that if you type pip install foo without permission to install into that global directories you'd install into --user instead of raising an error. I think that it's unlikely that anyone is out there really depending on pip raising an error in that case. I also think that pip install --user as the root user is unlikely to be what anyone expects or want and it's really just making a footgun for users.

So then my updated proposal would be:

• When --user is passed we always install into the user site packages.
• When --global is passed we always install into the global site packages.
• When neither is passed we implement a fallback method which:
  
  
  
  1. Detects if we're in a virtual environment and if so acts as if --global was passed.
  
  
  2. Detects if user site-packages is disabled and if so acts as if --global was passed.
  
  
  3. Looks at default-install and if that exists uses --user or -global based on it.
  
  
  4. Detects if the effective user does not have write permissions to global site packages, and if they do not act as if --user was passed.
  
  
  5. Finally, if all other detection methods failed, act as if --global was passed.
  
  

This would mean no deprecation period and that the expectation is that in the absence of --user or --global we'll try to make the best guess based on the capabilities of the Python we're running in, the effective user, and any configured default install method.

(and it should come before /usr/bin and /usr/local/bin just like the user
site packages comes before site-packages)

Hmm, @ncoghlan just said the opposite, that (on Windows, in that particular
context) the user scripts directory should be _after_ the system one. Which
I agree with as long as the user scripts directory is unversioned. If it's
versioned, I'm less concerned. But that probably reflects the fact that
Windows users have never needed to endure unversioned scriipts clashing and
monstrosities like pip2/pip3, the way Unix users have ;-)

I'm not even thinking in terms of the script names clashing. I just think it's flat out wrong for our $PATH variable to act differently than sys.path does. As far as I know sys.path's ordering is: stdlib > user site packages > global site packages. I think it would really confusing if some-script's ordering was global bin dir > user bin dir while python -m some-script was user site packages > global site packages.

If there's a problem with the script directories (and versioned vs unversioned) then I think we should fix that problem but that it's not super relevant to what order they should be in, because I feel like anything other than matching what sys.path does is super wrong.

Good point, we should assume (and recommend) that $PATH follows sys.path, with (in Windows terms) C:PythonXY before %APPDATA%PythonXYScripts before C:PythonXYScripts. I'll note that on the python-dev thread.

Donald's latest proposal sounds good to me (same as my original one plus all the important details I left out :) ). I agree 100% with the "So thinking about it more..." paragraph, too.

As for configuring PATH on Windows, that's going to be nearly impossible. A system-wide installer can only configure system-wide environment variables and they can't include expansions, so you can't set different paths for each user. (The per-user installer can do it, but in that case --global will succeed and so there's no need.)

Also, because of Windows's broken PATH handling, system-wide settings _always_ beat per-user settings. This lead to my half-suggestion a while back (on python-dev, IIRC) about using the py.exe launcher for unversioned scripts, so that pip.exe would always go to the latest system or user Python rather than the one that installed it, and (e.g.) pip.exe -3.5 would allow specific version selection.

The only way to make this really work is for the Python installer to start installing batch files to configure PATH on request (and maybe a shortcut that runs the batch file). So the first thing users would type is activate-py35 and then their PATH is configured correctly for 3.5. I'm very torn on this though, as I don't want to get into the vcvarsall.bat situation, but at the same time it'll be great for scripts that currently make assumptions about install locations or try and go through the registry to find it.

and maybe a shortcut that runs the batch file

In my mind, this is a "Python 3.5 (32-bit) Command Prompt" shortcut. Visual Studio users should recognize the parallel.

I don't know enough about Windows to fully understand the implications of all of that, but I feel pretty bad about reverting to a situation where pip install foo which installs a script foo can't immediately be executed as foo on the command line. If we're installing to --user and that isn't on their path by default then that feels like a pretty major regression to me.

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice and whether or not it would actually solve the problem. I'm not personally tied to creating scripts exactly as we are today so if it _does_ work I think we can maybe do it but I'd have to better understand the implications before I could give my +1 on it.

As for configuring PATH on Windows, that's going to be nearly impossible. A system-wide installer can only configure system-wide environment variables and they can't include expansions, so you can't set different paths for each user.

Oh, yuck. I'd forgotten that.

and maybe a shortcut that runs the batch file
In my mind, this is a "Python 3.5 (32-bit) Command Prompt" shortcut. Visual Studio users should recognize the parallel.

... which is horrible for those of us who default to Powershell for everything.

but I feel pretty bad about reverting to a situation where pip install foo which installs a script foo can't immediately be executed as foo on the command line.

Agreed. This would be a bad regression, and needs to be addressed. I just wish someone could think of a way :-)

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice

I think the point (which is only somewhat related to py.exe) is that the exe wrapper, rather than invoking a specific, hard-coded Python interpreter as it does at present, should dynamically choose the interpreter based on "what is the default" somehow (registry, the py.exe ini file, whatever Python %PATH% gives you, ...). I'm not sure how it would help, though, as we'd still need a versioned user-scripts directory (to avoid files overwriting each other or files being run with interpreters that don't have the package installed) and we still wouldn't be able to put that at the right point in %PATH%.

... which is horrible for those of us who default to Powershell for everything.

I just tried it, and I can easily make activate-py35.bat and activate-py35.ps1 files that look and behave identically (i.e., everyone just gets to write activate-py35 and get the paths updated). Still not ideal, but possibly the best of a bad situation.

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice

I think the point (which is only somewhat related to py.exe) is that the exe wrapper, rather than invoking a specific, hard-coded Python interpreter as it does at present, should dynamically choose the interpreter based on "what is the default" somehow

The "somehow" is where py.exe comes in - the same rules should be used. If the launcher were updated to check its own name, then instead of launching python.exe it could try and launch 'scripts\\{}{}.{}.exe'.format(argv[0], major_version, minor_version) (with suitable extension trimming on argv[0], etc.) instead. Then installers need to use a different file for the unversioned launcher from the fully versioned one (which would be unchanged from today), but that file would just be the normal py.exe with a new name. (This could have been even simpler in the old days of pip-script.py files, but now that those are gone...)

But as you say, PATH is still the problem. I don't have any good solutions, and not even that many bad solutions. Environment variables are really meant for administrators or users to configure, not installers.

We can adjust how we install scripts to make a better user experience. It's all a balance, the combined .exe and .py file made things nicer for users since there's just the singular .exe file. However I feel like getting the best story for pip install foo && foo is more important than that particular thing, so if we need to make the .exe and the .py separate again I think that's a good tradeoff for a good answer.

The .exe/.py separation is the least of the problems :)

I'm actually warming up to (or grudgingly accepting?) the activate-py35 idea. I've never been a fan of the installer adding Python to PATH at all, and the py -m pip command hasn't really gained traction (and nor would it work for, say, Sphinx). I'll write up a more detailed description of how activate-py could work and post it to python-dev to see if it gets any traction.

Would these activate-whatever files be similar to the ones inside of virtual environment? Except instead of a virtual environment getting added to $PATH it's adding the real Pythons?

Yep, pretty much identical. On python-dev I've just suggested that they'd take a -x.y parameter, pass it to py.exe and use sysconfig to get the paths.

Ok. I don't have an opinion on how bad that would be for Windows users since I'm not one. It doesn't sound absolutely horrible (maybe?) but that's about as far as my Windows knowledge can take me.

On Feb 10, 2015, at 08:02 AM, Donald Stufft wrote:

Oh, and the script clobbering thing is a more general problem that isn't
really related to --user exactly. The same problem exists everywhere
except Windows --global installs. I think the solution to that problem is
a secondary concern/problem that we should solve separately to this issue.

Agreed.

We're going to use the --global flag.

+1

We likely want some sort of warning if we're installing into --user and
~/.local/bin isn't on the $PATH.

+1, perhaps with a config option to suppress the warning?

We'll want some sort of default-install option that'll essentially
disable the fallback behavior and just hardcode to either --global or
--user.

+1

As of pip 6.0 we have machine specific config files located in /etc/
however these are not specific to the version of Python that is executing
pip. I think that's a good feature though somewhat separate as well (and
might go hand in hand with solving the script clobbering thing as well).

I would suggest a search order such as:

• pipX.Y.conf
• pipX.conf
• pip.conf

rooted first(?) in /etc/xdg/pip then /etc, where X.Y of course is the
major.minor version number. First one found, wins.

I see one major open question still: Long term what do we want to happen when
someone does sudo pip install foo or rather, if someone has permission to
write to the site-packages directory?

Please note that there isn't just one "site-packages" directory. For example,
on Debian/Ubuntu we don't want sudo pip install to _ever_ install into
/usr/lib, but only /usr/local/lib. And don't forget, it's dist-packages here
not site-packages (except in ~/.local because $reasons).

This is really an important point that sometimes gets forgotten (even by me).
Donald describes it best when he says there is a site-local directory and a
vendor-local directory. On Debian, site-local is
/usr/lib/pythonX.Y/dist-packages and no pip command should ever touch that,
while vendor-local is /usr/local/lib/pythonX.Y/dist-packages and it is a
documented and popular use case for some system administrators to pip install
into that directory.

So what I would suggest, since we're already down the config file path, is to
make these configurable, e.g.

• global_install_directory
• global_install_as_sudo (true/false)

1. Install into --global. This is the most backwards compatible option
   and likely represents what users expect when they type that. However it has
   the downside that it'll (silently) mess with the global Python, which on many
   systems can cause broken systems.

With the above configurability, this would be totally compatible with Debian.
Debian superusers expect sudo pip install to go into
/usr/local/lib/pythonX.Y/dist-packages. I wouldn't say it "messes with the
system" because it doesn't break the package manager, and while it can
override system installed packages (which live in
/usr/lib/pythonX.Y/dist-packages), it does so in a defined, documented way.

1. Install into --user, this will protect from broken systems (I think?)
   but I don't think anyone would expect the root user to get a
   /root/.local/ install.

Agreed, that would be unexpected.

1. Just error out and require the user to select either --user or
   --global manually.

I'd be okay with that, but see the configuration switch above.

1. Pick one of the first two options, but provide a switch that distros can
   turn in in /etc/ that will enable the third option.

Hey, what a great option!

The fourth option might be the best one but I feel like we should try to
select one of the other options first if we can come to some sort of solution
that works the same on all platforms/scenarios and makes sense on all of them
as I would prefer to have consistent behavior cross platform if possible.

I'm okay with having different, platform-defined behavior, if say, pip had a
--dry-run option that would at least tell the user exactly what it was going
to do and where.

I feel like the third option is likely a bad one, since "I have write
permission" is going to be the scenario that anyone on Windows is on pre
Python 3.5, or anyone who uses Conda, or anyone who uses pyenv. Erroring out
seems like the wrong thing to do in all of those cases and is soemwhat user
unfriendly to boot.

I'd be happy if the default configuration was to allow --global install if you
have permission. I don't even think Debian would change it (maybe others have
a different opinion, but at least that gives us options).

So I guess between 1 and 2 it really comes down to how bad of a practice do
we think it is for people to (conceptually) do sudo pip install --global foo.

On Debian, it is an expected use case, as long as it goes to /usr/local/lib.
It cannot go to /usr/lib.

(All this describes outside-a-venv-behavior BTW.)

• When --user is passed we always install into the user site packages.

+1

• When --global is passed we always install into the global site
• packages.

Global _vendor_ packages == +1, global site packages == -1.

• When neither is passed we implement a fallback method which:
  
  
  
  1. Detects if we're in a virtual environment and if so acts as if
     
     --global was passed.
  
  

+1, although I'll repeat that this directory inside a venv is named
/lib/pythonX.Y/site-packages

1. Detects if user site-packages is disabled and if so acts as if
   --global was passed.

Hmm, I'm not sure about this one.

1. Looks at default-install and if that exists uses --user or
   -global based on it.

+1

1. Detects if the effective user does not have write permissions to global
   site packages, and if they do not act as if --user was passed.

+1

1. Finally, if all other detection methods failed, act as if --global
   was passed.

Also not sure.

This would mean no deprecation period and that the expectation is that in the
absence of --user or --global we'll try to make the best guess based
on the capabilities of the Python we're running in, the effective user, and
any configured default install method.

Sounds like a reasonable goal.

To be clear, when I talk about "global site packages", I primarily mean "whatever distutils tells us to install things to globally". Since Python itself doesn't have a "vendor local" vs "site local" distinction this is going to be the same directory on any downstream that doesn't patch their Python. On debian however it's going to actually be a dist-packages directory and "global" actually means "site local". That's not something that pip needs to do anything special to support though because that is taken care of by Debuntu's patches to Python.

In other words, pip already installs to /usr/local/../dist-packages/ on Debuntu since Debuntu has patched distutils to support that distinction. The only time pip would mess with /usr/../dist-packages/without the user passing in a flag of some kind is that pip would uninstall files from there (because pip doesn't see that directory as special, it just sees it as a thing on sys.path). Debian has patched pip to prevent that uninstall from there though, and I think real support for that is dependent on getting real support for "vendor local" site packages in Python upstream.

We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH

How fragile is this warning likely to be? Would it convert to absolute path? Would it be case insensitive on case insensitive filesystems? Would it treat and / as equivalent on Windows? What about symlinks?

If you're installing a package that doesn't install any scripts, the warning's irrelevant anyway.

Personally, I think that a warning will probably do more harm than good.

We likely want some sort of warning if we're installing into --user and
~/.local/bin isn't on the $PATH.

+1, perhaps with a config option to suppress the warning?

We could just make it use a Python warnings thing so people could just silence
it using the built in Python warnings. If we feel the need to silence it at
all.

As of pip 6.0 we have machine specific config files located in /etc/
however these are not specific to the version of Python that is executing
pip. I think that's a good feature though somewhat separate as well (and
might go hand in hand with solving the script clobbering thing as well).

I would suggest a search order such as:

• pipX.Y.conf
• pipX.conf
• pip.conf

rooted first(?) in /etc/xdg/pip then /etc, where X.Y of course is the
major.minor version number. First one found, wins.

Split this out into #2417 as I believe it's only tangently related to this
discussion.

I see one major open question still: Long term what do we want to happen when
someone does sudo pip install foo or rather, if someone has permission to
write to the site-packages directory?

Please note that there isn't just one "site-packages" directory. For example,
on Debian/Ubuntu we don't want sudo pip install to _ever_ install into
/usr/lib, but only /usr/local/lib. And don't forget, it's dist-packages here
not site-packages (except in ~/.local because $reasons).

This is really an important point that sometimes gets forgotten (even by me).
Donald describes it best when he says there is a site-local directory and a
vendor-local directory. On Debian, site-local is
/usr/lib/pythonX.Y/dist-packages and no pip command should ever touch that,
while vendor-local is /usr/local/lib/pythonX.Y/dist-packages and it is a
documented and popular use case for some system administrators to pip install
into that directory.

So what I would suggest, since we're already down the config file path, is to
make these configurable, e.g.

• global_install_directory
• global_install_as_sudo (true/false)

pip doesn't really control these (I mean, obviously at the high level it does
because it's the one moving files around), they come from distutils/sysconfig.

I think that pip should probably never touch a vendor-local directory unless
some sort of flag was given like --vendor which would enable vendors to use
pip in their build chain. This concept doesn't really exist outside of debuntu
right now and their patches already handle this, so I don't think it's super
relevant to this discussion except to note that when I say
"global site packages" I meant /usr/local/.../dist-packages on Debubuntu
and the "site-local" packages at some point if Python accepts the Debuntu
system upstream.

1. Install into --global. This is the most backwards compatible option
   and likely represents what users expect when they type that. However it has
   the downside that it'll (silently) mess with the global Python, which on many
   systems can cause broken systems.

With the above configurability, this would be totally compatible with Debian.
Debian superusers expect sudo pip install to go into
/usr/local/lib/pythonX.Y/dist-packages. I wouldn't say it "messes with the
system" because it doesn't break the package manager, and while it can
override system installed packages (which live in
/usr/lib/pythonX.Y/dist-packages), it does so in a defined, documented way.

Yea, "messes with system" may be too harsh. Mostly I mean that
sudo pip install foo may break things if the system depends on foo and
you install an incompatible version of things. However that same case holds
true for basically anything you install into`/usr/local``.

• When --global is passed we always install into the global site
• packages.

Global _vendor_ packages == +1, global site packages == -1.

You mean this the other way around right?

• When neither is passed we implement a fallback method which:
  
  
  
  1. Detects if we're in a virtual environment and if so acts as if
     
     --global was passed.
  
  

+1, although I'll repeat that this directory inside a venv is named
/lib/pythonX.Y/site-packages

Yea, again we just ask Python where that directory is, we don't compute it
ourselves, so "where that directory is" is a venv/virtualenv thing.

1. Detects if user site-packages is disabled and if so acts as if
   --global was passed.

Hmm, I'm not sure about this one.

I don't think there's any way around this, if user site packages is disabled I
don't think that we should install into there because we have no way to know
if that disabling means there _is_ no user site-packages, or why. We can only
assume that there is none.

1. Finally, if all other detection methods failed, act as if --global
   was passed.

Also not sure.

This basically boils down to:

If we haven't found a reason we should use --user, like a --user flag, or
not having permissions to write to the directory, then don't use it. This
is what will keep sudo pip install foo without a --global flag
working. This also means this is the most importnat rule for not breaking
working software that expects sudo pip install foo to act this way.

We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH

How fragile is this warning likely to be? Would it convert to absolute path? Would it be case insensitive on case insensitive filesystems? Would it treat and / as equivalent on Windows? What about symlinks?

If you're installing a package that doesn't install any scripts, the warning's irrelevant anyway.

Personally, I think that a warning will probably do more harm than good.

Well we could implement it as the (cross platform equivilant of):

def user_bin_on_path():
    paths = os.environ.get("PATH").split(":")
    for path in paths:
        if os.path.realpath(path) == os.path.realpath(USER_BIN_DIR):
            return True
    return False

Something like that should handle symlinks, path seperators, and the like I think. Devils in the details and maybe we can't get it accurate enough to cover the various cross platform corner cases.

I do agree that the warning is irrelevant if there are no scripts being installed, and I meant to say that we'd only bother with the warning when we're installing scripts.

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and get a "command not found" without any guidance about why it might not be found. A warning at least will tell them they need to add things to the PATH. It may also help prompt users on Windows to add their user dir to their PATH manually (or run some script to do it) which might alleviate the need for the activate scripts that Steve pointed out. Although I think that would still mean that System values take precedence over user values so that still might be confusing since the order of PATH and sys.path would be reversed still.

Doh. I forgot that realpath canonicalised the path. Yes, of course that works well enough.

The thing that bothers me is that _if_ the test mistakenly sees an issue, the user gets an annoying and incorrect warning with no way of avoiding it other than changing their system to work around a mistake on pip's part. But given that the test you're proposing is probably robust enough, and we only do the check if we're installing scripts, the issue is probably rare enough to be ignorable.

Alright, I think we have some broad agreement here. I think the next step is to get a patch up. I'll go ahead and try to get something up later tonight or tomorrow.

On Feb 10, 2015, at 12:14 PM, Donald Stufft wrote:

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and
get a "command not found" without any guidance about why it might not be
found.

Debian, and maybe other Linuxes, actually have a command-not-found package
which prompts the user to install a package if they invoke a command that
isn't installed.

% inkscape
The program 'inkscape' is currently not installed. You can install it by typing:
sudo apt-get install inkscape

Maybe we could hook into that for supporting platforms.

On Feb 10, 2015, at 12:06 PM, Donald Stufft wrote:

• When --global is passed we always install into the global site
• packages.

Global _vendor_ packages == +1, global site packages == -1.

You mean this the other way around right?

I could have my terminology mixed up, so I'll be explicit:

/usr/local/lib/pythonX.Y/dist-packages == +1
/usr/lib/pythonX.Y/dist-packages == -1

but as you say previously, this isn't something that pip really needs to worry
about since it's inherited from distutils and Debuntu's distutils is patched
to do the right thing.

1. Detects if user site-packages is disabled and if so acts as if
   --global was passed.

Hmm, I'm not sure about this one.

I don't think there's any way around this, if user site packages is disabled I
don't think that we should install into there because we have no way to know
if that disabling means there _is_ no user site-packages, or why. We can only
assume that there is none.

Oh, I see what you mean now. I guess that makes sense. My concern is that it
may be more difficult to predict what pip is actually going to do, but as I
say, I'd be happy with a --dry-run flag so that pip can unequivocally _tell_
you what it's going to do.

1. Finally, if all other detection methods failed, act as if --global
   was passed.

Also not sure.

This basically boils down to:

If we haven't found a reason we should use --user, like a --user flag, or
not having permissions to write to the directory, then don't use it. This
is what will keep sudo pip install foo without a --global flag
working. This also means this is the most importnat rule for not breaking
working software that expects sudo pip install foo to act this way.

I see. I think that's okay, as the main anti-use case we want to avoid is:

normal_user$ pip install foo
HEY YOU HAVE NO PERMISSIONS
normal_user$ sudo pip install foo
HEY YOU JUST BORKEDED YOUR SYSTEM

and the "not having permissions to write to the directory implies --user" rule
takes care of that.

Oh yea I forgot about command-not-found. Could be useful. By the way do you know the right place to propose that devubtu has the user local bin on the path by default?

On Feb 10, 2015, at 4:05 PM, Barry Warsaw [email protected] wrote:

On Feb 10, 2015, at 12:14 PM, Donald Stufft wrote:

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and
get a "command not found" without any guidance about why it might not be
found.

Debian, and maybe other Linuxes, actually have a command-not-found package
which prompts the user to install a package if they invoke a command that
isn't installed.

% inkscape
The program 'inkscape' is currently not installed. You can install it by typing:
sudo apt-get install inkscape

Maybe we could hook into that for supporting platforms.

—
Reply to this email directly or view it on GitHub.

On Feb 10, 2015, at 01:17 PM, Donald Stufft wrote:

Oh yea I forgot about command-not-found. Could be useful. By the way do you
know the right place to propose that devubtu has the user local bin on the
path by default?

Not definitively, but maybe the adduser or passwd packages? The former owns
/usr/sbin/adduser and the latter owns /usr/bin/useradd.

First pass: https://github.com/pypa/pip/pull/2418

I am not sure if this helps, but…

• ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?

Some other projects use /etc/profiled.d/ for adding directories to $PATH. At least on Antergos I see the following (unless I misunderstand what these do) jre.csh, jre.sh, perlbin.csh, perlbin.sh.

# Do not change this unless you want to completely by-pass Arch Linux' way
# of handling Java versions and vendors. Instead, please use script `archlinux-java`
setenv PATH "${PATH}:/usr/lib/jvm/default/bin"

# Do not change this unless you want to completely by-pass Arch Linux' way
# of handling Java versions and vendors. Instead, please use script `archlinux-java`
export PATH=${PATH}:/usr/lib/jvm/default/bin

# Set path to perl scriptdirs if they exist
# https://wiki.archlinux.org/index.php/Perl_Policy#Binaries_and_Scripts
# Added /usr/bin/*_perl dirs for scripts
# Remove /usr/lib/perl5/*_perl/bin in next release

[ -d /usr/bin/site_perl ] && setenv PATH ${PATH}:/usr/bin/site_perl
[ -d /usr/lib/perl5/site_perl/bin ] && setenv PATH ${PATH}:/usr/lib/perl5/site_perl/bin

[ -d /usr/bin/vendor_perl ] && setenv PATH ${PATH}:/usr/bin/vendor_perl
[ -d /usr/lib/perl5/vendor_perl/bin ] && setenv PATH ${PATH}:/usr/lib/perl5/vendor_perl/bin

[ -d /usr/bin/core_perl ] && setenv PATH ${PATH}:/usr/bin/core_perl

# If you have modules in non-standard directories you can add them here.
#export PERLLIB=dir1:dir2

# Set path to perl scriptdirs if they exist
# https://wiki.archlinux.org/index.php/Perl_Policy#Binaries_and_Scripts
# Added /usr/bin/*_perl dirs for scripts
# Remove /usr/lib/perl5/*_perl/bin in next release

[ -d /usr/bin/site_perl ] && PATH=$PATH:/usr/bin/site_perl
[ -d /usr/lib/perl5/site_perl/bin ] && PATH=$PATH:/usr/lib/perl5/site_perl/bin

[ -d /usr/bin/vendor_perl ] && PATH=$PATH:/usr/bin/vendor_perl
[ -d /usr/lib/perl5/vendor_perl/bin ] && PATH=$PATH:/usr/lib/perl5/vendor_perl/bin

[ -d /usr/bin/core_perl ] && PATH=$PATH:/usr/bin/core_perl

export PATH

# If you have modules in non-standard directories you can add them here.
#export PERLLIB=dir1:dir2

As a random pip user, I just want to say _thank you_ for working on user experience issues like this and like the others captured in the Improve our User Experience milestone. Is there someplace I can contribute financially to this effort?

As an aside, I use Homebrew and always appreciated the fact that I could just brew install something and it would work without any fuss over permissions.

Once I understood why installing with Homebrew never required sudo, I tried to get into the habit of doing pip install --user something, but ironically Homebrew disables this due to a reported bug in distutils.

Given the popularity of Homebrew on OS X, perhaps y'all want to look into this purported incompatibility of Homebrew and pip install --user.

Anyway, +1 from me on making --user the default.

A quick follow up. Didier uploaded a change to Ubuntu's pip in vivid that set --user as default. I mostly also allowed it but now I think it was a mistake and I intend to revert that for wily (but probably won't SRU a change for vivid). See https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1460203 for details and rationale.

+1 for letting upstream drive the timing of this change, as the user vs system distinction is already confusing enough without platform differences coming into play.

That said, it may be reasonable for the Linux distros to start flipping the default to --user as soon as --global support lands in upstream pip (pip 8?), since doing global installs on Linux, rather than per-user or per-venv installs, is well established as being a bad idea with potentially unpredictable side effects.

I told @warsaw privately, but I'll say it here too:

I think this change is important and I plan on coming back around to my pull request to finish it up. However, I'm one person stretched over multiple projects and right now Warehouse is more important to me than this change is because PyPI legacy is becoming increasingly burdensome to operate and I feel we need to get that replaced as soon as possible. Therefore my time on pip isn't focusing on making many changes myself, rather it's focusing on guiding through changes other contributors make as well as any release process stuff.

That being said, if anyone really cares about this feature landing sooner rather than later they can pick up my work and finish it and I'll be happy to review (and hopefully merge) it. I'm happy to talk through that with anyone who wants to do it and provide guidance on the change. Otherwise it's going to wait until after I finish up the items that are more pressing to me.

Perhaps worth repeating at this juncture:

If we end up deciding against 'default to --user', we should consider the less disruptive alternatives

1. Fallback to --user (if install fails due to perimissions errors)
2. (If install fails due to perimissions errors), encourage the user (in large friendly letters) to try --user

FWIW, now that the Windows installer for Python 3.5 strongly encourages a per-user installation, I'm not so concerned about this change occurring in pip. It would probably make a better configuration option for distros (with the --global override) than a new default or a fallback.

Just wanted to pop in to +1 this issue and say thanks for working on this! We're running a scientific Python summer school currently and have had many people sudo pip install when they ran into permissions issues.

FWIW I'm in favor of defaulting to --user, or falling back to --user if install fails due to permission errors. In this case, I would prefer to give them the directions to reverse what they've done (e.g., We installed to your user directory. If you don't want this, then 'pip uninstall my_package') rather than failing and telling them how to try again.

I see a lot of bug reports on mock due to --user - at least on Ubuntu where --user's place on the path is after dist-packages, and so what pip installed isn't actually used, for any common dependency (like six) that is in dist-packages.

They put it _after_ dist-packages? That's a broken installation IMO.

Right, user installs should be before site(/dist)-packages, and then system scripts & utilities should be running with -I (or -Es in Python 2).

Yes, I agree :). It may be fixed in vivid, but I've definitely seem reports where things are coming from totally the wrong place.

I just used the system pip to install setuptools on my vivid machine, to experiment.

>>> import setuptools
>>> setuptools.__path__
['/home/robertc/.local/lib/python2.7/site-packages/setuptools']
>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/robertc/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/home/robertc/work/mock']

So this one is sane. But confusingly this was introduced in vivid - so IDK. I'll see if I can dig up a reproducer at some point (to file in the Ubuntu BTS)

Paging @warsaw

User site-packages can end up getting rearranged to after global site-packages by setuptools' evil .pth files. These were a recurring problem for me until I learned never to use setup.py directly - I'd install something, and suddenly I was using older versions of all kinds of other things. After a few times, I wrote an aggressive tool to un-mangle my system.

I think at the very least, for the time-being, pip should just print a friendly message asking the user to run --user if the command fails. While switching it to be the default has it's benefits, printing a better message shouldn't be too hard.

It's a nice "quick-fix" for the problem till we actually move to defaulting to --user.

A --global, --system, or --no-user is also required in case user mode is set in /etc/pip.conf. There is no easy way for users to override this as far as I can tell.

It's a nice "quick-fix" for the problem

Late update: #4233 did this.

People (especially newcomers) who are not using Windows should really be directed to https://github.com/pyenv/pyenv if they want to have a good experience. PEP 370 doesn't make a whole lot of sense to me today, especially since ~/.local is XDG_DATA_HOME.

I think I won't be the only one who will be opting out of this behavior if/when it comes. Hopefully the transition will be smooth and I won't have to change up how my system works and pass additional CLI flags, because I'm quite happy with pyenv. Of course, I don't know how well pyenv works with Windows but the bash subsystem could help there...

@jcrben pyenv makes quite a few assumptions about how folks are using Python (and in particular how they're acquiring their Python binaries), which means that it works very well for folks for whom those assumptions are valid, but isn't sufficiently universal to be the default recommendation (it's similar to conda in that regard).

The --user default at the pip level will have no effect inside properly configured Python virtual environments, so the impact of this change on 3rd party environment managers like conda and pyenv will be that they'll need to ensure that they're correctly signalling the environment status in a way that pip and other tools will detect (i.e. either the way virtualenv reports it, or the way the standard library's venv does).

Related proposal: https://github.com/pypa/pip/issues/1056#issuecomment-218130183.

~Has allowing the user to customize the directory (probably through an env variable) to avoid installing directly into ~/.local been considered? I suppose ~/.local does make some sense, but in my case lots of applications use ~/.local and I'm not sure I want pip installs mixed up straight inside it. I'd probably rather nest them into ~/.local/pip or something - as far as I'm concerned Python has no special right to the top level of that shared space.~ On 9.0.1 it seems to nest stuff under ~/.local/lib/python<version> plus adding executables to ~/.local/bin which I guess is fine...

@jcrben Aye, the target location is actually Python's user site directory, which mirrors the naming scheme for the system site-packages directory: https://docs.python.org/3/library/site.html#site.USER_SITE

That's already configurable at the interpreter level by way of the PYTHONUSERBASE environment variable: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUSERBASE

For those interested, I just released https://github.com/ofek/hatch which has package commands install, uninstall, and update that defaults to this behavior. For system altering, users can use the -g/--global option similar to https://github.com/npm/npm.

Given the recent issue with malware infected packages on PyPI, I think it's in pip's best interest to do what it can to discourage the use of sudo pip install ..., especially taking into account that the setup scripts can execute arbitrary code on installation. Installing infected packages as a user is bad enough, but it's much worse if they are installed as root.

Defaulting to user installs means fewer people will prepend sudo to their pip command, which means accidentally installed malware can do less.

As a concrete next step, I broke out a separate issue for the addition of a --global flag: https://github.com/pypa/pip/issues/4865

This is almost certainly a bad idea in virtualenv or conda environments. Whatever gets installed in .local when one environment is active could easily break another environment. So what's the thinking in those cases?

cc @msarahan @kalefranz

This issue doesn't apply when inside an active virtual environment - it solely applies when no active environment is detected, and none of the target definition options (--prefix, --root, etc) have been given on the command line.

How is this being determined?

@jakirkham https://github.com/ofek/hatch/blob/df2c9d46ee7713a1bc156c361cfd0f78e5935297/hatch/utils.py#L19-L25

We've been considering disabling by default ~/.local on sys.path for python inside conda environments, which by definition happens to be any python installed by conda. It's tricky though. Either way we might end up violating expectations of a group of users. Discussion at https://github.com/conda/conda/issues/7173

Wasn't pip able to load any argument from environment variables starting with PIP_? Does this work for --user? What would be the full variable name?

That would be PIP_USER.

Docs: https://pip.pypa.io/en/stable/user_guide/#environment-variables

PIP_USER=yes to be exact, thanks! Using environment variables is much easier for CI usage as you only have to define them once and they will be used whenever the commands are called.

I had to do some really ugly logic for travis where apparently some stages are run as normal user without virtualemv and some inside a virtualenv, and you have no idea which one it would be.

inside virtualenv is likely that —user will fail if you didnt use site-packages (which is causing other problems).

This ment that I hade to detect virtualenv in bash and avoid adding the —user or the run will fail.

This is ugly and caused me to waste a lot of time debugging, i am sure others will face the same.

I think we need an option that activates a fallback regarding installation: like install it on user if you can and fallback to system.

pip should fail only if fails to find a place to install a package.

Any movement on this?

I didn't see it mentioned, but perhaps pip should complain if --user is passed when run under sudo.

These unsolved issues lead programmers away from pip and even python and force them to use other packages or even other programming languages like javascript and npm with less issues like this. For a new starter who wants to use python these extra necessary options are frustrating.

Why would --user be default? Can not it be specified explicitly? I cannot run pip --target because of that it complains that it cannot be used in conjunction with user. Any idea how to avoid it?

@dmikov : it's not the default. Unless your using Debian/Ubuntu broken version, in which case you can workaround --user being automatically set by using: PIP_USER=0 pip install -t .... Or better yet, install an official version and use that.

I've dreamed at night of the --user option becoming the default (not kidding), any chance of my dreams coming true anytime soon?

I need help installing pygame it keeps saying invalid syntax

I used
python3 -m pip install -U pygame --user
is this wrong what do I do?

@flamedro56: instead of commenting on a random issue, please open a new one, and provide more information: Python version, pip version, operating system, the command output...

Installer reCAPTCHA

How about having a new config option default=system,user? It would be explicit enough that no one would think it means "always user, even in virtualenv". It would allow a graceful migration path.

See #4575.

Looks like Manjaro is trying to set --user as default.

Is there anything new from upstream about this issue?

@Tids: "filesystem-2018.9-3 adds $HOME/.local/bin to your $PATH by default." is the key extra piece that Manjaro is doing right.

Debian combines "default to --user" at the Python level with "$HOME/.local/bin is not on your $PATH by default" at the user account level, and that's the combo that breaks things.

In #7002, I've tried to make pip do a user install by default when the global site-packages dir is not writeable, and user site-packages are enabled. This should largely avoid doing a user install when you meant to install into an env, so long as you have permission to modify the env.

I'm looking for discussion of the idea first - let's not get bogged down reviewing the implementation before we've thought about whether it makes sense to do that thing at all.

It's worth noting that mixing user installs and non-user installs (of pip itself) is fraught with issues - see #7209 as one example recently.

While not directly related to this issue (and indeed if we default to user, that may ease the situation somewhat) it does emphasise the need for careful thinking about how we handle any transition to --user as default.

(I'm mainly adding this comment here so it's recorded somewhere - it's mostly not a pip issue at all, but rather a complication of how Python's USER_SITE directory works in general, that's not sufficiently well understood by users installing with --user).

It looks like the issue there is that PATH (for finding commands) and sys.path (for finding Python imports) have opposite priorities, so the launcher script from one installation tries to load the package from the other. It's presumably not specific to pip - it could affect any package that installs a script.

I'm not sure what, if anything, we can do about that, but I agree it's worth thinking about.

Yep, that's the main "complication of USER_SITE" I had in mind. IMO, deprecating wrapper scripts and only supporting python -m pip is the most effective solution. (I'm not against wrappers per se, but the problems we see with them are usually people accidentally running the wrong wrapper, so fixing wrapper script issues is usually not a pip issue as such, more of a "diagnosing and fixing user environment misconfiguration" issue)

You mean deprecating the pip wrapper script, or deprecating wrapper scripts as a thing pip can install?

I'm certainly -1 on the latter - the ability to install commands is very useful, even if it causes problems.

Deprecating pip in favour of python -m pip might be more reasonable. As pip installs into the Python environment it's running in, it makes sense to be clear which one that is. But it would still be a loss of convenience, and a big break in what people are used to.

I did mean specifically the wrapper scripts for pip itself. Agreed it's a loss of convenience, and I wouldn't be against retaining them in some form (#3164 suggests having a new pip-cli project that just supplies the wrappers) but the key point would be for python -m pip to be the supported means of running pip.

Couldn't calling the pip wrapper script show a warning if the current python (in the PATH or whatever would be used with python -m pip) is not the python used by pip? Maybe even delegate to python -m pip if called this way?

Similarly, pipenv warns if the user is already in a virtualenv, and if so uses that virtualenv instead of managing it's own, after emitting a warning.

Okay, noting here that #7002 has been approved by 3 of 6 pip maintainers at this point.

It does make pip switch from non-user installs to user installs by default, in most cases where we really should be doing user installs anyway. I have two broader concerns with making this change, that we should figure out before making the release that makes that enhancement.

• Do we want a mechanism to explicitly opt-out of user installs once we switch to it by default? If so, how would it be different from --no-user? (I'm on board for renaming --no-user to --global as an explicit opt-out).
• What is our "transition" strategy?
  
  
  
  • How do we want to be communicate this change?
  
  
  • tell users to be careful when making this upgrade?
  
  
  • What kinds of issues do we expect users to raise, when this change is implemented + released?
  
  

I'd say --global is misnamed if you're inside an environment. For flit install, the opposite of --user is --env, which I loosely meant to include the "system Python environment", but I don't think that's totally clear either. Maybe --no-user is still the best choice - it's clearly the opposite of --user.

I think --no-user (or the equivalent environment variable or config entry) should get you back the existing behaviour, since the default at present is effectively user = False.

The only scenario I've come up with where the change would be confusing is if you think you have permission to install into an environment (with user site enabled, e.g. conda) when actually you don't: a failure is clearer than doing a user install, even if there's a log message about it. I haven't thought of any good way to improve that.

It's also possible that the writability detection could go wrong. A false positive just gives you the existing behaviour (try and fail to do a normal install). A false negative would do a user install when it could have done a normal one.

It does make pip switch from non-user installs to user installs by default.

Not (for example) on Windows, where (by default) the Python install is a per-user install and site-packages is writeable by default. That's something I prefer in general about #7002, because my experience since this issue (#1668) was opened is that it's way too easy to get in a mess with "mixed" environments where you have packages like pip installed both in site-packages and user-site. So I'm now in favour of doing system installs if at all possible, and only doing user installs in cases like Linux system managed Pythons, where site-packages can't be written to.

Having said that:

1. There seems like no point to a --no-user option, as #7002 means you get a user install only if a system install would fail anyway.
2. The biggest thing I think we need to communicate regarding transition is that people need to be very careful about multiple installs, and that's something that can only be dealt with by user understanding and education, not by any workaround in pip. It's basically a core Python issue. We could extend pip check to report when there's a site and a user install of a package and the user one is shadowing the site one, I guess...

There seems like no point to a --no-user option

To be clear, this already exists, although it's probably not often useful. If you've got user=true in a config file, --no-user should override that.

To be clear, this already exists

Sorry, I should have been clearer. I see no point to having "a mechanism to explicitly opt-out of user installs once we switch to it by default", at least in the sense that I assume @pradyunsg meant, which is about opting out of the #7002 behaviour, which is not the same as "switching to user installs by default", as I said.

Actually, I'm not sure that is any clearer ;-) Maybe all I want to say is "we don't need any more options than the ones we currently have"...

Hmm... @pfmoore After #7002, what is needed to call this issue resolved?

PS: I'd written that earlier comment in a hurry and it was accidentally posted. The bullets are OK; I was still drafting the paragraph before it, when it got posted. Sorry about any confusion that might've caused.

@pradyunsg Given that I'm now much less inclined to feel that --user as a default is a good idea unless it's absolutely necessary (the cases #7002 addresses) I'd rather say that we can abandon this issue as no longer a good idea now that #7002 is done.

Taking your 2 bullet points as things we might need to count #7002 as complete, my comments above cover that.

I followed the release notes here.
https://pip.pypa.io/en/stable/news/#id3
"Default to doing a user install (as if --user was passed) when the main site-packages directory is not writeable and user site-packages are enabled. (#1668)"

My builds are now failing with

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

This seems fixed by removing the --user from my pip commands.

Was this expected behavior? The release notes to not seem to reflect this.

link to the script that now fails on pip 20.0.1 (removing the --user fixes the hard fail)
https://github.com/lfit/releng-global-jjb/blob/master/shell/python-tools-install.sh

If it is correct that user site packages aren't importable from that environment, then that is the expected behaviour, but it's orthogonal to this PR. That check was introduced years ago (#567), but it looks like it was broken with venv until recently (#7155). There's a release note for that:

Correctly handle system site-packages, in virtual environments created with venv (PEP 405).

Ah yes, thank you for your comments. I figured this out...
my script was populating .local/bin/
when I ran

python3 -m venv ~/.local

and then in a ubuntu login shell (#!/bin/bash -l) .local/bin gets added to path (so calling python3 now calls the .local/bin/python3 executable) Which we don't want.
I never needed to run python3 -m venv ~/.local
or call my scripts from a login shell, and removing both or either of these problems fixes it.

#fresh ubuntu docker
root@7d8107816f64:/# python3 -m pip install  --user --upgrade pip
Successfully installed pip-20.0.1
root@7d8107816f64:/# python3 -m pip --version
pip 20.0.1 from /root/.local/lib/python3.6/site-packages/pip (python 3.6)
root@7d8107816f64:/# ls root/.local/bin/pip
pip     pip3    pip3.6
#Now we populate ~/.local/bin/
root@7d8107816f64:/# python3 -m venv ~/.local
root@7d8107816f64:/# ls root/.local/bin/
activate          activate.fish     easy_install-3.6  pip3              python
activate.csh      easy_install      pip               pip3.6            python3
root@7d8107816f64:/# ./root/.local/bin/python --version
Python 3.6.9
root@7d8107816f64:/# ./root/.local/bin/python -m pip install  --user --upgrade pip
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

So if we ever run the executable .local/bin/python we can not pip install with --user

Oh, I'd missed that you were doing python3 -m venv ~/.local . That's likely to confuse tools like pip, because ~/.local is the prefix for --user installs (on Linux). venvs and --user are two _different_ ways you can install packages without making systemwide changes: you should use one or the other. Creating a venv in ~/.local makes some weird hybrid of the two.

The new behavior is presumably convenient for many, but 'user' installs have caused me some grief as a web developer - I often need to prepare virtualenvs that will be deployed to our production machines, and if any package is installed in my user library, then by default pip will refuse to install it to the virtualenv I'm preparing, and will still exit with success. The notification that it decided not to install the package is buried inside 30 or 40 pages of output from the script I use to prepare the virtualenv. The end effect is that once the virtualenv is deployed, the packages from my home dir are no longer available, so some random packages are missing from my production web app, and it falls over dead.

I'm willing to admit that my use case is not the typical one, so maybe this was still a good idea from a general usability point of view.

For others who have had similar problems, there are a few ways around it:

• Add --force-reinstall to all your pip install command lines in scripts etc
• Delete ~/.local/lib/python* and edit ~/.pip/pip.conf to add user = no in the global section:

[global]
user = no

• Do something more drastic, like change ~/.local/lib to a file rather than a directory :smirk:

As always, dear packaging maintainers, thank you for all you do - I'm sure it's virtually impossible to change anything without breaking someone's stuff, and the fact that you forge on anyway is actually great, because it lets us make progress.

Ah, user installs can also cause huge problems in continuous integration environments, where lots of different stuff is run as the same user account, but people still expect builds to be isolated from each other. We solved this by making ~/.local read-only, but the solutions above would probably also work.

When you create a virtualenv with the default options, it's isolated: it can't see your user or system site-packages, and consequently pip should only work with packages installed in the env. This became the default many years ago, because of exactly the kinds of issues you describe.

If you create virtualenvs with the --system-site-packages option (or very old versions of virtualenv where that was the default), they act as an overlay on the packages you'd otherwise have installed, both system-wide and in your home directory. If you want to have system packages visible but not user packages, you can run Python with the -s option or the PYTHONNOUSERSITE environment variable.

Oh! I'm very sorry, you're totally right. My organization's weird system-site-packages policy strikes again. Thanks for pointing out some solutions.

No problem. System site packages was more common a few years ago when we often relied on system package managers for things like numpy, so maybe it's worth revisiting that policy. But there may still be sensible reasons to use it today.

Closing in line with https://github.com/pypa/pip/issues/1668#issuecomment-544569965.

Many thanks @takluyver! ^>^

@takluyver Thank you for solving the problem that has troubled me for a long time
But I noticed that when installing packages with non-privileged users, Windows 10 python 3.8.2 with pip 20.0.2 is still hard failing.

ERROR: Exception:
Traceback (most recent call last):
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\cli\base_command.py", line 186, in _main
    status = self.run(options, args)
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\commands\install.py", line 253, in run
    options.use_user_site = decide_user_install(
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\commands\install.py", line 604, in decide_user_install
    if site_packages_writable(root=root_path, isolated=isolated_mode):
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\commands\install.py", line 548, in site_packages_writable
    return all(
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\commands\install.py", line 549, in <genexpr>
    test_writable_dir(d) for d in set(get_lib_location_guesses(**kwargs))
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\utils\filesystem.py", line 140, in test_writable_dir
    return _test_writable_dir_win(path)
  File "c:\program files (x86)\python38-32\lib\site-packages\pip\_internal\utils\filesystem.py", line 153, in _test_writable_dir_win
    fd = os.open(file, os.O_RDWR | os.O_CREAT | os.O_EXCL)
PermissionError: [Errno 13] Permission denied: 'c:\\program files (x86)\\python38-32\\Lib\\site-packages\\accesstest_deleteme_fishfingers_custard_m59lch'

I thought this could cause by OSError's errno being errno.EACCES but errno.EPERM.

@catPill please open a new issue so that can be tracked properly. It's perfectly plausible that I missed something on Windows.