"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.

Original Comment By: Carl Meyer

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.

Original Comment By: Jannis Leidel

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in /lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.

Original Comment By: Ian Bicking

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.

Original Comment By: Carl Meyer

+1 for this bug

Original Comment By: smyrman

Issue #167 was marked as a duplicate of this issue.

Original Comment By: Carl Meyer

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).

Original Comment By: Tomasz Elendt

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?

Original Comment By: Anonymous

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's

Original Comment By: Kelsey Hightower

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!

Original Comment By: Carl Meyer

Carl: Cool, I will get started and update this ticket with the results.

Original Comment By: Kelsey Hightower

While working on the upgrade command the following questions came up:

• What methods should pip upgrade support to specify which packages to
  upgrade? Should we support a requirements file?
• How should pip upgrade handle packages that are not already installed?
  Should we install missing packages?

pip upgrade use cases and how to handle them:

# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?

Original Comment By: Kelsey Hightower

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.

Original Comment By: Carl Meyer

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.

Original Comment By: Kelsey Hightower

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

• Tests
• Filter requirements file; add non-editables to list of packages to
  upgrade.

Running pip using the upgrade command

# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower

Last set of questions (I hope):

• Should pip upgrade parse the requirements file and filter out
  editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

• Should pip upgrade search the indexes for a later version to install?
  (This is what I am doing now). If not how should the upgrade command determine
  if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

• The package is not installed
• An upgrade is available (later version from the indexes), and no explicit
  version was requested
• The requested version is different from the installed one (Version miss
  match).
• I am deferring the requirements file to the install command until I filter
  out the non-editable requirements.

Original Comment By: Kelsey Hightower

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.

Original Comment By: Kelsey Hightower

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.

Original Comment By: Kelsey Hightower

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.

Original Comment By: Carl Meyer

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c

Original Comment By: Kelsey Hightower

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command

# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.

Original Comment By: Carl Meyer

Thanks Carl, I will wrap this up and start looking at #13

Original Comment By: Kelsey Hightower

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py

Original Comment By: Kelsey Hightower

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

https://github.com/pypa/pip/pull/313
https://github.com/jedie/pip/commit/7a31d70dabe0e809184fe1b5280c5a7ccf420dd5

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

+1

@kelseyhightower Any progress?

+1

+1!

+1

+1

Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.

Instead I'd be thrilled to see a comment "patch done!" ;)

+1

Any Updates? Is there any plan to add this, it's 3 years old now..

@pradyun See https://github.com/pypa/pip/issues/59#issuecomment-9418336

+1

I thought this was merged already. I can dust off my python skill and give it another go.

Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...

+1

What is the status of this issue?

It is blocked by #988

If it's really important for you, there are workarounds for upgrading all packages; I threw together a script to do so in parallel(https://github.com/ariccio/update-pip-packages), and there are many other implementations elsewhere on the internet.

There's two parts to this issue. upgrade-all may be blocked by gh-988, but I don't see how upgrade is blocked. pip upgrade can be a simple alias for pip install -U --no-deps. This would resolve one of the main problems of using install_requires in setup.py files. Can't this be done sometime soon?

pip upgrade can be a simple alias for pip install -U --no-deps

from the description:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

that is not pip install -U --no-deps

“non-recursive” in this context does not simply mean –no-deps. A non-recursive upgrade will upgrade dependencies, but only if needed to fulfill parent requirements.

@qwcode thanks for the clarification. Then that's not what I care about. Why would you call this "non-recursive", it's still recursive but just a bit smarter/different?

I was under the impression from the discussion in gh-571 that really non-recursive was the desired default. That would certainly make sense, and prevent having to always write code like https://github.com/scipy/scipy/commit/8e7ee0c4b3c16.

in #571, "non-recursive" is not --no-deps it's the "smarter/different" recursive as you say.
notice the test_upgrade_with_needed_recursive_upgrades() test from that PR.

without getting stuck on terms, there's 3 things:

1. the "smarter/different upgrade", i.e. the kind that occurs in OS packaging that can also imply upgrading dependencies (but only if they actually _need_ upgrading).
2. what pip does now, which is to upgrade all dependencies, regardless of need or conflict resolution.
3. --no-deps

some possible ways to distinguish #1 and #2

1. "non-recursive" vs "recursive"
2. "normal" vs "recursive"
3. "only if needed recursive" vs "do it regardless recursive"

I'm open to using the best terms... just not sure what those are.

I think I'm liking this "only-if-needed recursive" phrase. maybe i should use that here in the docs:

I like it too. If you'd describe the three options all together as

a. non-recursive
b. only if needed recursive
c. recursive (or "do it regardless recursive")

that would be clear.

Then you want to pick good defaults. For both upgrade and install -U, either (a) or (b) could make sense. I strongly prefer (a), but (b) makes sense too given that that's what OS packaging does.

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U.

To clarify why I strongly prefer (a): an unsuspecting user wanting (b) and getting (a) will have to read a message saying "non-recursive upgrade can't satisfy all dependencies, please use only-if-needed recursive", which is not that big a deal. If the default would be (b), that unsuspecting user may end up with an upgrade or even a broken install of a package he really didn't want to be touching. That can take hours or even days to recover from.

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U

the reason to leave install -U alone is just for compatibility reasons, and then to eventually deprecate it.

If the default would be (b), that unsuspecting user may end up with an upgrade
or even a broken install of a package he really didn't want to be touching

if a user wants necessary dependency upgrades to go unfulfilled, they should specifically have to ask for that using --no-deps. there's no way in my mind that could ever be the default. that behavior would do more damage than what you're considering (which is the outlier case). Over and over again, users would be left not having the dependency upgrades they needed.

Deprecating install -U sounds good.

I agree (b) is more common than (a). Even if it would be 100x more common, which is not the case I think, more damage is untrue. Reading a clear error message before install begins is so much better than for example a compile error halfway through, that imho (a) is still the better default.

Relying on --no-deps may be fine for very experienced users, but new users are only going to learn about it after things go wrong.

Anyway, looks like I won't be able to convince you. Then back to the manual

try:
   import dependency
except ImportError:
    install_requires.append('dependency')
else:
    if dependency.version < 'x.y.z':
        raise ValueError('Please upgrade dependency to x.y.z')

it is.

@qwcode thanks for the detailed explanation. I hope this will be moving forward in the near future - only if needed recursive would be a huge improvement over the current behavior.

Spending again many hours this week fielding issues on why we don't use install_requires, I'm adding a :+1: to moving away from the current behavior.

Are there any updates on this issue?

We have now 2015 and I still need to copy from pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U stackoverflow

Yeah, essentially all the progress is linked to this issue.

GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)

Yeah for sure but I don't get the reason why this "simple" feature has such a delay.

What is the reason for this?

Am 16.04.2015 um 05:28 schrieb Alexander Riccio [email protected]:

Yeah, essentially all the progress is linked to this issue.

GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)

—
Reply to this email directly or view it on GitHub.

as for pip upgrade-all, the consensus seems to be to wait for work related to #988 before releasing any shiny new commands that are still ultimately flawed and can be dangerous to a working environment. A simple version of upgrade-all that doesn't properly resolve requirements can easily break existing packages

+1

What are you doing along 4 years?

+1

+1

+1

@muhasturk Currently, waiting... https://github.com/pypa/pip/issues/59#issuecomment-93646462

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1 ...sorry for the spam, but running pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U hurts!

+1

In case anyone is interested in working on this, I think the comments above are somewhat confusing -- AFAICT the actual status is that pip upgrade-all is currently blocked by the need for a proper dependency resolution system, but pip upgrade could be implemented at any time. If someone wants to work on that part then it would be extremely awesome :-).

(See also the thread starting here and @dstufft's reply here and comment here agreeing with the above assessment.)

here's another discussion from the pypa-dev list from a year ago (that agrees that pip upgrade FOO could be done now) https://groups.google.com/forum/#!searchin/pypa-dev/pip$20upgrade/pypa-dev/vVLmo1PevTg/oBkHCPBLb9YJ

Thanks @qwcode! I also just saw the new description at https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade, that's helpful.

+1

If I'm not mistaken:

If xxx is not installed:

• pip upgrade xxx would be equivalent to pip install xxx
• pip upgrade xxx --recursive would be the equivalent of pip install xxx --upgrade

If xxx is installed:

• pip upgrade xxx --recursive would still be the equivalent of pip install xxx --upgrade (by design)
• but there is currently no equivalent for pip upgrade xxx, this could be pip install xxx --upgrade-only-if-needed/--upgrade-lazy

It's not clear what the added value of a new command is over adding a new option to pip install ?

It's not clear what the added value of a new command is over adding a new option to pip install ?

The default pip install -U behavior is unacceptable to many projects. See my comment above (https://github.com/pypa/pip/issues/59#issuecomment-52434862). And here's a longer explanation: http://article.gmane.org/gmane.comp.python.distutils.devel/24218

If it is unacceptable, then I guess you plan to change your usage of pip install --upgrade to pip upgrade ?
Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?
What does a new command provide that a new option could not ?

It's all paving cow paths. It's not @rgommers personal usage he's worried about, it's his users. People are going to reach for the "obvious" answer if they don't know better and right now the obvious answer is problematic for some major pieces of the Python ecosystem.

Indeed. People don't read docs. Indeed we'll fix all the install docs we can find, but as soon as a user sees -U or --upgrade that's what he'll use. The chance that people will use --no-deps or --only-as-needed or whatever before they've been seriously bitten by it is remote.

Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?

And to be clear: we avoid pip install --upgrade now _and_ do not use install_requires because of this. That's how bad it is.

I think there's a minor misunderstanding here -- IIUC @xavfernandez agrees with adding the non-recursive upgrade functionality, their question is just why the UI to that functionality has to be as a new top-level command instead of a new option to pip install.

@xavfernandez: Note that there's currently a discussion happening in #3194 about what would be the clearest UI for this functionality.

(And pip install -U will be deprecated and then removed, so that answers the question about how users will know not to use it :-).)

I think there's a minor misunderstanding here

I'm pretty sure that there isn't. And yes, we're only debating UI. But that's important.

Yup only debating the UI and I agree it is important. But what I see coming is that once pip upgrade is out, there will be no reason to keep using pip install...
And the UI to install any package will become "pip upgrade foo"

there will be no reason to keep using pip install

which is a problem why? The issue (at least for @qwcode in comments above) with changing pip install --upgrade to the right behavior is breaking backwards compatibility. So pip upgrade is the way to avoid that break _and_ get the right default.

@rgommers: to be fair though, we _will_ get some non-zero number of confused questions about why did you tell me to run pip upgrade foo when I don't even have foo installed, it didn't work (argue back and forth until they actually try running the command instead of assuming that it won't work and pretending that they ran it)

On #3194 I made the comment that maybe the right way forward is to just remove --upgrade from pip install, make pip install always do an upgrade of the explicitly named items, and default to a minimal upgrade strategy for dependencies with a --recursive flag to get the current behavior on upgrade.

I think I agree that I feel like with #3194 the default UX will become pip upgrade foo instead of pip install [-U] foo. The only problem I have with that is I think it's sort of confusing to have the two commands and given that I think pip upgrade would be the "right" one for people to use, I think it's crummy to have the obvious name be the wrong name.

I have a migraine so I could also just not be thinking entirely correct, but I'm sort of feeling like figuring out how to handle the deprecation path to that suggestion might be the right way forward.

Well, I won't be arguing that backwards compat is important here. I was always in favor of changing the behavior of pip install --upgrade. It's the cleanest, and the cost seems small. But it looked to me like it was vetoed by @qwcode, so pip upgrade was the next best thing (and already OK'ed by the pip devs a year ago).

If we now want to go back to changing defaults/behavior for pip install, great.

@dstufft that makes total sense to me - the headache can't be that bad:)

Since we agree to deprecate --upgrade, we could keep that and add two options to pip install: --upgrade-only-needed and --upgrade-eager , the second one being an alias for the deprecated --upgrade

@xavfernandez Forcing decisions onto users is kind of crummy I think. Unless I understand the fairly subtle issues I'm not sure that I would really know if I wanted --upgrade-only-needed and --upgrade-eager.

@xavfernandez so you're proposing that in the final situation the command would then be pip install --upgrade-only-needed? Seems ugly.

It would be much better to have something that maps to the equivalent of the semantic version pinning available in e.g. npm, hex, or Cargo. That would certainly need adjustment in the Python context, since (a) PEP 440 versions do not and cannot map precisely to semantic versioning and (b) the Python community at large doesn't necessarily hew to semantic-like versioning in its releases even within PEP 440.

Nonetheless, something along those lines would be very helpful, as would the notion of pinning a specified version.

Given the constraints, in the short term one viable option might be doing something analogous to homebrew's upgrade and upgrade --all commands, where the latter simply goes ahead with upgrading everything (potentially with a warning the first time).

In the long term, however, it would be enormously useful to create shared conventions about what versions mean in the Python packaging community, and building pip support around that could be a big part of it.

The obvious migration path would be:

pip version X:

• add options pip install --eager, pip install --no-eager, where --eager means that for requirements with no version restriction attached we attempt to install the latest version even if there is already some version installed, and --no-eager means that we're satisfied if any version is installed
• also add option pip install --eager-recursive with the current -U semantics
• --no-eager remains the default
• if none of these options are specified, and pip install is passed a requirement with no attached version number, and this requirement is already satisfied, then output a warning saying that we didn't upgrade your package, but in the future we will, so you should use --no-eager if you want to keep the current behavior
• add a deprecation warning to pip install -U referring people to pip install --eager-recursive

pip version Y:

• switch the default for pip install to --eager

pip version Z:

• remove pip install -U

I made Y and Z different versions because I am eager for --eager so maybe we could make that change on a more aggressive schedule, while keeping pip install -U around for a bit since a lot of documentation already refers to it, and it doesn't cause any harm. But obviously Y = Z is an option :-)

@chriskrycho: That all sounds like great stuff to have, but this discussion is about how to handle the situation where the user is explicitly trying to request an upgrade to the latest version of a package, which is a situation that commonly happens already today and we don't have a good answer to. I don't know if there's a bug open requesting pinning support yet, but if not then maybe you should start one?

Absolutely; my point (which I now recognize I totally failed to make explicit!) was simply to note that whatever solution is adopted here shouldn’t conflict with taking that kind of a tack in the future.

What's the normal deprecation path for pip? Time/version-based? How long does something need to be deprecated before removal?

By the way, I think "eager" is a word that users will not understand. recursive / only-as-needed recursive / always recursive is much clearer.

We typically do a two version deprecation cycle.

Major versions I assume? Minor versions are really fast (7.0 -> 7.1 was one month). Typically half a year between major versions, so 1 year of deprecation warnings?

Yea, sorry. Major versions. If we deprecate something in 8.x it will be a yellow warning for the rest of 8.x, a red warning for all of 9.x, and removed in 10.x.

Thanks.

Copying @dstufft's comment from https://github.com/pypa/pip/pull/3194#issuecomment-152357835 here, because it's the best summary of the situation:

I guess the tl;dr is that I totally think that we need to fix the "default" behavior, we just need to settle on the specific implementation of fixing it:

1. Keep the UX be pip install --upgrade and change the default.
2. Move the UX to pip upgrade with the "right" default and add a --recursive flag.
3. Move the UX to pip install, remove --upgrade, and add a --recursive flag.

My 2c:

(1) has a good UX, can be done now
(2) a bit worse UX (have both install and upgrade), can be done now
(3) second best UX, changing pip install can be done now, removing --upgrade will only be done in 10.0 (deprecated in 8.0 and 9.0).

I don't see why the backwards compat issue (which should be minor either way) for (1) would be worse than for (3). So (1) seems best. If bw compat is a real worry, then choose (2).

Anyway, time to call it a night.

I think the difference between (3) and (1) comes down to whether we think that install-or-upgrade is the most common operation people want -- if so then making it the short command as in (3) is probably best. It's not a big deal either way though.

By the way, I think "eager" is a word that users will not understand. recursive / only-as-needed recursive / always recursive is much clearer.

I'm not at all attached to the "eager" spelling, but recursive just doesn't make sense as the main word for toggling upgrades on and off. I guess we could use --upgrade-non-recursive (future default), --upgrade-recursive, --upgrade-none or so, but it still foregrounds the confusing recursive behavior (I would have no idea what --upgrade-non-recursive meant if I weren't already familiar with the weird recursive behavior of the legacy -U option), and --upgrade-none misleadingly sounds like it will make sure that no packages are upgraded even if needed to preserve self-consistency.

If we go this route then I don't care that much about the spelling period, since the option that most people want would just be the default and the recursive-upgrade and no-upgrade options would both be special rarely used things that most users could ignore.

--upgrade-non-recursive (future default) ...

of course that's confusing. but you're ignoring the simplest option, which is not to provide this switch at all. because if it's identical to the default behavior, why would you ever need it?

I think the difference between (3) and (1) comes down to whether we think that install-or-upgrade is the most common operation people want

I'd expect "install". At least I know that I only upgrade packages that I heavily use on purpose, but each time I see something interesting/useful it's a simply pip install pkgname away.

(not that I'm a typical user though, so my expectation could be wrong)

I think I'd be in favor of:

• No new pip upgrade that would be an alias for pip install
• pip install (without --upgrade* option) behavior does not change
• pip install --some_name would only try to upgrade the packages specified on the command line and not their dependencies
• pip install --some_other_name to keep the old --upgrade behavior available

And then there are two options:

• we don't need/want a deprecation path, then --some_name can be --upgrade and --some_other_name can be whatever seems best
• we need a deprecation path, then --upgrade in pip8 will give a warning saying it is deprecated and we should use --some_other_name to keep the old behavior or rather switch to a safer --some_name to only upgrade the specified packages and their dependencies only if needed.

In this second case, --some_name can not be --upgrade. So we have to find two new options names for --some_name and --some_other_name

It seems to me that the obvious "best" solution is to keep pip install as the command, and to change the behaviour of --upgrade to not upgrade dependencies. The only real issue is backward compatibility, but have we had _any_ users arguing that they rely on the current behaviour?

Personally, I'm inclined to the "go for it, and to heck with backward compatibility" view. I could argue that the current behaviour is actually a bug, and this change would be a bugfix. But I do think that at some point we need to be able to say that we've reached a good point, and we're going to take a much stricter view on backward compatibility. I don't think we're at that point yet, but we may need to start letting the community know what we feel still needs to be done in order to get to that point.

There is still (IMO) a need for some sort of pip install --upgrade :all: command to upgrade all installed packages. But that's new functionality, so isn't relevant here.

The current behavior _is_ useful, particularly for projects like Pyramid which are not a single project but are actually a collection of projects. It'd be useful to be able to do something like pip install --upgrade-recursive Pyramid in order to upgrade Pyramid and all of it's dependencies to the latest version. Removing the recursive thing all together means I either have to manually track down all of the dependencies and do pip install --upgrade Pyramid dep1 dep2 dep3 dep4 (with the entire dependency tree) or I need to do a hypotheticalpip install --upgrade :all: and possibly upgrade more things than I actually want to upgrade. In #3194 at least one user mentioned that they would like the current behavior to be available via a flag because it's useful in some cases.

Is there any reason (other than backwards compatibility) to not make it so pip install implicitly does a "minimal" upgrade? I'm trying to figure out in what situation I would actually want it to only install and not do an upgrade (IOW, one where the latest version is fine if I don't already have it installed, but not if I do have it installed) and I'm having a problem coming up with a case where I would want the current behavior.

I like the idea of a _new_ pip upgrade [--recursive] command as in #3194 (and deprecating install --upgrade)

I have concerns about any choices that break compatibility or have complex deprecations or that burden an already option-heavy pip install with more changes or complexity. Also, I'm personally attracted to an "upgrade" command, that by default, just upgrades similar to how distro tools work. "install" installs, and "upgrade" upgrades...

I'm inclined to the "go for it, and to heck with backward compatibility" view

I'm not : ) at least for core logic like this.

make it so pip install implicitly does a "minimal" upgrade

for me, considering this seems like a non-starter, right? consider the # of automated build breakages that might occur. also, people get locked into a conceptual model at a certain point about what pip install is... and this would force a reload of a new mental model. people don't like that.

@qwcode I don't know, I don't think it'd be OK. The main problem with pip upgrade is either you remove the way for people to say "give me the latest version of this regardless of if it's installed or not" or you have two commands that do almost entirely the same thing (pip install and pip upgrade). There might be some breakage in the short term, but I'm a bit worried about the actual mental model of pip upgrade because I can see people just wide spread replacing their install instructions from pip install foo to pip upgrade foo (but why am I upgrading something that I don't have installed already?!).

Er, I think it would be OK I mean.

wide spread replacing their install instructions pip install foo to pip upgrade foo

roger, that's why I think upgrade should just upgrade. as for the "turd" --fill-in-missing, I'm not stuck on that, so don't take me to mean that I have to have that, but see below.

"give me the latest version of this regardless of if it's installed or not"

• if I don't have FOO, then I pip install FOO and I get the latest version.
• if I have FOO, then pip upgrade FOO and I get the latest version.

I think we're talking about 2 cases:

1. I don't know whether I have FOO, and I want _one_ command to get the latest FOO, whether I have it or not.
2. I want _one_ command to get the latest of FOO and BAR. I have FOO installed, but not BAR.

Do we owe users _one_ command for these? I'd prefer simplicity and clarity in the commands over giving people shortcuts. But if users demand it, then that's where something like --fill-in-missing comes in.

Also, I'm personally attracted to an "upgrade" command, that by default, just upgrades similar to how distro tools work. "install" installs, and "upgrade" upgrades...

Point of clarification: I just checked how three popular distro tools work, and AFAICT (partly this is based on man pages b/c I don't use all these myself):

• apt:
  
  
  
  • apt install <pkg> performs an "upstall" -- either upgrades or installs depending on whether it's already installed or not.
  
  
  • apt upgrade upgrades all installed packages
  
  
  • apt upgrade <pkg> doesn't exist, the only way to upgrade a single package is apt install <pkg> (which also accepts various kinds of version specifiers)
  
  
• yum:
  
  
  
  • yum install <pkg> performs an "upstall"
  
  
  • yum upgrade upgrades all installed packages
  
  
  • yum upgrade <pkg> upgrades a specified list of packages. I don't know what happens if they're not already installed.
  
  
• homebrew:
  
  
  
  • brew install <pkg> performs an "upstall"
  
  
  • brew upgrade upgrades all packages
  
  
  • brew upgrade <pkg> upgrades a specified list of packages. I don't know what happens if they're not already installed.
  
  

So using install for upgrades is actually what distro tools do :-). Doesn't mean we have to as well; just a data point.

I don't know whether I have FOO, and I want one command to get the latest FOO, whether I have it or not.

I think the most obviously compelling use case for this is in documentation. Example: the django docs that I linked in the other thread that instruct people to run pip install -U <some package>. You don't want to tell people to "run pip install <pkg> except if you already have it installed run pip upgrade <pkg>" because that's way too confusing for newbies, but if install doesn't upgrade then just telling people to run pip install <pkg> is also going to create confusion where people start working through your tutorial with an old version installed and then have no idea why things don't work right and blame you for it. So you definitely need a single "upstall" command, and it needs to be simple and obvious to include in docs.

Follow-up on Homebrew's behavior for comparison: brew upgrade fails if you try to install a non-installed package:

$ brew upgrade git
Error: No such file or directory - /usr/local/Cellar/git

Also, the current brew upgrade behavior is changing (see discussion here); in a future release, it will switch to requiring --all to upgrade all installed packaged.

Point of clarification

I knew someone would pick up on this. : )
I almost responded to myself. I was focused on upgrades.

yum upgrade [...] I don't know what happens if they're not already installed.

it doesn't do anything

yum install performs an "upstall"

true, but the default behavior is to prompt for confirmation, which pip doesn't have.

the django docs that I linked in the other thread

to be clear, they weren't the actual Django docs. the Django docs say pip install Django (https://docs.djangoproject.com/en/1.8/topics/install/)

it's actually a bad thing I think, given the way -U currently works, to get people in the habit of using install -U.

in your linked example, it was for redis which doesn't have any dependencies I think, so it's ok in this case.

going to create confusion where people start working through your tutorial
with an old version [...] So you definitely need a single "upstall" command

IDK, It's not so obvious to me that tutorials should be telling people to upstall in general. If someone is doing a tutorial for FOO, and they already have FOO installed, then maybe they shouldn't be blindly "upstall"ing.... maybe an upgrade would damage the existing environment.

I think the tutorials should be telling people to "upstall" (great term!) because a fair number of them are written by the projects themselves and when people go to look at tutorials they rarely look first to see what the existing version of FOO they have installed is, rather they tend to go look at whatever the latest docs are.

I also agree that the way -U currently works, it's a bad habit to get people in the habit of using it indiscriminately, but the key thing I think is withe the proposed behavior I don't think it's a bad habit, but rather a good thing to do. If you need to ensure a particular version is installed than you should already be using == and we can add a flag to turn the default "upstall" back into a plain "install" for people who need to make sure that the currently installed version OR the latest version is installed (though I still can't come up with a scenario in which this is actually what anyone wants).

I think the tutorials should be telling people to "upstall"

again, though, what is this environment that they're in, where it's obviously ok to upgrade? If it's a distro system python environment, then certainly not. If it's some other environment, they made for an application, then an upstall could be damaging. if it's an environment they just made for the tutorial, then they don't need to upgrade.

a key point here is that PyPI is not a "curated" repo, like distro repos are. you can be fairly assured that an "upstall" is safe from a default distro repository... you don't know that with PyPI. Every upgrade is a gamble somewhat. "install", as it is, is conservative.

keep in mind that making pip install act like an "upstall" w/o fixing #988 will break things.

consider where A and B are installed and A requires B<2. Now I go run the new pip install B, which upstalls B to v3 or whatever (due to not considering the installed constraints), and now my environment is busted.

for people who need to make sure that the currently installed version OR the latest version is installed (though I still can't come up with a scenario in which this is actually what anyone wants).

It's trivial to come up with such scenarios. Example (not made up unfortunately): currently statsmodels is broken by the latest pandas (0.17.0). I'm a user of both. I have a bunch of Python versions installed and venvs lying around. So I know that for any Python/venv I'd like to install pandas only if it's not there yet. If it is, I will probably also have statsmodels and then "upstall" will break it.

@rgommers Hmm, so you're absolutely sure that you don't have a pandas 0.17.0 in a virtual environment anywhere? I guess it could happen though I think I'd probably spell that by doing pip install pandas!=0.17.0 or pip install pandas<0.17 but I guess it's not entirely unreasonable to rely on what you already have installed.

Hmm, so you're absolutely sure that you don't have a pandas 0.17.0 in a virtual environment anywhere?

That's not what I said. I do have a venv with pandas 0.17.0, just one in which I don't have statsmodels. pip install pandas<0.17 will do the job, but it hadn't occurred to me to use that (mainly because I didn't need to, the current install command works for this purpose).

Anyway, I don't have much of a preference for or against upstall. I just wanted to point out a realistic scenario when you said you couldn't come up with one.

+1

This discussion seems to have stalled without coming to a conclusion. Any option on the table is a major improvement on the current status. It looks like all the relevant pros and cons have been considered; there's only a difference of opinion on the relative importance of backwards compat vs. the optimal API. Maybe the pip developers can attempt to finalize a decision?

@rgommers they probably did decision? https://pip.pypa.io/en/stable/user_guide/#only-if-needed-recursive-upgrade

@rgommers they probably did decision? https://pip.pypa.io/en/stable/user_guide/#only-if-needed-recursive-upgrade

@Liso77 thanks, but no - it really is the discussion on this thread that needs finalizing. That documentation you linked to just points back here.

@rgommers from link I posted above I read that developers are going to make upgrade command. And don't plan to add "only if needed" option to install command because they didn't create term for this option and use only description in quotation marks. So from your (or dstuff's) comment on 30 october - it is: _2. Move the UX to pip upgrade with the "right" default..._ (btw I prefer this)
I am sure you were thinking more about this topic and you see more subtle thing that I do. So please write what (if you think that something) is still open. If you meant that it could be good that developers write here explicitly their decision and eventually close this issue - I agree.

Btw. situation could be much less problematic if we had something like

pip freeze > checkpoint.txt
pip checkout checkpoint.txt

which could guarantee revert installation to any "checkpoint". (pip install -r is not good in this moment) I was thinking to add this proposal in new issue but I am sure I have to study and analyze more to bring really useful proposal. I see there are many caveats but I really like to have possibility to

pip checkout git+https://github.com/somebody_trusted/pip_distro4ubuntu@bleeding_egde_branch
#or
pip checkout git+https://github.com/somebody_trusted/pip_distro4ubuntu@stable

I mean that could solve (with somebody_trusted (or community with lot of unit tests) help) something like pandas and statsmodels problem you described above.

(useful to see for this topic is also: https://pypi.python.org/pypi/peep)

@Liso77 it's simple: the discussion on this issue is not finished. There is a PR waiting with the upgrade command implemented (gh-3194), but the pip devs (at least qwcode and @dstufft, and probably also @xavfernandez and @pfmoore) need to decide whether that's what they want, or they want to change the behavior of pip install instead. Until that happens, nothing will be changed.

+1

So, it appears that introducing the new upgrade command was agreed on and the latest unsettled debate is whether or not there should be an "upstall" command and if yes, which main command (install or upgrade) should get it as a switch or default.

Throwing in my personal opinions:

I think having a fail-safe singular command for upstalling is "worth it", _if only_ for cases where you want to verify programmatically that an environment is up to date for any reason. Why "programmatically"? Because it is not interactive, i.e. there is no user to react to an error, warning or other message pointing him to the brother or sister command because a package does or does not exist already.
Optionally, a prompt could be added asking the user whether he wants to upgrade or install instead of just complaining.

That said, this "upstall" command should not be the default of either command (install/upgrade) and thus require an option to be invoked. This adds clarity to the semantically clearly defined "install" and "upgrade" verbs, and in my opinion all package managers that upstall by default on one of these are doing it wrong. Prompting for action (as yum appears to do) is alright.

This leaves us with upgrade --install (shorter than --install-missing) and install --upgrade. Semantically, "upgrade to latest [and install if doesn't exist]" and "install latest [or upgrade to latest if already exists]". Not much of a difference here imo. install --upgrade is the already existing version and would be recognizable, but backwards-incompatible.
Here's another one for the mix: A new command named install-upgrade (or upstall?), where neither install nor upgrade get the option to "also do the other" but a new command is introduced with clear semantics in an important position: the command name itself. This would be my preference.

TL;DR: Introduce upgrade and upgrade-install, deprecate install --upgrade. All of the functionality with clear semantics.
Add messages or optionally prompts to install and upgrade commands if their operations fail because of a package already or not existing.

Did this again manually this morning. Are there any news on this feature request?

+1

+1

+1

Everyone, please use the thumbs up reaction on the OP and stop spamming everyone's inbox.

since pip has a different QA model than linux distributions, i'm fairly certain, a random upgrade all command is just a breakage in the making

pypi has no database of package versions and compatibility that is actually verified and tested,
so unlike a distribution with QA pypi has practically no own QA and relies completely on the projects uploaded to it (which vary in quality)

with the current sets of constraints i'm convinced that its more of a curse than a blessing
and if anyone wonders why distributions are so outdated, QA takes actual time and effort ^^

Everyone, please use the thumbs up reaction on the OP and stop spamming everyone's inbox.

Note that we don't actually need _any_ sort of "I'd like this" response from people. We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

So personally, I'd appreciate it if people would refrain from pinging this issue unless they have working code that at least offers a starting point for the implementation - and they are willing to follow it through to implementation.

Otherwise, we'll get to it when we can. I personally hit this issue regularly, and I'm a core pip developer, so I can guarantee that if I ever have sufficient time myself to work on this I will. In the meantime, "any progress?" comments tend to simply demotivate me because they mostly just feel like people demanding the right to dictate how I spend my hobby time...

a starting point for the implementation

I thought that was the purpose of https://github.com/pypa/pip/pull/3194? From there on the discussion drifted into several alternatives for command naming and behavior without a consensus by the core developers. Would it help to implement multiple of those alternatives? Or does https://github.com/pypa/pip/pull/3194 just need a refresh?

@sfriesel You missed out "and are willing to follow it through". Looks like the person who made that PR either ran out of time or interest (a common problem). The next step on that one is probably for either the OP or someone else who's willing to take over ownership of the PR, to write up a PEP-style summary of the current position, what the various outstanding issues are, what he's suggesting as the resolution for these various issues, and a call for comments to restart the discussion. If the debate is too big for github, maybe he needs to take his document to distutils-sig for more input. If he can't decide on a resolution that he's happy with for the various issues, or can't get consensus from the debate, then yes, #3194 is probably dead (until someone else picks up the problem again, at which point I'd hope they incorporate all the work done their into their new PR).

Well over 50% of the work on a proposal like this one is "management" (documenting proposals, managing discussions, drawing out consensus). It's not necessarily the bit people like to do, but with a codebase as widely used as pip, it's an essential part. (And yes, that's a problem - it's much nicer to say "I write code as a hobby" than "I do project management as a hobby" :-))

@pfmoore Totally agree.

The number of proposals and amount of discussion, spread across many locations, related this issue over the years makes it extremely difficult to approach this issue and get a clear picture in a short while.

@RonnyPfannschmidt

pypi has no database of package versions

It's true. This is information provided by the packages at run-time (of the setup.py) and it adds a bit more complexity to the process of resolving the dependencies. Moreover, due to how the current logic behaves, some packages (even prominent ones like scipy) don't list a dependency (like numpy) if it is installed to avoid re-compiling it unnecessarily.

That would be solved by PEP 426, among other things but I don't know what's the status of that PEP.

@pfmoore

Note that we don't actually need any sort of "I'd like this" response from people

It's just tempting to say +1 or put a thumbs up to put it out there that you would like to see this happen. It lets the person feel like they have added a little something to this issue, contributed to it. I'm not saying that pip devs need to be motivated or persuaded, it's just that everyone wants to say that. And...

I'd appreciate it if people would refrain from pinging this issue unless they have working code

Putting a thumbs up is not going to give anyone (annoying) notifications/emails and we still get to see the number of people who have shown interest. I don't think anyone should mind that but I could be wrong.

Putting a thumbs up is not going to give anyone (annoying) notifications/emails and we still get to see the number of people who have shown interest. I don't think anyone should mind that but I could be wrong.

Ah, I hadn't realised that doing that avoided the notifications. Yes, that's a good idea.

The next step on that one is probably for either the OP or someone else who's willing to take over ownership of the PR, to write up a PEP-style summary of the current position, what the various outstanding issues are, what he's suggesting as the resolution for these various issues, and a call for comments to restart the discussion.

For what it's worth, I was earlier doing a write-up on this and dependency resolution, PEP style. I'm going split it up and update it now, adding all the things that have been proposed in the past few months. I'll post the one for upgrade command as a Gist once it's done (~2 days?) and link to it from here.

I'm not willing to take over ownership of the PR just yet. I intend to help reignite the discussions and get to a final design that can be implemented. If some stuff around me goes to plan, I should be able to follow this through to implementation but I don't want to commit to it following this through just yet.

/cc @qwcode @dstufft @pfmoore @xavfernandez

Here's the link to my writeup: https://gist.github.com/pradyunsg/a5abeac4af90fbdc54bb266c32c0d2d8

I initially just linked to various places (~30 links) for the reader to refer to and tried to copy-then-edit-to-fit all the comment threads. The document was pretty long and my opinions slipped through. I ended up starting from scratch. It's shorter now though and not opinionated. I've still marked it as WIP though.

If there's any issues in it, please comment on the Gist. I'll make the corrections as soon as I can.

WARNING: _SLIGHTLY_ LONG COMMENT AHEAD

I like the idea of fixing the existing --upgrade behaviour. Are there any concerns regarding this other than backwards compatibility? Adding an upgrade command doesn't seem like a nice idea, due to the high overlap between install and upgrade.

FWIW, git did something similar to what we're discussing about --upgrade, changing the default behaviour of git push in 2014. link

If there's interest in changing the behaviour of pip install --upgrade, here's a seeding proposal (refer to this as P1) for that discussion:

• Starting next major version (9.0), pip install --upgrade prints a warning:

```
PipDeprecationWarning: pip will stop upgrading dependencies to their latest
version unconditionally by default starting version 11.0.

To maintain current behaviour after the default behaviour changes, to
the install command pass --upgrade-strategy eager.

To start using the new behaviour immediately, to the install command
pass --upgrade-strategy non-eager.

To quelch this message, .

For more details, read .
```

The documentation page linked would provide a quick overview of the change and it's effects. It would then address the most common question: "What should I do?" and "Which upgrade strategy should I use?" in the most direct manner reasonable.

• The message could use a cleanup.
• The option could even be 2 flags, --upgrade-eager and --upgrade-non-eager or something. That's bikeshedding, i.e. last step.
  
  
  
  • Unless something drastic happens, in version 11.0, pip install --upgrade switches behaviour to be non-eager when upgrading dependencies.
  
  

Edit: I put a 3 major version cycle for the change, just to give more time to people to switch over. It might be unnecessary. Maybe, 1 major version printing warnings and the next making the change is enough?

@dstufft had come up with a similar (same?) idea.

If an upgrade command is the way forward, here's a seeding proposal for reinitiating that discussion (refer to this as P2, if you want to):

1. pip upgrade copies over most options and behaviour from pip install.
2. --dry-run flag will be added to install and upgrade commands that print what pip will try if it were to actually run.
3. pip install <out_of_date_pkg> does not change behaviour.
4. pip upgrade <not_installed_pkg> would fail.
5. pip upgrade <up_to_date_pkg> would do nothing, say why it did nothing and exit with a zero exit code.
6. pip upgrade <out_of_date_pkg> would perform a non-eager recursive upgrade.
   
   
   
   • :white_check_mark: Default to non-eager recursive upgrade
   
   
7. pip upgrade --eager package would behave like today's pip install --upgrade.
   
   
   
   • :white_check_mark: Allow opt-in to current behaviour
   
   
8. pip install --upgrade would print a RemovedInPip10Warning.
   Because it would be removed.

Additionally, what are your thoughts on:

1. Making the upgrade command interactive? (something like what apt-get does)
   
   
   
   • making pip install <pkg> behave like OS package manager's install i.e.upstall?
   
   
2. Having --only-binary by default in the new upgrade command?
   
   
   
   • If it's ok, what about adding --only-source and --no-source in upgrade command?
   
   
3. Adding an option to "hold" the update of a package? IMO, this is a requirement for upgrade-all.
4. Does it make sense to distingush between these cases i.e. require different CLI?
   
   
   
   • install: not-installed -> up-to-date
   
   
   • upgrade: out-of-date -> up-to-date
   
   
   • upstall: {not-installed, out-of-date} -> up-to-date
   
   

I didn't propose anything about the "install as upstall" because I do not feel strongly either way about it. It is consistent with OS-level package managers but would be a major breaking change. I'm not sure how useful it would be or what the exact behavior and interface would be like...

Maybe sleeping over this might help.

Personally I don't see much value in keeping consistent with a behaviour few if any people are happy about. I wonder if having one for Python 2 (embracing stability) and another for Python 3 (embracing change) would be better or worse.

@ekevoo

I wonder if having one for Python 2 (embracing stability) and another for Python 3 (embracing change) would be better or worse.

One, That's off-topic. Two, if you do so, there will end up being divergence. That'll alienate people currently using Python 2 because it'll (hopefully) reach EOL one day and they'll have to shift over to Python 3, having to adapt for a differently behaving tool. Thus, pip shouldn't do divergence (and won't, IMO). Might as well keep it together in a single tool then.

OTOH, I do see what you think about how much backward compatibility is acceptable...

A warning in pip 9, plus additonal options to tweak the behavior of pip install --upgrade, --eager/non-eager (or some other name) seems fine to me.

Bump. Another notification for everyone.

@pfmoore @dstufft @qwcode I don't want to be nosy but could you please take out some time for this, over the weekend or coming week?

OK, so my comments:

Re the two ways forwards, I don't have an opinion. Pick whichever one you feel is the better option, implement it and we'll see how it goes from there.

Re your additional questions:

1. I'm -1 on making the command interactive. Unless there's a specific issue that the user needs to make a decision about, along the lines of "this consequence of what you asked for is something you were clearly unaware of, and could be a problem - do you want to proceed" then it's just useless noise. I'm against "are you sure" prompts in general. If you have a specific proposal for a prompt you think is worth adding then feel free to ask again with the details included.
2. We shouldn't make upgrades different from installs, so no to --only-binary by default. For my personal use, I'll probably always want pip upgrade --only-binary in the first instance, but I _want_ it to be explicit.
3. I'm not sure what you mean about "holding" updates. Please clarify. But as a general response, I'd say don't bother in the first instance, let's keep the initial PR free from "optional extras" - they can be added later.
4. Isn't that the main topic of debate on the various discussions of this feature? For a start, doesn't it affect the basic question of install --upgrade vs upgrade? I don't recall there having been a resolution of that question, so I'm a little surprised you expect a "simple" answer to this one? I certainly don't have a good answer.

(By the way, I don't have easy access to the gist, so if there is any content on there that isn't in your summary, I've missed it sorry).

I know it's hard work to write a PR, and demotivating to have it seem to get bogged down in debates and questions, but ultimately, I think someone is going to have to write one, and put it out there as "this is the solution I propose - any comments?"

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.

If you probably always want something, it should be default¹. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

I'm not sure what you mean about "holding" updates.

I'll let this rest until we get to the point of talking about adding upgrade-all functionality.

note to self: apt-mark concepts for pip

I'm -1 on making the command interactive. Unless there's a specific issue that the user needs to make a decision about

Ditto.

Isn't that the main topic of debate on the various discussions of this feature?

Exactly why I wanted to know what you guys think.

I don't recall there having been a resolution of that question, so I'm a little surprised you expect a "simple" answer to this one?

I don't expect a "simple" answer just yet. I'm hoping we arrive at one, preferably simple, answer through discussion.

By the way, I don't have easy access to the gist, so if there is any content on there that isn't in your summary, I've missed it sorry

More than a summary, it's a follow up to the write-up. Please do read it as soon as you can do so.

Pick whichever one you feel is the better option, implement it and we'll see how it goes from there.
[snip]
I think someone is going to have to write one [PR], and put it out there as "this is the solution I propose - any comments?"

I was thinking along the lines of "let's figure out exactly what we want first" and then go ahead and implement that.

¹ unless it breaks someone else's world

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.
If you probably always want something, it should be default. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

If _everyone_ probably always wants something, it (maybe) should be a default. But I was merely commenting on my own preferences. I have honestly no idea whether --only-binary is what most people would want (it probably depends on whether we're talking long or short term).

There's a strong difference between binaries (wheels), sources that can be built anywhere (typically pure Python) and sources that need a compiler or other external prerequisites. Sadly, pip can't distinguish the latter two, and that makes --only-binary less useful (particularly as a default).

And I value consistency between commands over "do what I mean" defaults, which again may just be my personal preference.

More than a summary, it's a follow up to the write-up. Please do read it as soon as you can do so.

I've read it, it didn't add much that I wasn't aware of (as I've followed the various threads on this), so my comments stand. But it's a good summary of the state of play, so thanks for writing it.

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.

If you probably always want something, it should be default. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

If _everyone_ probably always wants something, it (maybe) should be a default. But I was merely commenting on my own preferences.

Clarification: The "you" in my comment was referring to the end-users (plural one), @pfmoore being one in that specific case. I really should have used some other word.

Some thoughts on upgrade strategies. Personal opinion, all of this, of course :-)

Suppose I do pip upgrade foo

1. If I don't currently have foo installed, I expect an error. For me, "install" and "upgrade" are two separate activities that I don't want merged.
2. If there's no version available that's newer than what is currently installed (see below re --only-binary) then I expect a notification that there's nothing to do.
3. Otherwise, I expect foo to be upgraded to the latest version. That's what I asked for, so it should happen.
   
   
   
   • I'm not convinced that adding constraints makes sense. What would pip upgrade foo>1.0 mean if I have 1.1 installed but 1.2 is available? It's not an upgrade if it leaves 1.1 there, but if it upgrades to 1.2, it's the same as pip upgrade foo. Possibly you could ascribe meaning to something like pip upgrade foo<2.0 but IMO that would be a very unusual case, and not worth the complexity. So let's assume pip upgrade just takes a list of package names (as opposed to requirements).
   
   
   • Similarly, I don't know how to interpret pip upgrade <path_to_archive/wheel>. Let's assume it's not allowed in the first instance.
   
   
4. I may or may not want to allow pip to attempt to build from source (this has to be a user decision, as pip can't tell whether building from source will work, and has no capability to try again with a different version if a source build fails). So --only-binary must be a user option, and if specified, means that pip should ignore source distribution when working out "what is available". (We could of course default to only binaries, and have a --allow-source option, but either way upgrade should match install in this regard).

OK, so that's handled the explicitly specified packages. I doubt there is anything controversial in there (except maybe for the parts I say we should omit until later). Now onto dependencies.

• The primary rule in my mind is that we should never do anything to dependencies, unless we are _actually_ upgrading the user-specified package. So that's the starting point. If the target doesn't get upgraded, don't even look at dependencies. Ever.
• My view would be that _all_ dependencies have to be handled the same way - whether they are direct or indirect dependencies of the user-specified package, is irrelevant. I don't think this is controversial.
• A fundamental assumption here should be that the user doesn't know what the list of dependencies is. After all, the point of having implicit dependencies is so that the user _doesn't_ have to manage them. So any solution that needs the user to know explicitly about a dependency is flawed, in my view (unless a command fails with a message about a particular dependency, in which case the user might rerun with that dependency specified in an option - but we should be looking to work first time in as many cases as possible, so this should not be seen as the norm).
• I would say that by default, the approach should be to only upgrade dependencies that _have_ to be upgraded to satisfy the new constraints. There's a nasty problem lurking here, as constraints may change depending on what gets upgraded. But whether we do a complete solution to that issue, or simply attempt some "best effort" solution isn't that important (I doubt complex dependency graphs with conflicting constraints are common in real life). The important point is the principle, that we don't upgrade anything that the user hasn't asked to be upgraded, unless we have to.
• Having a flag to say "upgrade everything to the latest version" might be useful (for backward compatibility, at least). Even better would be to have tools (probably external to pip) that analyze and report on upgrade options. Then users could decide for themselves. But I don't know that I would ever see a need for either of these options myself.
• Rather than an "eager upgrade" option, I'd be much more likely to want an upgrade-all command (or upgrade --all if we want to keep to a single command). This should be semantically identical to upgrade with every installed package listed on the command line. Once I'm doing blind upgrades of packages (typically I may not know or have documented the dependency tree of my packages) I probably just want "everything". Even more so if I use virtual environments to isolate things.
• The --only-binary question appears again here. Certainly if the user specifies --only-binary for the main upgrade, dependencies should be assumed to imply --only-binary. But even if the user allows for source installs when upgrading the main package, introducing the risk of failure involved in doing a source upgrade of a dependency seems unreasonable. So I'd suggest we should only ever consider binaries for upgrading dependencies, unless the user explicitly allows source. That would imply needing an option --allow-source dep1,dep2,.... I expect this proposal to be controversial, but consider a pure Python package foo that is distributed only in source form, that depends on numpy. We have foo 1.0 depending on numpy >=1.10, and foo 2.0 depending on numpy >=1.11. The user has foo 1.0 and wants to upgrade. They don't have the means to build numpy. They have to allow source upgrades for foo, but they don't want to waste time trying to build numpy 1.11 (or worse still, building it might work but give an unoptimised build, which breaks their system). They could of course specify --only-binary numpy but they may not even be aware that foo depends on numpy.

That's about it. For me, the requirements are pretty clear (and honestly, I don't think they are particularly controversial if we ignore implementation issues). However, implementation is the killer here (basically the above pretty much demands a full SAT solver approach as has been discussed previously). What compromises we are willing to make because implementing a SAT solver is too hard, is where the debates will likely lie.

As far as I know, the following will be the implementation challenges:

1. A full SAT solver is hard. I don't know enough to understand what alternatives are simpler, and how they differ from a SAT solver. (Well, eager upgrade is, I believe, simple enough - that's why it's what we do at the moment). Specifically, when we'll get a situation where we upgrade something that the simple "don't upgrade anything you don't have to" principle says shouldn't have been upgraded.
2. I've glossed over anything but the most basic dependency constraints. Once we get involved in upper limits on versions, or lists of allowed versions, things get nasty. But realistically, such cases are likely to be pretty rare (I'd love someone to do research on actual dependency information from PyPI - I may try to do that, but I doubt I'll get the time).
3. When we have multiple targets - pip upgrade a b c - do we treat this as 3 separate actions, "upgrade a", then "upgrade b" then "upgrade c", or do we merge the 3 into a single "combined" action somehow (note that because I propose treating dependencies different from targets, this is _not_ the same as pip upgrade dummy where dummy depends on a, b and c and we assume dummy was upgraded).

If anyone wants to take issue with any of the above comments or assertions, that's great. Most of this is just my opinion, and I honestly don't work on complex environments - my usage is likely to be mostly about maintaining a system install and a few virtual environments on Windows (hence, debates about binary vs source installs are important to me, but complex dependency graphs not so much). The more use cases we can check our assumptions against, the better.

So we currently have two operations, pip install and pip install --upgrade, however I think that both of these things do something that nobody actually wants to happen in real life.

For pip install, we get this weird sort of behavior where you might get the latest version (or you might not!) based on what is already installed in your system. I think this sort of weirdness makes pip harder to use, and I don't think it really makes a whole lot of sense (what situation would you care about not upgrading but you still want to install the latest if you don't already have it?). I think that the fact that this command has this weird maybe-latest-maybe-not behavior is the reason why we see a lot of people reaching for pip install ---upgrade as their primary command rather than pip install.

However, pip install --upgrade isn't much better, while it gives you consistent behavior, it's overly eager, it extends into a full on upgrade of _everything_ which the user may not even be aware will be implicated in their pip install --upgrade command. I do see some use for this behavior, but I don't see it as a good default for a top level command.

What I think we should do here, is figure out a path to making `pip install ...more consistent. In that I meanpip install should always end up having the latest acceptable version (given specifiers and modifier flags like--only-binary``) regardless of what was previously installed. I think that we should otherwise attempt not to touch anything that wasn't explicitly mentioned, except where required to do so (and if we are required to do so to satisfy a constraint, then we should again upgrade to the latest available to us that matches all constraints).

One way to think about the difference between these: if x is not installed, then it would be okay for pip require x to install some random old version. (And heck, maybe it should, to force people to be robust against it.) But no-one would ever accept pip install x installing some random old version.

(This is a thought experiment, I'm not actually advocated that plain Dist-Requires: x or pip require x in an environment with no x should pick a random old version of x to install.)

Well, I guess there's also one other case where users type pip install x, which is when they already know it's installed, but they're used to systems with the Debian-style behavior where install always upgrades. Obviously these users also want it to upgrade :-).

I'd rather not add a pip upgrade command.

pip users already have some expectations of the behavior of pip.
The main pain point comes from the pip instal --upgrade default behavior so let's focus on that.

A warning in pip 9, plus additonal options to tweak the behavior of pip install --upgrade, (--eager/non-eager) followed in pip 10 by a change in its default behavior seems simple enough, should remove the main pain origin and does not break pip users mental model of pip.

Yea, I'm definitely trying to approach this from a "what operations does a user want to do" versus "what operations should X command do". I am then taking this high level operation that I _think_ users want to do, and trying to map it to a single named command (as explicit as it would be, pip install-the-latest-version`` is not very user friendly).

This is obviously all very fuzzy, but I can say that 99% of the time what I do is pip install -U <whatever> because that best matches what I expect from an installer given what is currently available. I also see in various automation scripts people using pip install -U. It's also what other popular package managers for languages do by default like npm. In the cases where I do see people not using -U, it's because of the recursive nature of it, not because they don't want the latest version of things installed.

pip users already have some expectations of the behavior of pip.

TBH, though, the main expectation I have about pip as a user is that on about 50% of invocations it will do something surprising and obviously wrong. And I'm not the only one -- see e.g. @glyph's talk at pycon last week where he observed that pip is great except that all the defaults are broken and require unbreak-me flags. This isn't a criticism of the pip developers, I understand you/we are all working under a complex set of constraints, and it isn't an argument that we should just break things willy-nilly for the sake of breaking them -- pip has a lot of pieces and lots of them are fine! But given the overall state of pop's defaults, I'm _really_ not convinced by arguments of the form "pip has always done X, therefore pip should always do X". If you want to argue for pip install to refuse to upgrade then that's cool but I'd much rather see that argument made on the actual merits, not just on inertia.

Yea, I certainly agree with @njsmith and @glyph here. We have a number of bad default behaviors, and part of moving forward I think needs to be figuring out how we can get rid of those and deal with those breaking changes to move things towards better defaults.

This particular change may not be one of those, but I think it is.

Yea, I'm definitely trying to approach this from a "what operations does a user want to do" versus "what operations should X command do". I am then taking this high level operation that I think users want to do, and trying to map it to a single named command (as explicit as it would be, pip install-the-latest-version`` is not very user friendly).

OK. Suppose I'm willing to count myself as convinced (somewhat) about this. However, if we assume this is the right thing to do, what about dependencies? I'm 100% convinced that "try to install numpy from source" is almost always not what is wanted. So we only install numpy from wheels, unless the user explicitly mentions numpy. Take that as a given for now, and then suppose we have the situation I described earlier.

• User has foo 1.0 and numpy 1.10 installed, foo 1.0 depends on numpy >= 1.10
• PyPI has foo 2.0 available, depends on numpy >= 1.11. There are no numpy 1.11 wheels.

What does pip install foo do? Presumably leave the user at 1.0, as it's a working installation? But should it succeed (as foo is installed) or fail (as it couldn't install the latest version)? If the former, how does the user find out that his system is out of date? If the latter, how does the user say "I just want to make sure foo is there"? OK, the user can do pip list --outdated, see that foo 2.0 exists and do pip install foo (BTW, that still seems utterly weird to me, I have foo, but I know there's a new version, so I do pip install??? Nevermind...) And get a success, but 1.0 remains installed?

One reason I prefer two commands is that the user's intent is completely clear, so edge cases like this can be handled correctly because we know the user's expectation.

Maybe this is all obvious if you're used to apt-get. But I can definitely say it's not at all clear to someone like me who isn't.

If you want to argue for pip install to refuse to upgrade then that's cool but I'd much rather see that argument made on the actual merits, not just on inertia.

My argument is on the basis of explicit rather than implicit. Most definitely _not_ on "we've always done it this way". I have no issue with the idea that maybe apt users are used to "install" meaning "possibly upgrade". I'm really not so sure that other users would be.

One thought - does apt have a "package already exists - upgrade?" prompt? I could imagine install-as-upgrade being less surprising to me if it was "install-or-ask-if-I-should-upgrade"... Of course, pip doesn't behave interactively like that at the moment, although obviously making it do so is an option.

That is an interesting question-- Other package managers mostly get around that by either not really having binary packages at all so it's _always_ by source, or by pretty much only dealing with binary packages so it's _always_ binary. We're in a sort of weird in between that makes it harder.

Given that, I think that by default for right now, we should pull down numpy 1.11 source package and attempt to install it, but if they've specified --only-binary, then our hypothetical resolver (which we desperately need, SAT or back tracking or whatever) would see that foo-2.0 is not a resolvable installation and will then fall back to installing foo-1.0. This is not a great default, particularly for users on Windows where compiling is _much_ harder, but I think it reflects the reality of today.

That being said, one thing I really want to do is to start to try and push things towards a world where we can change pip's behavior again so that by default we can be binary only, and require opt-in for source releases, but I don't think we're at a place yet that we can do that.

@pfmoore: I think the question of binary vs. source installs is somewhat orthogonal? It seems to me like the exact same questions arise for a dedicated pip upgrade command, so while they're real issues we need to address, splitting upgrade and install just moves the issues around rather than simplifying them? Also, in the particular case of numpy, we now ship wheels for basically all the platforms that we care about supporting :-).

But here's how I would suggest handling these issues for pip install foo (specifically this command -- I'm not talking about pip install foo==whatever or pip install ./foo-*.whl or pip install bar where bar has Requires-Dist: foo):

1) Query the index to find the latest candidate version of foo; call this $LATEST. If no candidate versions exist, then error out.

2) If $LATEST is already installed, then finish successfully.

3) Check if there is a distribution of $LATEST that can be installed on the current environment. (Example reasons why there might not be: there are only wheels, but no sdists, and the wheels don't match the current environment. There isn't any matching wheel, and there is an sdist, but the user has passed --binary-only :all:. There isn't any matching wheel, and there is an sdist, but the sdist has some flag saying "I only work on python 3" and the user is running python 2 -- the ipython/jupyter folks will probably propose this as a new feature soon, b/c they want to drop python 2 support for new releases in January while still providing a python-2-supporting LTS.)

4) If $LATEST _does not_ have a viable distribution: issue a warning to tell the user that a newer version is available but not for their environment, ideally with a hint as to what they need to do if they really do want the new version (e.g., "ipython 6.0.1 is available but requires python >= 3.4, and you have python 2.7 -- consider upgrading python", or "numpy 1.12 is available, but there is no binary for ppc64 and you have disabled building from source -- consider --allow-source numpy). Then remove $LATEST from the list of candidate versions and go to step 1.

5) If $LATEST _does_ have a viable distribution, attempt to install this distribution.

6) If installing it fails (e.g. b/c it's an sdist and there's no compiler), then error out. Otherwise, finish successfully.

@njsmith binary-only is somewhat orthogonal, agreed, but IMO if we're trying to design commands that "do what the user expects" then it's critical to get this right at the same time.

@dstufft the problem with "install numpy unless the user says --binary-only was explained in the example in my previous epic post - (1) say that foo is only available in source form, and (2) the user may not (and indeed should not need to) know that foo depends on numpy. Then the user can't say --only-binary :all: and has no idea they need --only-binary numpy until _after_ a (long) failed compile. Or (possibly worse still) a successful compile that leaves the user with a non-optimised numpy (these days, numpy compiles out of the box on Windows, but gives a non-optimised build).

I completely agree that long-term we should default to binary-only, but we're nowhere near that yet (at a minimum, it should imply that pretty much every pure-python package is available as a wheel).

@pradyunsg As you can see, there's still a lot of unresolved issues here. Are you still interested in taking this forward? I'm reluctant to fire off another long debate if it's just going to stall again...

@pradyunsg As you can see, there's still a lot of unresolved issues here. Are you still interested in taking this forward? I'm reluctant to fire off another long debate if it's just going to stall again...

I expected there to be. I am interested in taking this forward. Let's do this! :smile:

I suggest collecting a list with all of what a _user_ wants pip to do and then propose solutions for each of them until all (or a sufficient amount) of them are solved by a pip command (with options/defaults) or can be handled with "no action".

To start things off, here's a most likely incomplete list:

1. A user wants to install a package that is not yet installed.
2. A user attempts to install a package that is already installed.
3. A user wants to upgrade a package that is installed.
4. A user attempts to upgrade a package that is not installed.
5. A user wants to ensure they have the latest version of a package installed, whether it was already installed or not.
6. A user usually does not want all dependencies to be upgraded and instead have them only be satisfied.
7. A user wants to upgrade/install a package but does not want to build from source (neither the package nor its dependencies). Probably more likely than:
8. A user is willing to upgrade/install from source.

My personal proposals for this, as a user:

1) & 7) pip install foo should attempt to resolve to the latest available version (considering dependencies) and install it. Algorithm would be @njsmith's.
2) pip install foo → show a warning that foo is already installed and propose using pip upgrade foo
3) & 7) pip upgrade foo attempts to install the latest available version of foo, again following @njsmith's algorithm. If a newer version cannot be installed because it isn't available for the platform and there is no sdist or the user doesn't want to build from source, show that. Succeed in either case and only fail if installation itself failed.
4) pip upgrade foo fails if the package is not installed.
5) pip install-uprade foo or pip install foo --ensure-latest or pip install foo --upgrade (basically the same as currently except non-eager).
7) All operations should be non-eager and an --eager flag would be available to get the old functionality of install --upgrade. If a dependency is not yet installed, install the latest. If it is already satisfied, do nothing. If it is not satisfied, install the latest still satisfying version (for upper-bounded requirements).
8) pip upgrade foo --allow-source or pip upgrade foo --allow-source numpy, if foo depends on non-binary numpy releases. Edit: I don't know if this would be applicable, see comment below.

Feel free to extend the list and post your own proposals.

@pfmoore I'm not sure what the alternative is though? The state of the world today is that wheels are gaining ground but they're no where near ubiquitous so I'm not sure I really see a good option that doesn't allow source releases by default right now.

@dstufft my proposal was to allow source for explicitly named packages, but default to only binaries for dependencies. That way the user gets no "surprise" build steps. It's a compromise, sure, but it reflects what I (have to) do manually at the moment.

@FichteFoll Above all else, my main use case is for an "upgrade all" capability. Look for everything that's currently installed, and if there are newer versions available, upgrade them.

The packages I have installed don't typically have anything other that ">=" dependencies (and most don't even have that) so there's nothing complex here. Just grab the latest version. My biggest restriction is that there are some packages I can't build (numpy, scipy, lxml, pyyaml, matplotlib, pyqt) so I only want binaries for those. I can probably just put --only-binary <these> in my pip.ini (or at least I hope I can...)

Secondary: Install package XXX (latest version), plus any of its dependencies I don't already have. Don't upgrade dependencies I already have if they satisfy the new package's constraints. I always know that I don't currently have XXX.

Tertiary: Upgrade a single package XXX that I (know that I) have installed. Don't alter any other packages unless required to maintain dependency constraints (and even that is theoretical - I've never encountered the situation in real life so I don't know what would be the best resolution for me). My intent is always "upgrade to latest version". I have never encountered a situation where this would break dependencies of already-installed packages. If it did, I _think_ I'd like a warning that I hadn't got the latest version (and why) plus an upgrade to the latest version that is acceptable. In my mind, this situation currently translates to pip install -U although the behaviour for dependencies is not what I want. The main reason I'd do this though is because of the current lack of a suitable "upgrade all" (or to deal with cases where a new "upgrade all" command failed to work like I wanted).

All of the discussion about dependencies and constraints is, in my experience, almost entirely theoretical. I currently have 160 packages installed in my system Python (a mix of scientific, data analysis, web and general programming modules). 100 of them have no requirements. For the remainder, none have anything more complex than a list of packages - no version constraints or anything more complex than Requires: six, dask, pillow, networkx. The longest list of dependencies had 9 items.

@pfmoore Isn't that going to break a lot of things? A quick list of things that I can think of off the top of my head that I know are very popular packages is anything depending on:

• SQLAlchemy (optionally requires a compiler, will use pure Python if not).
• PyYAML (optionally requires a compiler, will use pure Python if not).
• Markupsafe (optionally requires a compiler, will use pure Python if not).
• PyCrypto (always requires a compiler)
• pycparser (never requires a compiler)
• httplib2 (never requires a compiler)
• anyjson (never requires a compiler)
• zope.interface (optionally requires a compiler, will use Pure Python if not).
• docopt (never requires a compiler)
• Mako (never requires a compiler)
• itsdangerous (never requires a compiler)
• amqp (never requires a compiler)
• ordereddict (never requires a compiler)

and so on, you can see a long list of the most popular packages and whether or not they have wheels at http://pythonwheels.com/.

@pfmoore

Look for everything that's currently installed, and if there are newer versions available, upgrade them.

I sourced this command from some SO question a while ago that I am currently using for this. It's suboptimal, but works for most of my packages, except one. (It uses the py launcher because I'm on Windows.)

pip list -o | cut -d " " -f 1 | xargs -n1 py -m pip install -U

Now, the one problem I have with this is the flake8 package, which has these requirements:

Requires-Dist: pyflakes (>=0.8.1,<1.1)
Requires-Dist: pep8 (>=1.5.7,!=1.6.0,!=1.6.1,!=1.6.2)
Requires-Dist: mccabe (>=0.2.1,<0.5)

Specifically pyflakes is a problem since that has a newer version available and gets updated with the above command, causing flake8 to fail doing anything (since it checks the version).
So this is indeed something that needs to be considered and I also would like to have proper upgrade-all functionality (without breaking requirements!).

@dstufft why? If foo depends on pyyaml and I ask to upgrade foo, pyyaml doesn't get upgraded (no new binaries) but foo can still be upgraded, as there's still the original pyyaml present.

For new dependencies (or on install where a dependency isn't always present) you have to install, so if there's no binary you take source. I'd personally consider "choose older version with binary over newer version with source" but that's getting perilously close to defaulting to --binary-only which I agree we're not ready for.

Hmm, maybe the issue I have is actually with the --only-binary option, which is too coarse. If we had a --prefer-binary option, that said "only use binaries, unless that means there are no candidates, in which case retry allowing source", I suspect many of my concerns that over-eager upgrading would result in breakage might be alleviated. Which, as @njsmith suggested, means that the binary/source distinctions I'm focusing on may well be orthogonal to this ticket (although it would just change my position to "there's no satisfactory solution to my requirements without something better than --only-binary being available"...).

Specifically pyflakes is a problem

OK, so that's not a situation I have (as I say, I don't have anything with dependencies that complex installed). I don't have a problem with refining "upgrade all" to upgrade things to "the latest version that doesn't result in breakages" but AIUI that needs the "SAT solver" approach to work out the correct solution. That's an implementation issue though - the design should indeed always give correct results.

@pfmoore I think a --prefer-binary flag could be a good option, regardless of the outcome of this ticket. Likely bundled with a warning when that ends up not installing the latest version that would otherwise have been installed.

@FicheFoll: I don't think trying to rederive the entire ui from first principles is going to be very productive. There's a relatively clearly defined piece of the problem that's on topic for this particular issue, and if we try to expand the scope to everything at once then it'll just bog down again.

On that topic, it looks like the key place we differ is this: suppose that a user has the mental model that pip install foo is only for transitioning things from uninstalled to installed, and that they understand that foo is already installed. I assert that a user with this mental model _will never type pip install foo_. Therefore, when some user _does_ type pip install foo when foo is already installed, we can conclude that their mental model is not like your (2). _Either_ the first part is wrong: they know foo is installed and they expect pip to upgrade like some other popular package managers, _or_, the second part is wrong: they are unaware that foo is installed, in which case they are expecting install to leave them with the newest version (because that's what install does when packages are not installed, and they think this package is not installed).

For the record, one of the reasons I don't like the pip install ... only ever goes from uninstalled to installed and pip upgrade ... only ever goes from installed to a newer thing installed is because I find the user experience pretty crummy. Software that knows what you wanted it to do, but instead of doing that thing it tells you to invoke some different command is incredibly frustrating.

$ pip install foobar
I'm sorry, but foobar is already installed, you want to run ``pip upgrade foobar``
$ pip upgrade foobar
...

Would do nothing for me except annoy me, even though it's technically "correct".

The flipside of that, if you say "ok, if pip install foobar already has foobar installed, then we'll act like it's not installed, and if you do pip upgrade foobar then we'll act like it's already installed, we end up with two commands that do basically the same thing, except with _maybe_ some minor differences in exactly how things are processed, which says to me that they belong as a singular command with some --options to deal with the edge cases. I think this is better because it means users don't have to try to make a choice between which one they want up front, there is one command to install stuff and for most users that will generally do the right thing. If some user in a specific scenario that requiers some choices on their part, then they have to pay the cost of making some choices about what --flags to use.

OK. Consider me convinced. I've done some research, and even the Windows installers I'm (supposedly :-)) familiar with do install-as-upgrade (if you install something like VirtualBox, it says "you already have a previous version installed, do you want to upgrade?") Powershell has install-package, which doesn't say anything specific, but there's no upgrade-package. Etc. So I guess just having the single "install" command is the norm.

Which of course means that, unless someone else wants to argue, technically this PR can simply be closed as "not going to be implemented". But it's still as good a place as any to discuss _how_ we want to remodel the install command, I guess.

OTOH, maybe we do close this as rejected, and someone opens up a new issue, with a concrete proposal of how they suggest the install command should be modified. It might at least provide a clearer starting point for discussions.

A question. Does anyone think we're at a point yet where someone could put together a complete proposed behaviour from all of the suggestions here and elsewhere? Covering how dependencies are handled, what happens with constraints (and when they conflict), binary vs source, how we support the "upgrade all" scenario, etc? My personal feeling is that we need someone to make that decision, to give the discussion a reference point, or we could just debate details forever. I could probably do that, but I am unlikely to be able to implement what I propose (e.g., I would propose an "optimal dependency resolution" approach, which implies a SAT solver AIUI). So it would be better for someone who was willing to implement their proposal to step up (and deal with the inevitable debate and bikeshedding :-)).

I'm still concerned about some of the implications in the points made here, but I'm not sure I have the energy to debate them until there's a real possibility of an implementation.

I fully agree with @dstufft 's latest https://github.com/pypa/pip/issues/59#issuecomment-224341218
That's why (once again) I'd advocate for the simple solution of --eager/--non-eager options.

I also agree with @dstufft's comment, as noted (we go for a single install command and no update command).

However, I'm not sure what --eager / --non-eager entails. I guess --non-eager means don't upgrade anything that doesn't have to be updated (either because the user specified it explicitly, or because it's too old to satisfy the new set of dependencies). Does --eager then mean upgrade every dependency to the latest possible version, whether necessary or not? Which would be the default? (I'd argue for --non-eager)

And a question - would this encourage scientific packages to correctly declare their dependencies? That has to be an important consideration.

Bikeshed point. The option names --eager and --non-eager are pretty unintuitive. I think we need better terms. Maybe something explicit like --upgrade-dependencies.

This PR also suggests an upgrade-all command, which is the key use case for me. Are you saying that we reject that command, or simply that you don't have an opinion on it?

My understanding of what --eager and -non-eager means matches what @pfmoore just said (whether dependencies are only-if-needed or always installed), and I agree that --non-eager should be the default. I also agree the name is kind of crummy though I don't have a better solution that isn't a mouthfull. Maybe --[no-]recursive or something, I don't know.

I think something like an upgrade-all command could be a good addition (as long as it made sure not to violate the version specifiers of anything). I would probably just call this upgrade though and just let it take no arguments to restrict what it operates on.

tl;dr
Discussion for an upgrade-all-packages - stays here.
Discussion for prefer-binary - Over to #3785
Discussion for install-as-upgrade - Over to #3786

If we had a --prefer-binary option, that said "only use binaries, unless that means there are no candidates, in which case retry allowing source"

This is a good idea. While related to this issue, I do think it deserves it's own issue. (@dstufft's comment makes me think there's interest to pursue it). I took the liberty to open #3785 for further discussion on this.

My understanding of what --eager and -non-eager means matches what @pfmoore just said (whether dependencies are only-if-needed or always installed)

Eager upgrade would install the latest allowed versions of all (sub-)dependencies. Non-eager upgrade would upgrade a (sub-)dependency only if it no longer satisfies the requirements of a package.

The option names --eager and --non-eager are pretty unintuitive.

I agree. I like the idea of putting the behaviour behind a single flag.
--upgrade-strategy=eager/non-eager

It's a mouthful too but it does convey the intent more explicitly. Is it too verbose? Maybe.

I think bike-shedding is best done after finalizing semantics.

Does anyone think we're at a point yet where someone could put together a complete proposed behaviour from all of the suggestions here and elsewhere?

I think so. We do, at the least, need to establish some base facts we agree upon. I'm on this.

OTOH, maybe we do close this as rejected, and someone opens up a new issue, with a concrete proposal of how they suggest the install command should be modified. It might at least provide a clearer starting point for discussions.

I think we should leave this issue open for now, because it proposes an upgrade-all. You have already noted that upgrade is not happening. I opened #3786 for futher discussion on the install-as-upgrade command.

I'm still concerned about some of the implications in the points made here, but I'm not sure I have the energy to debate them until there's a real possibility of an implementation.

I am willing to take this forward all the way through to implementation. I definitely don't want to waste everyone else's effort in getting to a consensus about this.

I think everyone agrees that an 'upgrade the world' command would be really nice to have, but IIUC it's blocked while we wait for the resolver work to land. In the mean time, I posted a more concrete proposal for single-package upgrades on @pradyunsg's new dedicated issue for that discussion.

I think everyone agrees that an 'upgrade the world' command would be really nice to have, but IIUC it's blocked while we wait for the resolver work to land.

Indeed this issue is now properly blocked by #988. Mentioning the issue number for cross-linking the two issues.

I almost forgot...

I'm not sure what you mean about "holding" updates. Please clarify.

Now that this issue is exclusively for upgrade-all, I should clarify.

There might be some situations where it might be useful to prevent an upgrade of a certain package when we run upgrade-all. The specific one I have in mind is... If pkg is installed and I don't want to bother about going re-configuring a potential newer version, so, I want to prevent pip from upgrading that specific package when running 'upgrade the world'. Essentially I'm holding back the upgrade of pkg.

A flag that takes comma-separated list of packages to hold back from an upgrade would be fine.

Adding this once a SAT solver comes along should be easy. It's just a few extra clauses IIUC. (Yes, I'm brushing up on SAT solvers as well)

I don't know if this is something common or something someone wants. But, I don't think this originated in my head though. Someone must have mentioned it in some thread somewhere.

Ah, I see. That makes sense, and seems like a reasonable thing to want.

Quick request: Could someone please add to the description a tiny note that upgrade command won't happen? Possibly if you want, write a tiny summary of why it is so as well.

_{_goes to sleep_}

Two nice features that apt has (and I think other mature system package managers like dnf have similar features):

• Marking a package as "held", which is a persistent flag attached to the package and causes it to be skipped by upgrade-the-world commands. Often set on packages that you had to downgrade or patch locally.
• Tracking of which packages were explicitly requested by the user, versus which ones were only installed implicitly to fulfill dependency constraints. The latter can be removed by an upgrade-the-world if they stop being depended-on.

Both of these can be seen as special cases of the thing that more recent project-environment package managers like npm and cargo do, where they distinguish between some sort of human-oriented "requirements" file versus a fully specified "lock" file. An explicitly held package is like a package that has a human-specified version constraint, and am explicitly installed package is one that's listed in the human-editable file.

Ditto. If we're adding 'upgrade the world', we need to add the ability to mark packages (as held/user-installed and maybe more) to add information to better determine a course of action for the upgrades. I see this as a requirement, not as a nice to have.

I actually like the technique used by Cargo and likes. The use of files and not some form of meta-data-hidden-behind-a-cli-command makes it much more easy to grasp, manage and also possible to create a reproducible environment.

I would actually be happy if I see some form of pyproject.lock...

200th comment. Wow. :smiley:

Adding reference to #654 for "marking" of packages that we talked of.

Can you explain what cargo does? Where would you see the pyproject.lock file being stored on the user's machine?

Rust's Cargo's lockfile gets stored in the project root, to record the version of dependencies currently installed. Committing this file to git allows you to share a consistent set of dependency versions with other developers and CI. PHP's Composer has a similar lockfile, Composer.lock.

I always assumed that pip's 'pip freeze' and 'pip install -r' were meant to do a similar thing, and it was just unfortunate that the lock file format was easily readable/writable by humans and people choose to edit it directly. Was requirements.txt not originally conceived as a lockfile?

@triplepoint Thanks for the explanation. That does indeed sound more like a requirements file. But requirements files are optional, version-based and per project. Marking (as I understand it) should be per environment (virtualenv or system installation) and should simply say "don't auto-upgrade package X" (but allow manual upgrading if the user explicitly requests an upgrade of that package by name).

To help untangle the discussions on upgrade-all and "upgrade behavior", here are comments from @rbtcollins and @ncoghlan in the discussion on the list on the latter about a SAT solver not being required for a first implementation of upgrade-all:

Robert:

I realise the consensus on the ticket is that its blocked, but I don't
actually agree.

Yes, you can't do it _right_ without a full resolver, but you can do
an approximation that would be a lot better than nothing (just narrow
the specifiers given across all requirements). That is actually
reasonable when you're dealing with a presumed-good-set of versions
(which install doesn't deal with).

Nick:

"yum upgrade" has worked well enough for years without a proper SAT solver, and the package set in a typical Linux install is much larger than that in a typical virtual environment (although distro curation does reduce the likelihood of conflicting requirements arising in the first place).

That said, rerunning pip-compile and then doing a pip-sync is already a functional equivalent of an upgrade-all operation (as is destroying and recreating a venv), so I agree there's no need to couple the question of supporting bulk upgrades in baseline pip with changing the behaviour of upgrading named components.

IMO, pip upgrade-all is by far the most important proposal on the table out of all of the various "upgrade functionality" discussions. Having "upgrade all" would give an obvious way to keep your system up to date, making questions about non-eager updates leaving things at older levels much less pressing, as well as filling a gap that currently exists.

While a full solver would be nice, I see no reason why the starting point for pip upgrade-all shouldn't be that it does what pip install -U <list every package that's installed here> does. That's what I'd expect as a user, and for the majority of cases it does exactly what's needed. The complex corner cases around conflicting requirements can be handled in the first instance by reference to the above. If that isn't sufficient, then we can look at either modifying install -U behaviour to address it, or special-casing the update-all command, or even implementing a full solver at that point.

Are you going to implement this in next 10 years?

FWIW, I will be re-visiting this, this weekend.

@magicgoose said:
Are you going to implement this in next 10 years?

@pfmoore said it better than I can:

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

So personally, I'd appreciate it if people would refrain from pinging this issue unless they have working code that at least offers a starting point for the implementation - and they are willing to follow it through to implementation.

entirely personal opinion

i think by the very nature of pip (which is install packages from a repo that does not have global QA like say debian, redhat or ubuntu)
i feel it is necessary and/or acceptable to actually completely refrain from implementing and update-all functionality,

since we pip ever guarantee a known got state of a install-able set of python packages

@RonnyPfannschmidt IMO it's perfectly reasonable for users of pip to explicitly disallow the use of an update-all command, if that fits their requirements / workflow. But not all users of pip have the same strict requirements as these folks. For users with more relaxed needs, an update-all command is useful, and its lack makes it significantly harder for them to keep their systems "up to date". So I do think it's reasonable for pip to provide such a command. It's our job to provide the tools people need, not to enforce particular policies on how users choose to use those tools.

@pfmoore my personal experience with just updating all packages in an environment is, that it breaks things very regular ^^

users that need an relaxed update all sound like you mean normal end users (which should just use a normal linux distribution for example)

Whether upgrade-all is problematic or not depends a _lot_ on how many dependencies you have, and how well disciplined they are about their API maintenance. It can also be used in conjunction with a curated private repository to get control over what upgrades actually happen, rather than having to carefully configure which upgrade commands you run in each virtual environment.

@RonnyPfannschmidt Windows users still outnumber Linux users ~18 to 1, and they don't have anything comparable to a distro package management community to fall back on at this point (while the core technology is there in recent versions, the packaging and curation communities aren't). That means they're a lot more reliant on user level tools like pip and conda to pick up the slack.

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

@pfmoore are you aware that such comments can come across as denigrating the efforts of contributors? That is how it looks to me. I'm not sure if you followed the whole history of this issue, but in this particular case you're very much off-base. The issue is much more with the pip development team than with any contributors.

Rough summary of just a part of it (PR gh-3194):

1. There's a documented decision that an upgrade command is welcome (on the pip mailing list as well as the docs and GitHub).
2. Then a call goes out by a prominent developer (@njsmith in this case) that implementing this feature would be a very valuable.
3. New contributor shows up, implements everything, addresses all reviews comments quickly. PR ready to merge.
4. Core contributor changes his mind about wanting upgrade.
5. Very long discussions follow, without conclusion.
6. Contributor gives up and disappears (most people would, such things are frustrating).

And that was even before @pradyunsg showed up, who is showing remarkable persistence.

In a well-functioning project, the core devs that really care about the issue would organize a quick hangout and make some sort of decision. Or delegate one or two people to work on it enough to get that conclusion. Or at the very least apologize and say thank you to the PR submitter, instead of blaming him for "not following through".

I know you have the best of intentions, but please be a bit more careful with this sort of comment.

I think it's perfectly reasonable for requirements and opinions to change through discussion, particularly for a thorny issue like this one where there isn't a right answer. Part of getting a patch landed in any code base is keeping up with changes that get hashed out as part of review and discussion around any change. Some changes are fairly minimal and have less of them, some have more. There's nothing wrong with a person deciding they don't want to deal with that and dropping out (every bar to adding a change will cause some amount of drop off, including tests, documentation, etc).

This particular change is particularly nasty because it's changing the default behavior of the primary usage of pip. That's hard and scary and it would be a disservice to our users if we rushed it and didn't full hash things out prior to committing to one direction or the other. This command is used 10s to 100s of millions of times a month. This is not a small, easy change and it won't be the folks pinging this issue who have to deal with the angry backlash that comes from making any change.

The person who implemented the PR before, their time is appreciated, but it is a fact of the matter they didn't follow through till the end. As core devs here _our_ time is limited, we're either volunteers or spread amongst many projects and we float in and out of participation. There are a ton of different issues that all require attention and this is just one of them. Paul simply stated that pinging this issue isn't helpful (which it isn't) and that if someone wants it they'll need to either wait for someone (including one of the core devs) to decide to put in the considerable effort to change the default behavior of millions or do that themselves.

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

@pfmoore are you aware that such comments can come across as denigrating the efforts of contributors? That is how it looks to me. I'm not sure if you followed the whole history of this issue, but in this particular case you're very much off-base.

@rgommers Seriously? That comment was from months ago, and was quoted out of context here (but frankly, I have no problem with @pradyunsg quoting it in response to the unhelpful and sarcastic comment he was replying to). I'd suggest that if you'd reviewed the whole history, you would have seen my comment in context, at which point hopefully you would have understood what I was trying to say.

If I was that "off-base", then you could have said so in May at the time I said it, rather than picking on it now, out of context.

If I offended anyone, I apologise, that wasn't my intent. Honestly, I'm as frustrated as anyone else that this issue is proving so difficult to reach a design that is acceptable to everyone. In my comments, I'm trying to strike a balance - on the one hand, I genuinely appreciate the contributions people make, but on the other hand, I feel it's important to make it clear to people that on an issue like this, the coding is frankly the least of the work needed. Very often, contributors don't appreciate this fact, and that's where we get incomplete PRs, where people get frustrated with the work needed to persuade people that their design is OK, and/or to rework the change, possibly in ways they don't really like, to take into account other people's (often conflicting and inconsistent!) views. I'd rather they came into the process with an understanding of what's involved, rather than coming in with unrealistic expectations and as a result having a bad experience.

Everyone involved in the discussions on this issue has put in a _lot_ of time debating the pros and cons of various approaches. I don't imagine there's anyone who's happy over how long it's taking. Personally, the lack of an upgrade-all command (just one part of the change, and not likely to be the one that gets implemented first) hits me on a regular basis. We do occasionally (honestly, it's a lot more than just "occasionally") get people (usually people who haven't actually contributed to the discussion or the code) commenting "this is important, why haven't you implemented it yet?" Frankly, it's hard to remain calm and _not_ snap at such people.

In a well-functioning project, the core devs that really care about the issue

You do realise that this comment is open to being interpreted as saying that the pip developers don't care about fixing this (and by implication about pip)? We're all at risk of wording things in a way that can offend people. I'm sure you didn't mean this as a criticism of the pip devs, please assume I'm equally not trying to offend anyone.

would organize a quick hangout and make some sort of decision.

I'd be surprised if something like that would work here, but I'm open to trying it. Not sure who would get involved or how we'd manage it, but sure, if it helps and someone wants to try that approach. I would say that, as this change has such a large potential to affect pip users, any decision made on a private channel like this should probably be written up as a proposal and published for general comments - and I suspect that doing so would simply trigger yet another round of the same debates we've been having.

Or delegate one or two people to work on it enough to get that conclusion.

So you're saying that a decision this big should be made unilaterally by a couple of people? Maybe that's the only way we're going to get a resolution, but it's not really how decisions are made on pip (unlike Python, we don't have a BDFL with executive decision making authority). You can claim that makes us not a "well-functioning project" if you want, that's not my view but we can disagree on that if you want.

Or at the very least apologize and say thank you to the PR submitter,

I'm not sure what we should be apologising for, but if it helps then I'll happily apologise - for the fact that no-one gave him a clear understanding of what a difficult job it would be getting this proposal through to completion, or helped him to manage the debate and guide the participants to a consensus. But to be fair, I don't think anyone _else_ involved knew at the time that this would be the case, so I think it's mainly hindsight talking here.

He certainly has my thanks. It's rather easy to say "that goes without saying", but it shouldn't - open source projects don't thank contributors enough, and that's a problem. While I'm on the subject, I'd like to thank _everyone_ who's contributed to this debate - and I'm completely serious here - as I know from experience how draining it can be. But especially to @pradyunsg for being the current victim of all the endless discussions and changes of direction. Thanks for not giving up! (Yet!!)

instead of blaming him for "not following through".

Well, I don't think there's any blame attached (although it's hard to know whether he felt blamed, as he's not around any more). But it's true that his original PR didn't get managed through to completion. That's just a fact, though. I hope no-one is suggesting that contributors are entitled to have their PRs accepted _simply because they submitted them_. Not all PRs are acceptable when first submitted, they need to be revised and updated, and sometimes even after all the work they are _still_ not acceptable. Sorry, but that's life.

[If I seem harsh in the above statement, I apologise (again!). But a lot of my leisure time is spent reading and dealing with complaints that I (or projects I'm involved with) somehow haven't done enough. And it's a vicious cycle - I lose motivation to work on open source in my spare time due to the nagging, which of course means even less gets done. While I try to remain polite, sometimes it's not easy]

I don't plan on saying anything further on this meta-discussion about how the PR is being managed. I've spent an hour this evening working on this response, to try to avoid saying anything that might offend someone (and I bet I failed :-(). And I could have much better spent that time - with my family, relaxing, or doing something more productive.

So can I suggest we get back to trying to help @pradyunsg come up with a PR that we can all be happy with, and leave off the fruitless meta-discussions?

So can I suggest we get back to trying to help @pradyunsg come up with a PR that we can all be happy with, and leave off the fruitless meta-discussions?

Yes please. :innocent:

Oh, I forgot.

@rgommers said:
And that was even before @pradyunsg showed up, who is showing remarkable persistence.

I'll take this as a complement... Thanks.

@pfmoore said:
especially to @pradyunsg for being the current victim of all the endless discussions and changes of direction. Thanks for not giving up!

You're welcome.

(Yet!!)

I really hope it doesn't get to that state. It'll be a really bad state of affairs if it does, IMO. (I'm arrogant that way)

I wrote the following while I was reviewing the history and figured might as well put it up here, for future reference and letting others correct me just in case.

• Decided to work on this after realizing how the big non-technical:technical ratio would to be (It's much bigger than I _really_ conservatively thought) and realizing that there has already been an unsuccessful attempt at this.
• Did a write-up on the state of affairs, because I was bored and I needed to know what's happened anyway.
  
  
  
  • Probably spent more time (and space here?) on it than needed but at least I got a good overview of the issue for myself (and everyone else?).
  
  
• Got over-excited after writing it. :smiley: Showed it to the world!
• Initiated a (long!!!) discussion on implementing an upgrade command.
  
  
  
  • A few useful off-shoot ideas came up in the discussions. New issues were created for the same.
  
  
• Discussion leads to the idea of changing install behaviour to upstall, which would do non-eager upgrades by default and removed a part of the functionality provided by --target option - 3 things.
  
  
  
  • This is where we made a mistake - bundling up the 3 (fairly) independent changes and which would be implemented as one, because no one realized this part.
  
  
• I implemented the same. Since the 3 changes were bundled up, they all got stuck when there was no consensus on changing install behaviour to upstall, the potentially controversial change.
• The lack of consensus initiated a long discussion which came to the point where people were backing out of the discussion and I guess almost pushed to burnout.
  
  
  
  • Some of the earlier assumptions were broken and it was decided that we would do a minimal disruption change first.
  
  
• I ran out of free time to work on this so I created a few new issues so that someone else can work on them independently and ideally not make the same mistake as we did here.
• Stalled.
• I'm back! I'll try to get the switch to non-eager upgrades by default done by Sept 25.

Yes, let's just focus on fixing pip install --upgrade and leave everything else out for now. That's a requirement for any other work anyways.

@pfmoore @dstufft thanks for the thoughtful replies. I didn't mean to offend anyone, so apologies if it came across that way.

I will not respond to everything, because no one is looking for a long discussion here.

you could have said so in May at the time I said it,

I was away from Github for 2 months then, but it did bother me when I saw that comment first time around.

So you're saying that a decision this big should be made unilaterally by a couple of people?

All options on the table are way better than the status quo. And after 5.5 years and many hundreds of comments spread over multiple issues/PRs here and the mailing list, I'm not confident that yet more comments are going to resolve it. I hope it will get resolved, but if this stalls again then definitely yes - nominate one or a couple of people and just make a choice.

I'm not sure what we should be apologising for, but if it helps then I'll happily apologise - for the fact that no-one gave him a clear understanding of what a difficult job it would be getting this proposal through to completion, or helped him to manage the debate and guide the participants to a consensus.

I meant the latter. Sometimes I change my mind on a decision for one of my projects, that happens. Then I feel responsible for making the new direction clear. And apologize if I don't have the time to deal with the consequences of that change (takes only 30 seconds....).

But it's true that his original PR didn't get managed through to completion. That's just a fact, though.

That's a view, not a fact. In my view, the PR was complete - all that was left to do was either hit the green button or reject it. He did all I would expect from a contributor.

I realized after your reply that the expectations the pip devs have from contributors and core devs are very different from pretty much any other project I'm familiar with. I expect core devs to guide new contributors, encourage them, give them feedback where needed, and help them resolve contentious issues (which most contributors neither have the skills for nor the interest in, and those that do often end up becoming core devs) if needed. You're saying to new contributors: _"you have to manage us. we may change our minds, disagree with each other, or lose interest - it's your job to manage that"_. Maybe that's the nature of this project and it has to be this way, I don't know.

While I'm on the subject, I'd like to thank everyone who's contributed to this debate

Agreed. Thanks to everyone who contributed.

• and I'm completely serious here - as I know from experience how draining it can be.

It is draining. Personally, I stick around here and on distutils-sig because it's important for the Python ecosystem and the users of my packages, but both places don't exactly give me positive energy.

Just in case it slipped under the radar - #3972 :smile:

You're saying to new contributors: "you have to manage us. we may change our minds, disagree with each other, or lose interest - it's your job to manage that". Maybe that's the nature of this project and it has to be this way, I don't know.

I said I wouldn't continue this, but this point struck home. That's not how I feel about our approach, but now that you put it that way I do see that it can come across that way. To be honest "we may change our minds, disagree with each other, or lose interest" is true - after all we're all just people with other commitments as well - but I don't consider it as something a new contributor has to "manage". Rather, it's just a reality of dealing with people, but if making too much of it comes across as dumping the problem on new contributors, that's wrong.

Thanks for pointing this out - I'll try to keep it in mind and avoid giving that impression in future.

I think a lot of the problem really boils down to the fact that we're vastly undermanned which ends up making it hard to keep up and follow up on everything. There are more potential contributors then there are us so it is easy to feel overwhelmed when there are multiple folks all trying to make changes at once. Bigger changes, or changes where there is no clear cut consensus, tend to be the hardest ones to deal with so they are the ones that tend to suffer the most :(

The sad state of affairs is that while I think we would all like to be here to help guide through every contributors through the process, we simply don't have the man power. This tends to have a bit of a viscous cycle too, because we don't have the man power to do that, we don't readily find new people who seem to have really started groking the mentality behind how pip operates and who have learned enough (because we're not there to teach them) to give them commit rights to pip. This means we're constantly undermanned and struggling just to keep our heads above water (At least, that's how I feel. Constant 70-90 hour weeks is real hard on a person :/).

@pradyunsg Reviewing #3972 is on my TODO list, just haven't hit it yet.

@pradyunsg Reviewing #3972 is on my TODO list, just haven't hit it yet.

Thanks!

This means we're constantly undermanned and struggling just to keep our heads above water (At least, that's how I feel. Constant 70-90 hour weeks is real hard on a person :/).

That is hard. Numpy and Scipy were in that situation when I started working on those. Not fun. I appreciate all that you're doing.

This issue is now really long and really old, a lot of things have been discussed, too many things have happened and this issue has almost attained a low signal/noise. It's hard to see what all has happened.

FWIW, the reason that upgrade was on the cards was because of the fact that install --upgrade has a broken default. Since we are now one step closer to fixing that, I guess it's best we have a new issue for that.

I suggest that this issue be closed because of the above and new issues be created for whatever here is seen as unresolved. I can see 2 things:

• Shift the default upgrade strategy to only-if-needed.
• Add "upgrade the world" functionality that depends on #988.

On 9/15/16, Nick Coghlan notifications@github.com wrote:

@RonnyPfannschmidt Windows users still outnumber Linux users ~18 to 1, and
they don't have anything comparable to a distro package management community
to fall back on at this point (while the core technology is there in recent
versions, the packaging and curation communities aren't). That means they're
a lot more reliant on user level tools like pip and conda to pick up the
slack.

Isn't then
conda update --all
good enough?

@Liso77
Well, No. It is not.

Conda and pip are tools with different goals. This is a great read on this topic. The 3rd point is most relevant.

If the discussion for upgrade-all surfaces again (hopefully in a new issue) my vote is for spelling it as follows:

pip install --upgrade :all:

pip install --upgrade :all: is extremely weird. Let's stick to POSIX semantics, which basically everyone is doing nowadays: pip install --upgrade --all

What about just pip install --upgrade without any package names or specifiers? To easy to run accidentally?

pip-tools does it like this for pip-compile -P.

Maybe we should bike shed once we have some sort of working
implementation... :)

On Sun, Feb 12, 2017, 20:55 FichteFoll notifications@github.com wrote:

What about just pip install --upgrade without any package names or
specifiers? To easy to run accidentally?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/59#issuecomment-279225595, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADH7SfnSBflH8rK3nFLw1hvYBaovjbcGks5rbyRUgaJpZM4AJ4Py
.

How about a version of pip list --outdated that produces its list in a format that can be directly (i.e., without sed, cut, etc.) ingested by pip install --upgrade (e.g., pip list --outdated --format install | xargs pip install --upgrade or something similar with backticks)?

whatever syntax will be used the most important thing is introducing this command, It's unbelievable that is still missing

in meanwhile I suggest you to try
https://github.com/jgonggrijp/pip-review
with pip-review --local --interactive ask you package by package if you want to update, not very good but better than nothing

@SMH17 It's entirely believable that it's still missing, as there are exactly zero commercial Python vendors formally providing funded development time to work on Python packaging usability improvements on behalf of their customers.

So if you'd like to see the situation improve, it's likely that the most helpful thing you can personally do is to either encourage your Python support vendor to invest developer time in improving the tools you use, or if you don't have a support vendor yet, encourage your employer to pay one.

As an additional piece of context regarding the lack of urgency around this issue, it's worth keeping in mind that the general recommendations are to

• keep your working environment definitions under source control to improve reproducibility on other systems (using something like https://github.com/jazzband/pip-tools or https://github.com/kennethreitz/pipenv to keep those definitions up to date)
• aim to routinely upgrade to new releases of dependencies in order to minimise exposure windows to unknown or undisclosed security vulnerabilities

That doesn't mean the commands proposed here aren't useful, they're just markedly less valuable if the current working environment is already being maintained through pip-compile + pip-sync, or pipenv lock + pipenv install.

It would be valuable to update the original description as I guess there have been some changes made to pip install since the update made by @qwcode.

Hi everyone.

I have broken some of my python package dependencies because the following command:

pip install --upgrade packageName has upgraded packages recursively.

Why not change the default behaviour of the --upgrade option, that is to uninstall and reinstall ONLY the given package from the command line ?

How about this ?

@sebma I think that the default behaviour should not be changed. Maybe you could try using the -no-dependencies flag the next time. It should work :+1:

@sebma, @aaossa, I'll have you both know that it has pretty much been decided already that the default upgrade strategy will change in the future (ref: https://github.com/pypa/pip/issues/3871#issuecomment-247789343). The necessary feature (i.e. the --upgrade-strategy argument) has been added in https://github.com/pypa/pip/pull/3972.

As @pradyunsg mentioned earlier, this issue is kind of a left-over. The first part is sort of handled by now (see my first paragraph) and the second part is the only reason why this package is still open, it seems. I don't know if a separate "upgrade-all" issue has been created since.

I've released a nice interactive upgrader for requirements file: https://github.com/simion/pip-upgrader

Shift the default upgrade strategy to only-if-needed.

4500 did this.

Add "upgrade the world" functionality that depends on #988.

4551 for discussion on this.

Addressing the points made in the current top-post:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here.

It has been decided against adding an upgrade command or making pip install upgrade already installed packages. pip now does have non-recursive upgrades by default, with the recursive behavior available behind --upgrade-strategy eager.

pip upgrade-all would upgrade all installed packages.

4551 exists and it would be nice to have a fresh discussion on this; when #988 is done.

@dstufft @xavfernandez @pfmoore Do any of you think this issue should be closed?

_{Edit (05-18-2017): Punctuation + minor text added}

Seems reasonable.

Hello all,
I made a simple script/gist that does the job.

https://gist.github.com/serafeimgr/b4ca5d0de63950cc5349d4802d22f3f0

Why not simply do this?

pip install --upgrade $(pip list --outdated | awk '{print $1}' | tr '\n' ' ')

Because it's not that easy in reality since you may install versions that don't satisfy some of your other packages' dependencies.

Based on and with thanks to @serafeimgr's gist, I've written a possibly-useful command-line tool, pip_upgrade_outdated; source on github. Feedback welcome.

(See also this issue: Yes, the parallel execution is particularly dangerous, and, yes, this may break things. Nonetheless many people run something like this by hand all the time, so might find it useful.)

Thank you for taking the time to build a complete solution.
Even though my recommendation would be to find a way to push this feature to pip.

I think pipenv & pipfile is going to replace pip/requirements.txt anyway.
Maybe @kennethreitz knows more about the roadmap and the --upgrade all feature.

@qoheniac | tr ... is redundant.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.