Pip: New Resolver: Rollout, Feedback Loops and Development Flow

I don't know how you have it planned out, but one comment is that I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

One reason is that if you're sharing more code, there will be less chance of breakage when you're toggling the new behavior off and on, because you'll be exercising that shared code in both states and you won't have as many potential differences in behavior to contend with.

This would involve CTAs to users for asking them to try it out and provide feedback

Our track record on getting advance feedback on new features has been pretty bad. We've tried beta releases, releasing new features with "opt out" flags that people can use if they hit issues, big publicity drives for breaking changes, and none of them seem to have worked.

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

I wonder - could we get a PSF grant to get resources to either do a big "real world" testing exercise for this feature, or (better still) develop a testing infrastructure for us? Such a project could include a call for projects to let us know their workflows and configurations, so that we can set up testing paths that ensure that new pip versions don't break them. Or even just use a grant to get someone experienced in the communications aspect of getting beta testers for new features to help us set up a better user testing programme?

In terms of git/GitHub, this is probably the first "experimental" feature implementation within pip

I'm not 100% sure what you mean by that. We've certainly had new features in the past that have been added while the "old way" was still present. We've not tended to leave them "off by default, enable to try them out", if that's what you mean, but that's mostly because we've never found any good way to get feedback (see above).

I spent ~60 minutes (re-re-re-re-re-)writing this one post, so now I will go take a look at places in New York! If you don't see an quick response from me, it's because I'll be in tourist mode.

I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it. A bunch of the code will have to be :fire: and I agree that the rest should definitely be reused as much as reasonable.

We've not tended to leave them "off by default, enable to try them out", if that's what you mean

Yep, indeed. I'm also hinting at development flow here -- IMO it would be okay to merge empty infrastructure (classes with a bunch of methods that are just raise NotImplementedError() that will get fleshed out in subsequent PRs) or one doesn't cover all the cases (half-baked implementation) into the master branch as long as that's only used behind the flag that is explicitly noted as "experimental/alpha".

re: feedback

I'm young, dumb and optimistic -- I want to make this rollout an opt-in, to get proactive feedback and act on it. By "proactive", I mean from folks who are willing to take out some extra time to try out alpha/beta functionality and inform us about how it is. I think if we make make enough noise and strategically target/reach out to people, we can get good "proactive" feedback from folks who have the time and energy to try out new functionality to help iron out the details/issues.

Looking at our recent "major" changes, I think most of the feedback we received was reactive -- from users realizing issues with their workflows when it broke, and then reaching out to inform us about it. A lot of them may not have had the time to help iron out the details of the new functionality, which causes a lot of friction. These also cost us a lot of our "churn budget" [1], which I don't want to spend more of, since Python Packaging doesn't really have much left anyway [2].

FWIW, I plan to borrow some ideas from the PyPI launch, like making blog posts at fairly visible locations (i.e. not my personal blog), possibly going on podcasts, well-timed actionable emails etc. I'm also looking for more prior art/avenues to communicate via. One of the (many!) things I learnt at PyCon, was that there are channels that we don't use, that will help spread information but won't seek out to check if we have any to spread.

To be clear, I'm not criticizing against the rollout approach we took for PEP 517, I think it's going well, especially given the fact that we're all volunteers. I'm trying to see what we can learn and actionable items to try to avoid the problems we had. Most of these items do involve more work from the maintainers and the main reason I am even spending all this time thinking about this, is because I view this as a fun learning exercise of how to do change management.

re: grants

Yep, I think we can definitely use a grant/more experienced person to help us figure out the communication, roll-outs and testing infrastucture. That does however need someone to do the grant-writing work and figuring out more concrete plans than I can make right now, since I don't have a more stable number of hours / week that I can guarantee.

FWIW, PSF has an ongoing contract to help figure out PyPA/Packaging-related communication with Changeset Consulting, so maybe we can leverage that?

I'm intentionally not @-mentioning people since this is fairly early in the planning state to add more people in the conversation.

Footnotes:

1. A really nice term that @ pganssle used that I'm definitely going to use.
2. This is why I've put #3164 on the back burner, despite having an implementation of the "pip-cli" package proposed there and having reasonable consensus on how we want the rollout to look like.

I'm young, dumb and optimistic

:-) And I'm sometimes too old, weary and cynical. Let's go with your philosophy, it sounds much better :-)

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it.

Great!

From IRC just now:

[sumanah] pradyunsg: is there anything we the pip & packaging community can do to help you get more work done faster on the resolver?
....
[pradyunsg] Actually, right now, inputs on https://github.com/pypa/pip/issues/6536 would probably help me figure out how to approach the work / get feedback from people etc.
....
[sumanah] pradyunsg: re: New Resolver: Rollout, Feedback Loops and Development Flow #6536 -- the input you want is something like: is the feature flag approach a good idea? is it a good idea to get feedback via some mechanism other than the pip GitHub issues? is it a good idea to get a grant or similar to get realworld manual testing & robust testing infrastructure built, and/or proactive comms?
...
[pradyunsg] Yep -- whether the ideas I'm suggesting are good. Also any additional ideas/approaches/thoughts that might help the rollout + feedback be smoother would be awesome.

So:

Is the feature flag approach a good idea? Yes.

Is it a good idea to get feedback via some mechanism other than the pip GitHub issues? Yes. We should find automated ways to accept less structured bug reports from less expert users.

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

Could Changeset (me), under the existing contract with PSF to help with PyPA coordination/communications, help pip with proactive communications to get us more systematic realworld manual testing? Assuming that I have hours remaining in my contract by the time we want to start this rollout, yes.

is it a good idea to get a grant or similar to get more help with user experience, communications/publicity, and testing? Yes. The PSF grants would potentially be of interest, as would NLNet grants (for requests under 30,000 euros), potentially the Chan Zuckerberg essential open source software for science grant, and Mozilla's MOSS. The Packaging WG can be the applicant of record. If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod, I can start investigating those possibilities with the WG.

If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod,

It definitely sounds interesting to me :-)

@pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod

_nods_ yeah that sounds interesting

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

@brainwane Also relevant here is https://github.com/pypa/integration-test. I think getting this set up, is another potential area for funding -- we should add this to https://wiki.python.org/psf/Fundable%20Packaging%20Improvements.

OK! I've started talking with the PSF and with the Chan Zuckerberg Initiative folks about applying for a CZI grant via the Packaging Working Group. I've added some details to the Fundable Packaging Improvements page about why the new pip resolver's important, and added the integration-test project to that list. And I've started gathering names of user experience experts who have the capacity to research our complicated all-on-the-command-line package distribution/installation toolchain, talk with users to understand their mental model of what's happening and what ought to happen, and advise maintainers.

If we get money via grants from MOSS, CZI, or NLNET, I think we'd get the money ... October at the earliest, probably. A grant directly from the PSF would be faster probably but "Our current focus is Python workshops, conferences (esp. for financial aid), and Python diversity/inclusivity efforts."

One consideration is that I know Brett & the folks over on the steering council are talking about investing in project management and looking into having some sort of paid resources for managing these projects (triage, project management, etc) and they are talking with the PSF directly. It may be worth reaching out and finding out what they are doing or thinking, since I heard some talks of long term sustainability and it'd be a good thing to be involved in those.

Feature flags are good, opt-ins are good. One thing you might consider is whether you could randomly prompt users to try out the resolver (like, very very very infrequently and only for an install at a time, i.e. not forcing them to turn it on permanently). Then you could indicate how the resolver was helpful (e.g. what did it do for them? what conflicts did it encounter and resolve?)

People coming from javascript or rust for example will also expect a lockfile of some kind, so that may be something to consider...

Sorry to jump in, glad to see this moving ahead!

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

As one of the people that got bit by some PEP 517 issues for this very reason, I'd actually love to see an opt-in way of testing things out. But I only know about this kinda stuff because i subscribed to all the python packaging news sources I could after the --no-use-pep517 flag issue. What I'm saying is that spreading this kind of news is hard and is probably why feedback is hard to get.

I think more people would be interested in this if the information could be disseminated better. Is that what the resources you are seeking would allow for?

To continue on what jriddy is saying, I also feel it'll be really hard to get people to test various feature flags if they have to know about them, make changes to their CI setup for each new flag, etc.

What would seem much more doable, however, is if there is only _one_ feature flag to know about, to test "what's coming up next" in terms of changes that need to be tested. Then people and companies could setup their CI to run that also (without failing builds for errors). I'm thinking of something similar to Rust, where these kinds of changes bake in the "beta" channel of the toolchain, and it's easy to setup another CI channel to run things on the beta toolchain, and send errors to someone.

The key thing is, this setup needs to be learned about and done _only once_, instead of having to continuously learn about new individual feature flags, modify CI setups or test them manually.

What would seem much more doable, however, is if there is only _one_ feature flag to know about,

In a sense, doesn't this already exist in the form of --pre? Could the beta release channel for pip just be a matter of running pip install --upgrade --pre pip?

Sorry to jump in, glad to see this moving ahead!

@techalchemy please, of all people, _you_ definitely don't have to be sorry for pitching in this discussion.

Is that what the resources you are seeking would allow for?

To an extent, yes.

reg: beta releases/"channel" for pip

Thanks for chiming in @jriddy and @chrish42. While I think that generally that's definitely an useful/important conversation to have, I also feel it's slightly OT for this issue. None the less, I'll respond here once; if we want to discuss this more, let's open a new issue.

We've tried that in the past -- most recently with pip 10 -- but it hasn't worked out well. I am slightly skeptical of how well that might work going forward too, but I can also imagine that some changes to our process might result in this working smoothly for us. Maybe we could do a "beta only" set of features or something? I'd imagined -X all as a syntax for that in #5727. Maybe we could pick that up as a part of this rollout plan? Idk. We'll need to invest time and energy to figure this out. :)

As mentioned in https://github.com/pypa/packaging-problems/issues/25#issuecomment-520167480, I think it's important to have a consolidated explanation of how a solver changes the pip experience. Lots of people will be frustrated by the shift to a more rigid system (even though things should be more reliable overall, they'll get blocked in places where they are not currently getting blocked.

Having a central explanation of how things have changed and why it is a good change will make responding to those angry people much simpler. Post a link and see if they have any further questions.

The prerelease is a good idea. In conda, we have a prerelease channel, conda-canary. We encourage people to set up a CI job to run against canary in a way that helps them see if conda changes are going to break them. Ideally they let us know before we release that version. That channel has been a pretty dismal failure. The only time people really seem to use it is when they want to get the newest release to fix some bug that they are struggling with. We do not get many reports from our intended early adopters. I still think the prerelease is a good idea, because when a release goes poorly and people are angry with you for breaking their 700 managed nodes, you can say "well, it was available for a week before we released it. Why aren't you testing these things before you roll them out to 700 nodes?" You are giving people an opportunity to make things work better. Help them realize that passing on that opportunity means more pain for them down the line. It's worthwhile investment for them, and if they do it as part of their CI, it costs them no time aside from setup.

Regarding the flag: I think it's better to have a config option (perhaps in addition to a flag). I would not want to pass a flag all the time. I'm not sure if pip has this ability - maybe you tell people who want a more permanent switch to use the corresponding env var?

Regarding the flag:

pip's CLI options, automatically get mapped to a configuration file option and an environment variable, with the appropriate names.

@msarahan Thanks for chiming in, much appreciated! :)

Regarding the "let me do what I want" option to ignore broken dependencies, I think it would be desirable to structure the feature flag such that it can also serve as the opt out after the resolver gets turned on by default (for example, start with --avoid-conflicts as an opt-in, eventually move to --no-avoid-conflicts as an opt-out, but accept both options from the start)

You'll also want to consider how --ignore-installed interacts with the solver - when it is passed, you should probably ignore all the requirements for already installed packages.

Beyond that, handling things as smaller refactoring patches to make integration of the resolver easier is an excellent way to go (that's the approach that made the new configuration API for CPython possible: a lot of private refactoring that was eventually stable enough to make public)

@ncoghlan What does "opting out" of the resolver mean? Completely avoiding dependency resolution (and hence the resolver) is --no-deps. I understand that there's a need for an "ignore version conflicts on this package" or something along those lines.

Personally, I don't see any point in keeping the "keep first seen" resolution logic for longer than a transition period to a new resolver.

However, if there are use cases that these two options would not cover, I'd really like to know about them. :)

More broadly, if there are workflows that have issues with a strict resolver's behavior, I'm curious to know what those look like, as early as possible, to be able to figure out whether/how to support them.

Personally, I don't see any point in keeping the "keep first seen" resolution logic for longer than a transition period to a new resolver.

IDK, I use this "feature" to do some pretty crazy stuff with builds, like...

# install just the packages I've built specifically
pip install --no-index --no-deps --find-links=/path/to/my/local/build/cache -r local-reqs.txt

# ...snip to later in a dockerfile, etc...

# install the deps from public PyPI
pip install -r local-reqs.txt

In this case i'm asking it to resolve my dependencies after i've installed some very pre-determined packages from a local wheelhouse. I suppose i could read my exact versions into that local-reqs file to make a resolver happy, but i've actually found the current behavior of pip quite useful in allowing for these kinds of arbitrary build injections steps. Could be a case of the spacebar heating workflow though, I'll admit.

But maybe the "naive resolution" behavior still has a use.

I agree with @pradyunsg. I don't think it's viable to maintain the existing code and a new resolver indefinitely. Certainly as a pip maintainer I have no interest in doing that.

From an end user POV, I accept that there could well be weird scenarios where the new resolver might not do the right thing. And having an emergency "give me back the old behaviour" flag is an important transition mechanism (although it's arguable whether "temporarily roll back to the previous version of pip" isn't just as good - even though things like common use of CI that automatically uses the latest pip make advocating that option problematic). But long term, why would we need to retain the current behaviour? I can imagine the following main situations:

1. Resolver bug. Obvious possibility, easy fix - correct the bug in the next release of pip.
2. Cases where the old resolver is wrong (generates results that fail to satisfy the constraints). We don't intend to support that going forward, surely? (At least not via anything less extreme than the user pinning what they want and using --no-deps to switch off the resolver).
3. Cases where the old and new resolvers give different results, both of which satisfy the given constraints. Users can add constraints to force the old result (if they can't, that puts us back into (2)). We should give them time to do so, but then drop the old resolver, just like any other deprecated functionality.
4. An edge case that we consider too complex/weird to support. This is like (3), but where we aren't asserting that the new resolver gives the "right" result. Users can still modify constraints to avoid the weird case, or pin and use --no-deps. But ultimately, we're saying "don't do that", and if users ignore that message, then again at some point we remove the old resolver saying "we warned you".

Are there any others that I've missed? In particular any where deprecating and then removing the old resolver isn't possible?

By the way, where's the best place to post "here's an edge case I thought of" scenarios, so that they don't get lost? I think it would be useful to collect as many weird situations as we can in advance, if only so we can get an early start on writing test cases :-)

PS We should probably also as part of prep work for the new resolver, survey what the "typical" constraint problems are (based on what's on PyPI). For my own part, it's pretty rare that I have anything more complex than "pip install ". It would be a shame to get so bogged down in the complex cases that we lose sight of the vast majority of simpler ones.

1. resolver is too slow (see conda). If I have to choose between a 20 min plus resolver, or the current behavior, often I want the current behavior (or at least try; in many cases it will happen to give a result that's fine).

2. metadata wrong. not as much of a problem today, but it's easy to imagine cases that should be solvable but aren't. PyPI metadata is in worse shape than conda/conda-forge metadata, and it's already a problem for conda. if it's wrong and as a user I can't get a solution, I'll want to get some opt-out.

@rgommers For 6, the "ignore version conflicts on this package" style option could work, right?

Thanks, @rgommers - those are good points.

Resolver is too slow, I'd count as a resolver bug. If it can't give sufficiently performant results on simple cases, that's not fit for purpose in my view. If on the other hand you have a massively complex constraint network that takes longer with a full resolver (I hope 20 minutes is an exaggeration, I don't consider that acceptable under any circumstances!), then we're getting into "things we consider too complex to support" territory. Put it another way, has anyone tried asking conda to provide a "quick and dirty" inaccurate but fast resolver? If they won't do that (and I'm pretty sure they wouldn't) then why is it reasonable to expect pip to?

Bad metadata is definitely something I'd consider as "we wouldn't support that" (remember, I'm talking about "after a deprecation period" here!). Giving users time to get metadata fixed, and providing an "ignore version conflicts on package X" escape clause option is IMO sufficient, we shouldn't be expected to retain all of the old machinery just because some people won't fix their metadata.

But yes, publicising the fact that we need good metadata because an accurate resolver follows the "garbage in, garbage out" rule, and monitoring how people respond to that message, is very much part of the rollout process.

i'm asking it to resolve my dependencies after i've installed some very pre-determined packages from a local wheelhouse.

@jriddy The resolution strategy of "use the existing install if compatible" will work for this.

where's the best place to post "here's an edge case I thought of" scenarios, so that they don't get lost?

https://github.com/pradyunsg/zazo/

For 6, the "ignore version conflicts on this package" style option could work, right?

yes, that sounds like the right option

(I hope 20 minutes is an exaggeration, I don't consider that acceptable under any circumstances!), then we're getting into "things we consider too complex to support" territory. Put it another way, has anyone tried asking conda to provide a "quick and dirty" inaccurate but fast resolver? If they won't do that (and I'm pretty sure they wouldn't) then why is it reasonable to expect pip to?

There's a lot of people who are complaining about conda performance, and they're listening - but it's a lot of work, see their recent blog posts. I don't know if there will be a quick and dirty conda option. However, there's related solutions (hacks) like conda-metachannel which lets you prune the dependency graph (manually include/exclude packages) to get a solution faster. So I think that's along the same lines.

However, keep in mind that this will _definitely_ be needed unless you either:

• do a much better job than conda straight away (not too likely, it's not like those guys don't know what they're doing - it's just a really hairy problem).
• have only smaller problems to solve (hopefully not true, PyPI is large and with good metadata the problems should be large)

Bad metadata is definitely something I'd consider as "we wouldn't support that"

Fair enough. The whole stance on "we don't enforce correct metadata" is not helping here though. Unless that changed recently? (and I know, it's a PyPI thing not a pip thing - but it's related).

I'm not sure you really have the options that you think you do.

What does a quick and dirty resolver do differently from an accurate one? What does it drop to be faster? Constraints are constraints - they do not come in grades. You either satisfy them or you don't. Perhaps you can satisfy them only by name to start, then by version, etc.

When people are upset with Conda being slow, it is essentially always because of bad metadata. Keep in mind that you will get blamed for any perceived slowness, regardless of the root cause. Bad metadata is sometimes an incorrect constraint, but more often it is the lack of a constraint that allows consideration of a much older, undesirable option. Conda improved massively recently by doing one small thing: removing our older collection of software that had inadequate (mostly too open) constraints. These open constraints caused conda to explore really bad solutions requiring a lot of downgrades. This is where solving took literally hours. Downgrades are very, very expensive operations because of the way that they can cascade, with each step growing less constrained and more expensive.

The problem with "garbage in, garbage out" as a mentality is that as the maintainers of the solver, you are holding the bag. Unless you have very good heuristics for what is garbage, you are powerless. You end up being the one who has to investigate why the solver is slow, isolate the problem package, and then ask the source of the problem to fix things. It's not a great position to be in, trust me. We spend a ton of time trying to explain to people why conda is taking forever or won't work with some mixture of conda-forge, bioconda, or other 3rd party channels. We end up having to do the detective work and tell those 3rd party channels what they need to fix. It sucks.

Any comparison with conda needs to consider that conda is taking a very different approach to the problem. Conda has one giant source of metadata and solves for everything at once, but pip is recursively solving and optimizing each thing at a time (backtracking as necessary). That may afford you different optimization paths.

@wolfv recently explored using libsolv for conda. He was ultimately frustrated that he could not get it to give the same answers as conda. It boiled down to this difference between approaches. Libsolv is a backtracking solver. It may serve as an optional compiled add-on to pip to speed things up, though I know that you are sensitive to not including compiled code directly with pip.

(@pradyunsg just posted his August update on his blog.)

Some of the questions and needs people are bringing up now are things we need to start gathering now, such as test cases.

But also: what timeline are we thinking is realistic for this rollout? This depends a lot on Pradyun's health and free time, and code review availability from other pip maintainers, and whether we get some grants we're applying for, but I think the sequence is something like:

• build logic refactor: in progress, done sometime December-February
• UX research and design, test infrastructure building, talking to downstreams and users about config flags and transition schedules: we need funding for this; earliest start is probably December, will take 2-3 months
• introduce the abstractions defined in resolvelib/zazo while doing alpha testing: will take a few months, so, conservatively estimating, May 2020?
• adopting better dependency resolution and do beta testing: ?

Is this right? What am I missing?

I ask because some of the info-gathering work is stuff a project manager and/or UX researcher should do, in my opinion, and because some progress on https://github.com/pypa/packaging-problems/issues/264 and other issues might help with concerns people have brought up here.

Bad metadata is sometimes an incorrect constraint, but more often it is the lack of a constraint that allows consideration of a much older, undesirable option.

Since using unconstrained dependencies or >= some_old_version is the rule rather than the exception for setup.py / pyproject.toml, this will be a problem. I don't really care if it's "incorrect constraint" or solver needing to make different choices - this is the state of metadata on PyPI.

These open constraints caused conda to explore really bad solutions requiring a lot of downgrades.

You're the expert here, but aren't there pragmatic solutions to this? In the majority of cases, no downgrades are needed, so try only that first. Then, downgrade only 1 version for each package. If you need to downgrade further, it's almost always the wrong solution (can be due to bad metadata or something else).

Actually, I'm not sure if pip _can_ even downgrade things. It first had a too aggressive "upgrade everything" strategy, now "upgrade as needed" works pretty well. I've never seen it downgrade anything, and in practice it works pretty well for regular use.

An example downgrade that conda must contend with, but pip will not: users of anaconda or miniconda with python 3 start with python 3.7. Users sometimes need to install something that is available only for python 3.6. The solver must downgrade python and all other non-noarch packages. This is a special case that could perhaps is be optimized by having special behavior for python version changes, but it illustrates the point of how changes in one package may require downgrades to another. "It has worked thus far" is dumb luck, not what actually works all the time.

As for restricting versions, you can't apply that in the solve itself. You have your constraints, and any sort of of "open up by one version" can't hope to be general enough, because it's different for every package. You can cut metadata by version prior to the solver, though. That's what conda's "current_repodata.json" is. Only the latest version. It makes things go really fast when it works, but people would get really mad if that were the only repodata. Things would not be reproducible, and they would get frustrated that specs that work one day may not the next. We provide a fallback to the full repodata, and also plan to introduce time-based subsets, with only the newest versions available at given points in time. Incrementally opening up the available index data might be a more useful concept with the backtracking solver.

pip _can_ even downgrade things.

It can -- for pip, downgrading is just an uninstall-install step like upgrading. And it does downgrade when it sees a constraint that it's been asked to satisfy.

The following does downgrade setuptools -- as long as it's the first thing pip sees:

pip install "setuptools < 20.0"

where's the best place to post "here's an edge case I thought of" scenarios, so that they don't get lost?

https://github.com/pradyunsg/zazo/

Thanks, I'll keep that in mind. Although on rethinking, my "pathological case" is not actually that pathological. It's mostly just an extreme case of the fact that in order to know the dependency metadata for a package, you have to download and unpack the package, and in certain cases this can trigger downloading a lot of packages only to reject them. That might be an important limitation of the "simple index" protocol that we need to address, but it's not directly a resolver issue.

An example downgrade that conda must contend with, but pip will not: users of anaconda or miniconda with python 3 start with python 3.7. Users sometimes need to install something that is available only for python 3.6. The solver must downgrade python and all other non-noarch packages. This is a special case that could perhaps is be optimized by having special behavior for python version changes, but it illustrates the point of how changes in one package may require downgrades to another. "It has worked thus far" is dumb luck, not what actually works all the time.

It's also a case where you normally _don't want to downgrade_. As a user, I'd much rather get an exception telling me the package is not available, and let me explicitly install 3.6 if that's what I want.

Another example is inconsistency between channels. Recent example: we were at Python 3.7.3, then base got 3.7.4. I don't care about those version differences, and most users won't. A default "do nothing" would be way better than "hey .4 > .3, let's upgrade that and then change channels of other packages to base if we have to (even if that downgrades those)".

We provide a fallback to the full repodata, and also plan to introduce time-based subsets, with only the newest versions available at given points in time

That sounds like a very useful improvement.

The problem with "garbage in, garbage out" as a mentality is that as the maintainers of the solver, you are holding the bag. Unless you have very good heuristics for what is garbage, you are powerless.

Yeah, that's true. I think every IDE, distribution or "common interface to a bunch of stuff" has this issue.

The following does downgrade setuptools

That's a user-requested downgrade. An indirect one is what I meant (e.g. setuptools<20.0 in pyproject.toml`). That would work as well I guess, but it is rare in practice.

The problem with "garbage in, garbage out" as a mentality is that as the maintainers of the solver, you are holding the bag.

100% agreed. It's something that has to be handled very carefully. But conversely, trying to work out a sane behaviour for any old junk isn't viable - we have to draw a line somewhere.

Just as a reminder, this discussion was triggered by the question of whether we'd need to retain the existing resolver indefinitely. I'm not sure any of these points really impact that question - the existing resolver isn't right, the best you can say is that it's broken behaviour that people are familiar with. So I stand by what I said above, there's no reason to retain the old resolver beyond an initial transition period.

And actually, for a big chunk of use cases, the old and new resolvers will likely give the same results anyway.

"Right" doesn't matter to users when they get broken by your changes. Any change in behavior will absolutely infuriate some number of users. Telling them that their workflow is wrong and that they need to change hasn't been a terribly effective strategy for me. I think you need to play it by ear. I certainly wouldn't want to maintain the old resolver forever, but you probably need to give people a path to keeping it - like making it a plugin that someone else adopts and maintains, and people can install that separately from pip.

trying to work out a sane behaviour for any old junk isn't viable - we have to draw a line somewhere.

Yes, but what is "old junk?" What distinguishes it? What about bad junk vs good junk (bearing in mind that newer is not always better)? My advice is to spend a lot of time making the solver be very debuggable. Make it easy for users (not just you as the expert) to trace where things go sideways to make it easy to identify when bad metadata is the issue, and what that bad metadata is. This is a skill that most users currently do not have, and honestly most users I've seen don't want to have this skill. I don't think you (or conda for that matter) will ever be able to have a completely accurate automatic heuristic for bad vs. good.

It's also a case where you normally don't want to downgrade. As a user, I'd much rather get an exception telling me the package is not available, and let me explicitly install 3.6 if that's what I want.

@rgommers, conda 4.7 moved to this - requiring the explicit python spec to change minor versions. People have hated it. I have no idea what fraction of the population the vocal ones are, but lots of people really, really don't like that they used to be able to conda install something, and now they can't. They don't care much about the reason, and they're mostly mollified by the answer, but we still get the hostility to deal with in the meantime. Just another example of

"Right" doesn't matter to users when they get broken by your changes.

Another example is inconsistency between channels. Recent example: we were at Python 3.7.3, then base got 3.7.4. I don't care about those version differences, and most users won't. A default "do nothing" would be way better than "hey .4 > .3, let's upgrade that and then change channels of other packages to base if we have to (even if that downgrades those)".

This is a much more complicated point. You're basically proposing different optimization criteria. You might have seen the current 11 steps in our blog post at https://www.anaconda.com/understanding-and-improving-condas-performance/

These are extremely tricky, and there is no global optimum that satisfies every situation. Given binary compatibility, and channels as islands of said compatibility, the strong priority of channels is pretty essential. Python builds don't matter, you're right, but there's currently no way to express a binary compatibility constraint vs. a plain and simple version constraint. You would need that in order to think about your idea of just leaving things alone. Otherwise, you'd quickly be in binary incompatibility hell.

Hi, thanks @msarahan for at-mentioning me in this issue. I wanted to chime in earlier but haven't found the time.

Indeed I experimented quite a bit with using libsolv as a solver for conda specs. And it does work -- the remaining difference now is that conda does not care much about the build number, but libsolv in the way it is coded (and with the backtracking solver) does. Even when using the full repodata, libsolv is really fast – I would go as far as saying that the speed of libsolv is fast enough to not be annoying :)

My big contribution to libsolv was to make it cross-platform compatible so that it now compiles on Windows, OS X and Linux.

I would totally advocate to use libsolv for a new resolver, even though it's a blob of compiled code (at least it's fast) and @mlschroe might be available to help out – he helped a lot with the libsolv support for the conda matchspec.

On the other hand I am not sure in what stage the resolver development is and wether it's already too late now, and wether compiled code is acceptable or not.

You might have seen the current 11 steps in our blog post at https://www.anaconda.com/understanding-and-improving-condas-performance/

Indeed I did. That was a nice blog post.

You're basically proposing different optimization criteria.

not really. I think your _"binary compatibility constraint vs. a plain and simple version constraint"_ just points to something I'm missing probably. What I'd expect is that no package should have python >= 3.7.4 or == 3.7.4 in its metadata, it's always == 3.7 (just checked for scipy, meta.yaml says python and conda_forge.yml says max_py_ver: '37' - makes sense). So introducing a 3.7.4 should do nothing - the resolver choosing 3.7.3 and not changing anything else is much cheaper (and valid according to your 11 steps) than forcing 3.7.4 and triggering a chain of up/down-grades.

Guess this part is getting off-topic for pip rollout plans, so happy to take this somewhere else.

My advice is to spend a lot of time making the solver be very debuggable.

+1

Also: make (keep) it as repairable as possible. That's the nice thing with pip now, if it messes up usually one can do cd site-packages && rm -rf troublesome_package (possibly followed by reinstall with --no-deps) and things work again. The likes of conda, apt and friends are much harder to repair that way.

You're basically proposing different optimization criteria.

I don't think you're factoring the concept of channels into your thinking enough. I don't know how relevant it is to pip. Definitely much less than it is to conda, but I'm not sure if it's totally irrelevant. People can still gather packages from more than one index at a time, right?

Packages don't have python >=3.7.4, nor ==3.7.4. The standard in conda packaging is to have an upper and lower bound. These are generally automatically determined by conda-build using information provided by the recipe author regarding how many places of the version to consider a compatible range. Packages have constraints like >=3.7.2,<3.8.0a0, with the 0a0 awkwardness being there to account for the fact that prerelease versions are below .0 releases, and would thus match a <3.8.0 spec where people don't really expect it to.

Packages also have a channel associated with them. This channel is effectively part of the version optimization: https://github.com/conda/conda/blob/4.6.7/conda/resolve.py#L1074 - the channel is like a super-version, one place ahead of the package's major version. If a solve should not change python, then the python spec can't be python ==3.7 - that is a range, and channel will affect that range. Specifically, having a python ==3.7 spec and starting with an install from the defaults channel, then adding the conda-forge channel will result in a lot of churn, because you have introduced new python packages that are higher in "version" (including channel), and your python spec is permissive of that change.

Conda 4.7 introduced much more aggressive "freezing" of specs, and I'm pretty sure that behavior is what you're after. That's pretty complicated, though. It boils down to only freezing things that don't conflict with your explicit specs. How you determine "conflict" is the hard part. We think it's better to not freeze things that would prevent the solver from giving the user the newest packages that are part of the graph of dependencies for that package. This freezing is worth mentioning because it can be done on a per-spec basis for pip in a way that conda can't. I think it might be a great optimization for a backtracking solver.

Also: make (keep) it as repairable as possible. That's the nice thing with pip now, if it messes up usually one can do cd site-packages && rm -rf troublesome_package (possibly followed by reinstall with --no-deps) and things work again. The likes of conda, apt and friends are much harder to repair that way.

Yes, this is very important. I think pip has done a good job of judiciously vendoring things so that it's harder to break. That is very wise, and conda is learning from that. If you do end up using any compiled code, make sure that it is statically linked or otherwise impossible for dynamic loading to cause problems.

(libsolv tangent: back when I used to work for RH, I grabbed https://pypi.org/project/solv/ to close the "pip install solv" security loophole on Fedora, since the libsolv build process doesn't generate an sdist or wheel archive at the moment, let alone publish it to PyPI. Happy to chat to anyone that might be interested in make those real library bindings with a bundle copy of libsolv, rather than the innocuous placeholder it is now)

Regarding my "opting out" comment, I don't mean "fall back to the old installation logic", I mean "provide an option to skip installing packages that would cause constraint violations, rather than failing the entire installation request".

Yum/DNF offer that via their --skip-broken option (in DNF that flag is an alias for --setopt=strict=0), and I think pip's resolver should offer a similar option.

@ncoghlan Ah right. That makes sense.

"ignore version conflicts on this package" style option

I'd already mentioned that we'd do that, which is why I got confused by your comment.

We're on the same page then. :)

@ncoghlan replied to my proposed timeline on distutils-sig and said it sounds reasonable.

@pradyunsg - looking forward to your next monthly update!

I spent some time taking a look at this again, and filed #7317.

I think we're almost there w.r.t. the abstractions -- thanks to a lot of work on pip's index interaction, dependency resolution + build logic separation and a bunch of general cleanup.

I just closed #7317. As far as I can tell, the dependency resolution is now decoupled (enough) from the metadata build logic. The build logic refactor has progressed well and it is no longer a blocker for further progress now.

We can now begin working on implementing the [resolvelib] abstractions in pip, taking reference from [passa] and poetry's resolver where appropriate. :)

@pradyunsg I am planning on extracting the base resolver (based on PubGrub) from the Poetry codebase (see https://github.com/sdispater/poetry/tree/master/poetry/mixology). It's mostly decoupled from the rest of the code but there are still references to internal parts that I need to abstract.

If you are interested in helping with that please let me know. The idea is to have a standalone implementation of the PubGrub algorithm that can be used by third parties and will be put in https://pypi.org/project/mixology/ which currently holds the code of the old resolver.

@sdispater Definitely! I don't know if I can help directly (time constraints) but it'd be awesome if you could decouple the PubGrub port from the rest of poetry!

One of the things that'd be really nice, would be to have a consistent abstraction layer, such that pip, poetry and pipenv use the same abstractions. Right now, we have zazo (mine), mixology (poetry's) and resolvelib (pipenv's) -- all define an abstraction layer of some sort and they're slightly different but (ridiculously!) similar. If you're open to this, do let us know!

FYI, we (@wolfv, and the @QuantStack team in general) have responded to the RfP for the pip dependency resolver.

The proposed approach is to adopt the libsolv C library, and to contribute the support for pip's version constraints format to libsolv. We would expose the C library through new Python bindings.

Libsolv is battled-hardened library underlying the RPM ecosystem, and therefore already used at industrial scale.

• Libsolv's is distributed under the BSD-3-Clause license.
• Libsolv supports multiple packages and repository formats, such as rpm, deb,
  haiku, conda, arch. Importantly, the variety of formats and ways to express
  dependency constraints shows that it is a pluggable system that should be able
  to accommodate pip's syntax for constraints on dependency versions..
• Using libsolv instead of conda's solver in the thin mamba wrapper, we were
  able to improve significantly upon conda's performances. (conda's slowness in package resolution with large channels was our main motivation for working on mamba).
• It works cross-platform on Windows, OS X and Linux. (@wolfv did the windows port of libsolv)
• It performs full SAT solving to find optimal dependency combinations and if it does not succeed, it returns actionable hints for conflict resolution.

The proposed approach is to adopt the libsolv C library

There would need to be a fallback for platforms that libsolv does not support (for example, AIX support in pip is actively being worked on, and you didn't mention BSD). So libsolv as a performance option where available is plausible to me, but we're not really in a position to only use it. (Is there a pure Python version of libsolve, i.e. something that gives the same results, just slower?)

Also, how would get-pip.py work? Would we have to include binaries for libsolv for all possible platforms? Again, I'd assume not, we'd use a pure-python fallback.

Not being able to use external C code has been an annoyance for pip for a long time now. I'd like to see a good solution for it, but (a) I'm not sure there is one (short of some form of "mini-pip" bootstrapper solution that allows us to de-vendor altogether) and (b) it's a big enough piece of work that I'd hate the new resolver to depend on it.

Hi @pfmoore

I think additional platform support shouldn't be that hard to achieve as libsolv is fairly straight-forward C code. I am pretty sure that there is no pure Python version of libsolv, though (which I understand is a drawback, but there is also no pure Python version of Python, or the Python standard lib, so in my mind it shouldn't be a blocker).

I think for bootstrapping, one could have a pure Python pip that uses the current mechanism for resolving, which then installs the necessary resolver lib based on libsolv. E.g. one could pin the exact package of libsolv + pip-specific Python bindings, and install them from the boostrap-pip as you describe. It sounds totally doable to me, but you might know better what would be involved ...

I think additional platform support shouldn't be that hard to achieve

To be clear, I've no vested interest in the more niche platforms, I just think we have to be very clear if we're impacting what platforms pip is supported on (which at the moment is basically "anything that can run Python"). There's also the deployment question of how we ship a C extension (as pip is currently shipped as a "universal" wheel, and that's important for some use cases like get-pip.py)

I think for bootstrapping, one could have a pure Python pip that uses the current mechanism for resolving

I've argued above, and I'll repeat it here, I don't really want to have to maintain two resolvers in pip indefinitely.

But I don't want to be too negative about the proposal - I just wanted to flag some of the reasons we don't currently allow dependencies on C extensions, in case you weren't aware of them.

The discussion is probably better suited elsewhere, and might be totally redundant when/if more detail is revealed, but I really want to let my questions out now.

From my understanding, libsolv uses a fully SAT solver for dependency resolution, and requires loading in dependency information first before it starts solving. But PyPI as of now stores dependency metadata per-package. Even if you ignore setup.py’s runtime-dependent nature, it’d be difficult to efficiently fetch the information needed for the SAT solver.

How do you plan to handle this problem? Do you plan to implement infrastructure (and propose additional spec for third-party repo implementations) to generate .solv files when packages are uploaded? Or do you have some stragegies up your sleeves generathing appropriate solvable data as the solver goes, and maybe implement some backtracking as new dependency data comes in?

I am not aware of any existing work in this regard, and most resources I can find suggest the Python packaging landscape need something else/more than a straight-up SAT solver. So I’m very interested in any possibilities here.

Those .solv files are just for caching, libsolv doesn't need them for solving. But I do agree that the dynamic nature of PyPI's dependencies makes it hard to use a SAT solver.

(See https://docs.google.com/document/d/1x_VrNtXCup75qA3glDd2fQOB2TakldwjKZ6pXaAjAfg/edit for more information)

(Note that a SAT solver is just a backtracking solver that also does clause learning if it runs into a conflict, so I think it is possible to use a SAT solver for PyPI. But it needs to be a solver that allows to dynamically add clauses while solving.)

Sat solvers have some significant capabilities these days including dynamic additions, restarts, backtracking, random restarts, etc. But I think the challenges here are partly going to be technical ones related to the need to support platforms where there is no guarantee that you can build a C-based solver.

I'm at the airport right now so I can't respond to the points that have been raised right now but... I'm sure we shouldn't be discussing the technical choices/tradeoffs here -- this is more scoped to how we communicate and manage the rollout vs what we rollout. :)

I filed #7406 for further discussion on the technical trade-offs -- @sdispater, @techalchemy, @uranusjr, @wolfv I'd appreciate if we could have the further discussion around the various choices for the resolver design.

To set expectations early, I'm gonna be traveling for the next 2 weeks and hopefully will be able to catch up with all the discussion on ~Dec 9th.

Status update: the PSF was able to get some funding from Mozilla Open Source Support and the Chan Zuckerberg Initiative to hire contractors to work on the pip resolver and related user experience issues. You can see our roadmap (which I need to polish up) and blog and forum and mailing list posts and notes from recent meetings to keep apprised. I'll be posting something about this soon to distutils-sig and the Packaging forum on Python's Discourse instance.

We aim to have pip's resolver feature prepared for release in pip 20.2 in July. (Per the quarterly release cadence for pip, unforeseen difficulties may delay till 20.3 in the next quarter.)

@uranusjr:

From my understanding, libsolv uses a fully SAT solver for dependency resolution, and requires loading in dependency information first before it starts solving. But PyPI as of now stores dependency metadata per-package. Even if you ignore setup.py’s runtime-dependent nature, it’d be difficult to efficiently fetch the information needed for the SAT solver.

The prototype pip resolve command in #7819 uses two techniques to get this information performantly (see that issue for details):

1. > Extracting the contents of the METADATA file from a url for a wheel without actually downloading the wheel at all.
2. > Caching the result of each self._resolve_one() call in a persistent json file.

The technique used for (1) is able to convert a wheel URL into a list of requirement strings that it depends on very quickly, while downloading only a few KB of the wheel's contents itself. The prototype in #7819 then ensures that req.populate_link() is called on each of the dependent requirements returned by self._resolve_one(), and stores the mapping of (==Requirement, url) -> [list of (==Requirement, url) non-transitive dependencies] in a persistent json cache file. (1) gets new information fast, (2) makes old information fast to query.

While I am not yet familiar with libsolv, I believe that entries of this mapping from requirement URL to dependencies and their URLs may be exactly the atomic input required by a SAT solver. As demonstrated in #7189, the persistent json dependency cache file caused pip resolve invocations to become complete no-ops after the first run, taking 800-900ms on the command line from then on. If the techniques from (1) and (2) are used, I believe it may be possible to let a SAT solver run to completion each time pip is invoked without waiting an incredibly long time. It probably wouldn't be too hard to try hacking libsolv on top of the prototype in #7189 to make this number more concrete.

@techalchemy:

Sat solvers have some significant capabilities these days including dynamic additions, restarts, backtracking, random restarts, etc. But I think the challenges here are partly going to be technical ones related to the need to support platforms where there is no guarantee that you can build a C-based solver.

pants used to have some code that made it easier to expose a compiler and linker to a setup.py-based project (#6273), but we later removed this (see #7016) in favor of making it possible to build C/C++ in pants without using setup.py, which was appropriate for our use case inside Twitter which only needed to build a shared library for TensorFlow custom operators. We host prebuilt binaries for statically-linked GCC and binutils archives on our s3 for OSX and Linux (see https://github.com/pantsbuild/binaries/) so that pants users don't have to install anything besides python and a JDK to use pants at all.

I would be interested in helping to brainstorm and/or develop any type of tooling for portably building C and C++ that may enable pip to reliably depend on libsolv on all supported platforms.

@cosmicexplorer

pants used to have some code that made it easier to expose a compiler and linker to a setup.py-based project (#6273), but we later removed this (see #7016) in favor of making it possible to build C/C++ in pants without using setup.py, which was appropriate for our use case inside Twitter which only needed to build a shared library for TensorFlow custom operators. We host prebuilt binaries for statically-linked GCC and binutils archives on our s3 for OSX and Linux (see pantsbuild/binaries) so that pants users don't have to install anything besides python and a JDK to use pants at all.

The compiler/linker work is very interesting! There’s also discussion on decoupling the compiler from setup.py (or a PEP 517 build backend in general). It’s not really related to the resolver (at least not directly), but you might be interested: https://discuss.python.org/t/how-do-we-get-out-of-the-business-of-driving-c-compilers/2591

@cosmicexplorer

I would be interested in helping to brainstorm and/or develop any type of tooling for portably building C and C++ that may enable pip to reliably depend on libsolv on all supported platforms.

Myself and @wolfv have been looking at this for building parts of the DNF package manager stack across all major supported platforms for mamba and my own personal work with DNF. At this point in time, libsolv is now capable of being built for Windows, Linux, macOS, BSD, Haiku OS, and I'm vaguely aware of it being put on various UNIX systems as part of using DNF on UNIX. I know @dralley has also made solv binary wheels available for ~major platforms supported by PyPI~ Linux using manylinux2014 on PyPI.

We now have pip 20.1b1 out, a beta release that includes a very early (alpha) version of the new resolver (see #8099 for context on that, and a survey where people can give feedback). Here's the announcement. And https://github.com/pypa/pip/issues/7951#issuecomment-617851381 lists some places we have publicized the beta so far.

pip 20.1 is now out and includes the alpha version of the resolver.

We're discussing publishing another pip release in May, one that includes a further-along alpha of the resolver. And we're figuring out when to release the beta of the new resolver and make a "please test this" push.

We ran into delays as we were figuring out #8371, how to display certain error messages better, and handling a bunch of other hairy stuff; see https://github.com/pypa/pip/projects/6 and https://github.com/pypa/pip/projects/5 for more on our progress. #8206 is discussion of the upcoming beta, pip 20.2b2, which I hope we can publish by the end of June.

I've posted our mid-year report at the PSF blog. One key thing to know: later this month we'll be releasing pip 20.2 which will have a beta version of the new dependency resolver (pip 20.1 had an alpha version) available via an optional flag "--use-feature=2020-resolver". We'll be publicizing pip 20.2 a lot and asking a lot of users to put the new resolver through its paces.

Per #8511 we have now released pip 20.2. This release includes the beta of the next-generation dependency resolver. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag. Please see our guide on how to test and migrate, and how to report issues. The new dependency resolver is off by default because it is not yet ready for everyday use.

We plan to make pip's next quarterly release, 20.3, in October 2020. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

Please spread the word by pointing to this blog post -- spread the word on Hacker News, Reddit, Twitter, Facebook, Dev.to, Telegram, relevant Stack Overflow answers, and your favorite Slacks and Discords. Most of the people this will affect do not keep up with Python-specific developer news. Help them get the heads-up before October, and help us get their bug reports.

(Copying my note from #988.)

@zooba asked:

Do you think we should be updating the version of pip bundled with Python 3.9 at this stage (for the first RC)?

Similarly, is there a need to update Python 3.8 for its next release?

My assumption is that, yes, after the bugfix release early next week, yes, but @pfmoore @xavfernandez @cjerdonek @uranusjr @pradyunsg @dstufft what do you think?

Sorry, hit post too soon. My reasoning is that bundling pip 20.2 will make it far easier for seasoned developers to test the new dependency resolver easily while testing the newest versions of Python. But I don't know how much work it is to update that bundled version, or how often you want to do it.

Same here, it would be nice to include a 20.2.x in 3.9 for easier access to the new resolver.

what do you think?

You are correct, that's the plan. :)

OK, answered on python-dev -- yes, the version of pip bundled in Python 3.8 and 3.9 should be updated to 20.2.x.

Separately, on publicity, I'll note here some work in progress:

For the next ~6-8 weeks I'll be pushing to get wide publicity so that users try using the new pip. I suspect that the problem won't be so much "this individual package won't install"; it'll be unexpected conflicts among particular packages, perhaps dependent on environment and particular constraints files. We're trying to get some early feedback via the survey so that we can then fix bugs, set up more automated testing, etc., and so that those upstream packages can get heads-ups and push out fixed packages before pip 20.3 (example: the TensorFlow/numpy/scipy issue in https://github.com/pypa/pip/issues/8076#issuecomment-666493069 ).

No matter how much effort we put into this, there will be users dealing with inconsistencies who get tripped up with 20.3, and they will be frustrated and confused and hurt, and that will cause a support load for us and for all their upstreams. We aim to reduce this by getting users to test and by getting upstreams' attention.

So I plan to contact and leverage the groups that pay attention to their own particular domain-specific corners -- the data scientists, the teachers, the artists, the DevOps specialists, and so on.

I hypothesize that one way to get their attention is via the specific packages they rely on.

Yesterday I looked through some lists of widely-used packages and manually emailed a few people and created issues on a few repositories to suggest that they ask their users to test with the beta of the new resolver, to start the ball rolling and try out some wording/approaches to get some more publicity and testing. This led to confusion in at least one case -- see https://github.com/sqlalchemy/mako/issues/322#issuecomment-667546739 -- and once the bugfix release of 20.2 is out, I'll be a bit more systematic and clearer about

• why we're reaching out
• whom we've chosen to contact/when (linking to a public media strategy will help)
• why we can't use automated testing to find these problems ourselves

We have gotten a little attention on Twitter (1, 2) and Reddit (anyone want to respond to this question about PyPA funding?).

@zooba wrote (regarding the bundling):

Thanks. It looks like we can do it later this week and make the next round of releases. Please let us know asap if anything comes up that you wouldn't want to be released.

I find the --use-feature=2020-resolver for me generally fixes more problems than it causes.

is it too late to suggest an initial rollout via:

proposed 20.3 pseudocode

try:
    _2020_resolver()
except:
    legacy_resolver()

which would mean at least for my projects they would all pass without modification

following that, once the legacy resolver is disabled by default, I may for a while have some projects that work under 2020-resolver and some not under legacy-resolver, I'd like to be able to set one flag to enable a fallback:

proposed 20.4 pseudocode

try:
    _2020_resolver()
except:
    if not use_legacy_resolver:
        raise
    legacy_resolver()

We're planning to roll out 20.3 later this month. We anticipate that this will require a bunch of user support as confused users ask us questions.

@di has volunteered to help gather some volunteers to help with first response. These volunteers will answer questions and help with the user support load once the new pip comes out -- on Twitter, StackOverflow, and GitHub -- and escalate real bugs to the maintainer/contributor team's attention.

Dustin, I think you have a rough plan for how this would work - would you mind posting that here and then I'll get some consensus from other pip maintainers? Thanks deeply.

Here's my rough plan:

• Start a discussion on discuss.python.org asking for support
• Direct folks to a Slack channel that could serve as a communication channel between everyone
• Start a document outlining some FAQ and our responses
• Include a decision tree for new issue -> triaged issue
• Share this with the channel once we have a known release date
• Try and roughly schedule volunteers to be online & triaging in the days following the release

Thanks, @di. We are waiting till tomorrow to get OKs from other maintainers. And we are aiming to release pip 20.3 on Wednesday or Thursday, Oct 28 or 29.

@di I see that your plan is approved. Please go ahead!

@di heads-up about a possible delay in the release.

In case I am unavailable when we are able to publish 20.3 (which we hope will be next week), here's a publicity plan.

As I discussed in a comment elsewhere we decided to delay the release slightly, because of some CI problems cropping up and because of some external factors.

In today's team meeting we agreed that the 20.3 release will likely be tomorrow or Friday. You can follow #8936 for more.

I did not do as much per-package outreach as I had suggested in an earlier comment. But that does not mean we did no outreach. Some of the outreach we have done (some of which is catalogued in #8511 or this wiki page):

• Podcasts: Podcast.__init__, Test & Code, FLOSS Weekly, Software Developers Journey, Real Python
• A bunch of tweets
• A short YouTube video about the changes
• Emails to PyCon, PSF, and PyPI sponsors from their PSF sponsorship contact, asking them to tell their engineering departments about coming changes
• Email to pypi-announce and similar lists
• Blog posts like this one
• Email to civic tech and similar groups/think tanks

For the last few months we've been getting a steady stream of new issues from people testing the new resolver in 20.2 and the 20.3 beta, pip 20.3b1. Those reports have helped us improve the resolver, fixing bugs and improving the UX of its output. We've also substantially improved the "what's changing" user guide, partially in response to beta feedback.

Here's my rough plan:

* Start a discussion on discuss.python.org asking for support

* Direct folks to a Slack channel that could serve as a communication channel between everyone

* Start a document outlining some FAQ and our responses

* Include a decision tree for new issue -> triaged issue

* Share this with the channel once we have a known release date

* Try and roughly schedule volunteers to be online & triaging in the days following the release

@di I recognize that the constant uncertainty and delays have probably kept you from being able to do the scheduling. The new release date is tomorrow, Monday, 30 November. If you now have a discussion thread and a decision tree to share, please go ahead and share them!

pip 20.3 has been released, and it has the new resolver by default! Here's the release announcement on the PSF blog: https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html