pip 20.3 release (Q4 2020)

As always, need a release manager (RM) for this from @pypa/pip-committers.

I've self-assigned myself as the RM for this release (https://github.com/pypa/pip/issues/7531#issuecomment-569881977) -- as usual, if anyone is opposed to this, let me know. :)

Per our discussion in our meeting today, @pradyunsg believes we should be able to make this release in the second half of October, assuming we don't find a showstopper in a bug report before then.

I just spoke with @pradyunsg about the timing of the 20.3 release. Conclusion: @pradyunsg and I are leaning towards modifying the plans for 20.3 to:

• a pre-release this week
• a stable release in mid-November, around Nov. 11-12

Reasons: In terms of features and showstoppers, we are set to be ready to release this week. However, a few things have come up, just now, at the end of the 20.2 beta period:

• https://github.com/pypa/pip/issues/8664#issuecomment-716143700 We need to nail down our policy and implementation re: Python 2 support.
• #9030 and https://github.com/pypa/warehouse/issues/4321#issuecomment-716109820 We've just run into some CI blips/glitches that have gotten in our way and are not yet completely resolved.
• As Election Day in the United States (Tuesday, 3 November) approaches, we're seeing an increasing risk of civil unrest in the US (which may occur approaching, on, and immediately following 3 November), and we're considering the risk that a disruptive change to pip, released only a week or less before the election, would make it harder for civic tech developers/administrators and data scientists to use Python (in tools to assess/counter voting suppression and violence).

We planned an October release, in accordance with the usual release cadence, before the full scope of the pandemic and the US election situation emerged. But, as the release policy states,

The release manager may, at their discretion, choose whether or not there will be a pre-release period for a release, and if there is may extend that period into the next month if needed.

@pypa/pip-committers heads-up in case you are inclined to agree or disagree.

Looks like we should expect the prerelease in the next couple days (cc @pradyunsg).

We're being blocked (in making the 20.3.beta1 release); it's held up by a Python 2.7-specific CI failure that Pradyun is unable to reproduce locally. So 20.3.beta1 may not be out till Saturday.

https://dev.azure.com/pypa/pip/_build/results?buildId=29831&view=logs&j=f801bdd4-2ff4-5da8-3bd6-deea013b4470&t=89b17dec-01ee-56ca-032e-26fbc447775d&l=70

Alrighty, that's resolved now (see #9019 for details, and a very-annoyed-at-CI-stuff Pradyun). I'll be making the release (20.3.beta1) in the next few hours!

pip 20.3 beta1 is up on PyPI. get-pip.py has not been updated.

Happy Halloween folks! :)

Here's a rough plan for publicizing this release (separate from the script or checklist for actually making the release and uploading it to PyPI and updating get-pip.py):

• [x] blog post, to be cross-posted on pyfound.blogspot.com and blog.python.org (Ernest or other PSF staff can help post this to the PSF blog, and CPython core devs to the Python Insider blog, in case I'm not available) -- DONE, https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html and https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html
• [x] post a somewhat detailed, technical release announcement to Discourse, and distutils-sig and python-dev mailing lists -- should include release highlights, links to the changelog and surveys, thanks to the funders, and a brief paragraph about what to expect in 21.0 -- DONE, https://mail.python.org/archives/list/[email protected]/thread/56ILRE26U4KU55YNVY2GQMGYYCEXPW74/ , https://discuss.python.org/t/announcement-pip-20-3-release/5948 , https://mail.python.org/archives/list/[email protected]/thread/OU6SUVG5EU6LOAVMROAAZO4AI2G6S7EJ/
• [x] update #988 and #6536 with GitHub comments announcing the release and pointing to the blog post
• [x] update our meeting/update table on the wiki to add items about our blog post and Discourse announcement (Ernest or Ewa can do this)
• [x] have twitter.com/ThePyPA and twitter.com/ThePSF tweet about the release (I think Dustin can do the former, and Betsy W. at the PSF can do the latter; give Betsy specific wording to tweet), linking to the blog post -- DONE in https://twitter.com/ThePyPA/status/1333400979409137664 (asked Betsy to retweet from the PSF account)
• [x] post a LESS detailed release announcement (just the most crucial information, with a link to the blog post for more) to some further Python mailing lists:
  
  
  
  1. pypi-announce
  
  
  2. python-announce
  
  
  3. wheel-builders
  
  
  4. python-list
  
  
  5. psf-community
  
  
  6. psf-volunteers
  
  
• [x] forward the more detailed release announcement to Python Weekly, Import Python, PyCoder's Weekly, The Changelog, and LWN
• [x] also send the less detailed release announcement to some other mailing lists: NumFOCUS and the Carpentries discuss list -- DONE https://carpentries.topicbox.com/groups/discuss/T4bf0a9d08b83415f/python-users-important-changes-in-pip-20-3-dependency-resolver & https://groups.google.com/g/numfocus/c/D3JCVJuHQbU/m/PQXcDyPxBwAJ
• [x] update the /topic in #pypa and #pypa-dev on Freenode IRC to reflect the release
• [ ] ask Betsy W. at PSF to forward our less detailed release announcement to PyPI and PSF sponsors
• [ ] post a note in the CZI-Essential Open Source for Science chat announcing the release and linking to the blog post
• [ ] reach out to Chinese-, Japanese-, Portugese- and Spanish-speaking contacts for
  1. Facebook/Telegram/Reddit outreach
  2. Brazil list(s) like https://groups.google.com/forum/#!forum/python-brasil
  3. language-specific lists like
  - https://mail.python.org/mailman/listinfo/python-de
  - https://mail.python.org/mailman/listinfo/python-es

@pypa/pip-committers @pypa/pip-helpers @di Heads-up: we decided in today's meeting that we will probably be releasing 20.3 tomorrow or Friday. We'll of course know more tomorrow.

Pradyun has been chasing down several issues that have proven nastier or more critical than they originally seemed, notably #9011. (See the milestone.) We may be able to cut the new release on Sunday or Monday. I would really hate for it to be delayed much longer than that, such as going into December, but if we release something with known regressions that will create hard-to-debug trouble for enough users, then that'll cause high support costs for us and lowered credibility for future releases.

@brainwane - just wanted to say that although sometimes it might feel like you are speaking to the void with these updates, in reality they are extremely helpful for downstream planning and long may they continue! Keep up the great work (and to all the pip devs!) :+1:

I 100% agree with @pelson. It's so nice that I can just go to the issue tracker, find an issue about the pending release, and just... learn exactly what the state of it is. I can't think of any other project that does that so reliably. :slightly_smiling_face:

Sounds like it's fixed in the resolver upstream: https://github.com/pypa/pip/issues/9011#issuecomment-735222242 :tada:

Have you considered having another beta/rc release before going final?

The team was able to finally resolve #9011. In the past few days we also dealt with a new issue that's cropped up, around Mac OS Big Sur support,* and reviewed a few additional new or newly updated issues filed in the resolver.

IT LOOKS REALLLLLY LIKELY that we'll release pip 20.3 tomorrow, Monday, 30 November. @pradyunsg plans to publish it to PyPI around noon UTC/17:30 Mumbai/noon London/7am EST/4am PST on Monday. I'm writing release announcements today.

@pelson and @duckinator thank you. I really appreciate that.

@webknjaz We thought about it, but we do not have time to run a proper feedback cycle and listen and get feedback that might push the release date out further. So we may publish a beta in the next few hours, just in case there are people watching this & related issues who grab it and might find giant showstopper bugs, but we won't be publicizing another beta or release candidate before publishing 20.3 tomorrow.

* @pradyunsg and I discussed what to do about Big Sur support in pip 20.3. We had three options:

1. Release pip 20.3 while completely leaving the status quo situation in place, which breaks things for some people who have upgraded to Big Sur and are attempting to use pip with default configuration (for instance, pip will attempt to compile numpy instead of using a wheel, and this will break because most people don't have the necessary dependencies).

2. Merge some improvements to support Big Sur for most people https://github.com/pypa/pip/pull/9170#issuecomment-735430434 , and break a few tests for now, and unfortunately leave a certain chunk of behavior broken for users who set configuration to claim they are on a different/specific architecture on MacOS (behavior fixed in https://github.com/pypa/packaging/pull/361). Then release pip 20.3.

3. Wait for Pradyun's packaging.tags pull request https://github.com/pypa/packaging/pull/361 to be reviewed and merged, then get a new release of packaging, and vendor that into pip, and only then release 20.3.

To compromise between support and speed, we decided to go with option 2. See https://github.com/pypa/pip/pull/9170#issuecomment-735430434 .

I've been using pip check to check my pip installs have worked out. Does the new resolver make that unnecessary now?

@joelberkeley-secondmind Unfortunately, no, not yet, because of an issue we're talking about in #9094.

ok, thanks. But it isn't necessary if the whole workflow is pip install x y z; pip check?

If you start with an empty environment, yes.

pip 20.3 has been released!

• Uploaded to PyPI.
• Pushed updates/tags to GitHub.
• get-pip.py has been updated, and the change will propagate over the next few minutes.

@pradyunsg @uranusjr and the many others involved. Thank you! What a huge achievement!

We're planning a 20.3.1 release for some followup bugfixes.

I went most of the way through the release publicity checklist I made in https://github.com/pypa/pip/issues/8936#issuecomment-721440282 .

I screwed up -- I forgot to mention in our announcements and other materials that the old resolver is still the default when using pip with Python 2.x.

I am going to:

• [ ] update the blog posts
• [ ] update the Discourse post
• [ ] send an update to distutils-sig
• [ ] send an update to python-list
• [ ] send an update to python-announce
• [ ] send an update to wheel-builders
• [ ] update #6536
• [ ] send an update to python-dev
• [ ] see what needs updating in the user guide docs at pip.pypa.io and submit a pull request