Celery: Release version 4.3 of Celery

@auvipy @thedrow @georgepsarakis can we quick release 4.2.2 just for the redis limitation ? as it currently clashes with the latest redis which has incompatible changes

I'm not sure if I'll have the time. I'll try to do so.

I can release this time but need some clear instructions

It would indeed be gr8 to get a 4.2.2 including the redis dependency fix, because I spent too much time already to find the source of the issue, and would be good for any person jumping into it!

Will this new release support python 3.7?

yes

Will this new release support redis 3.0.1?

yes

@auvipy, @thedrow, Anything I can help with for this release?

Issue #5212 is critical to the release.
We still need to test Celery with Python 3.7.

@thedrow Looks like #5212 is fixed if we do a Kombu release, right?
IF that's correct then we need to add 3.7 testing to our CI. I can work on that.

py-amqp and kombu release is the pre-req, after that, I can polish my pr for 3.7 support to get it merged

@auvipy You mean this PR: https://github.com/celery/celery/pull/4859 ?

I released Kombu 4.2.2 with that fix only.
The build should pass now.

Any idea of an ETA for this version to release ?

There's still some work to do. We need to fix the build for Kombu as well.
I'll go over the remaining issues.

@thedrow Any issue I can take over?
Or are you going to list the remaining issues here?

@xirdneh you can check the milestone https://github.com/celery/celery/milestone/20 .

I think we can remove non-blockers from this milestone. and lean down the remaining issues. releasing py-amqp and related packages to check how they work on celery 4.3 rc1

Right now the build for master is failing on Python 3.7.
See https://travis-ci.org/celery/celery/jobs/473236382.

@thedrow I think this is due to Python 3.7 converting StopIteration exceptions raised in generators, to RuntimeError exceptions, see here.

My suggestion would be to add an exception handling branch, detecting the specific type of RuntimeError:
https://github.com/celery/kombu/blob/e4dc1688a2bfe422813ffc79d9db50c06f38fbaf/kombu/asynchronous/hub.py#L348-L359

except RuntimeError as e:
  if e.args != ('generator raised StopIteration',):
      raise e

Admittedly, the above could be considered somehow weak, but I think it is the only method of identifying the exception conversion, in this particular case.

That fix is incorrect.
See https://github.com/celery/celery/blob/master/t/unit/worker/test_loops.py#L386
According to PEP 479, generators should raise StopIterator anymore.

This should be fixed in #5263.

It seems we have uncovered a more serious issue which is a blocker in my opinion.
See https://travis-ci.org/celery/celery/jobs/473900629#L3204

Hopefully, https://github.com/celery/kombu/pull/972 will resolve this issue.

The build for kombu is broken so the entire test suite does not run.
See https://travis-ci.org/celery/kombu/jobs/472712374#L1215.

I can't reproduce the test failure for kombu locally :(

@thedrow I have fixed failing kombu tests - see PR https://github.com/celery/kombu/pull/978. Tests were using pyamqp library from master. The tests broke PR https://github.com/celery/py-amqp/pull/221.

Maybe a low hanging fruit would be to fix the couple DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

py-amqp 2.4.0 is now released.
Next in turn is Kombu.

@thedrow What's missing for the 4.3 release? The Kombu PR seems to be merged already - are we waiting for a new release of Kombu? Are we waiting for fixes to everything in https://github.com/celery/celery/milestone/20?

Trying to see where I can help.

I need to triage the issues which are critical for Kombu 4.3.
Anything in https://github.com/celery/celery/issues?q=is%3Aopen+label%3A%22Status%3A+Needs+Test+Coverage+%E2%9C%98%22+milestone%3Av4.3 or https://github.com/celery/kombu/pull/911 will help.
I think we're pretty much ready for a release of Kombu but I need to verify.
In the meanwhile I'm working on release notes.

Can we also please resolve the dispute in #5259?

Since I couldn't edit celery/kombu#911 I opened a new PR: https://github.com/celery/kombu/pull/991

And the same for #5206 . I opened #5289

About that StopIteration stuff: In Kombu it was changed to not "convert" the ValueError to StopIteration https://github.com/celery/kombu/pull/972/files

However, in celery worker.loops.asynloop a StopIteration error is catched and a new loop is created in that case. Won't the change in kombu without a change in celery cause some issues? Because now instead of creating a new loop, the existing (but broken?) loop is continued to be used?

https://github.com/celery/celery/blob/c1d0bfea9ad98477cbc1def99157fe5109555500/celery/worker/loops.py#L92

@larsrinn please open an issue about it so we won't forget to look into it.

While writing the ticket, I figured out there is no action required as returning from a generator raises StopIteration. The the external behaviour of kombu on which celery is relying here should not be changed. See for example:

def len_generator(max_):
    a = 0

    while True:
        yield a
        a += 1
        if a == max_:
            return


g = len_generator(2)
print(next(g))  # 0
print(next(g))  # 1
print(next(g))  # raises StopIteration

The failing test on CI for Python 3.7 is also passing if run against the kombu master branch. At least for me locally. So when the new version of kombu is released, the failing test in this job should pass: https://travis-ci.org/celery/celery/jobs/482062153

Will this include fix for the async issue?
i hope this gets released as soon as possible.

Yes, I'm waiting for it, too.

But I must say I'm a bit afraid about the future of celery. Python 3.7 is out for 7 months already and still not supported. I know this project is run by volunteers who probably have a ton of other things to do. But I contributed a couple of PRs in the past weeks, mostly adding test coverage and getting the CI to pass. It seems like none of them has been looked at, let alone merged, even though they are all so minor, reviews should be doable in a few minutes, if not even seconds. This is a pretty disappointing contributing experience.

It would be a great improvement to the project to do much smaller releases. 4.3 is a big release that still needs work, and in the meantime many users have critical issues like the severe memory leak, which could get fixed quickly in a 4.2.X release. Even contributing a particular fix is not very useful because it will not be released for months.

@larsrinn I will review your PR if I will have time.

It would be a great improvement to the project to do much smaller releases. 4.3 is a big release that still needs work, and in the meantime many users have critical issues like the severe memory leak, which could get fixed quickly in a 4.2.X release. Even contributing a particular fix is not very useful because it will not be released for months.

Yes, I can definitely back this. Considering that celery is pretty mature project with a massive feature set, I guess for most users it's much more important that critical issues get fixed quickly and compatibility to latest versions is ensured than adding new features/deprecating old stuff.

I definitely agree with the above statements. Python 3.7 support is the most important thing for me right now.

Is there an option to add additional maintainers?

Most of the companies that I have worked for past 9 years used Celery in one way or another. In the last one, we even had our own fork in order to apply the fixes that we needed, given that our issues were not fixed in the mainstream branch.

In my opinion, this project needs to start a sponsorship program, just like Django and DRF have. Make the companies pay for all the profit they make thanks to this amazing project and give that money to the developers so that they work full time on this.

Celery is slightly broken against current pytest (another popular module); #5271 describes the problem and #5097 fixes it. Perhaps if the 4.3 release is becoming too daunting, a minor version bump to 4.2.2 could roll up several smaller fixes (#5271 and other bug fixes)? I realize this is probably easier said than done, but it might allow a bunch of us to rejoin the release cycle instead of maintaining our own forks while awaiting 4.3. Thank you for this excellent module.

so which companies are going to sponsor our work? Is anyone interested? I will be very much interested to get some companies backing my time for working with celery part time/full time!

I'm sure if there was a proper setup for regular donations, it would be easier for companies to just subscribe for a monthly support and forget about it. This might be the source of a more stable income than a simple paypal one-time donate button. I could easily convince my company to do so for example, than asking them to give a lot in one go, or to remember them to make a manual regular donation.

I would add also, for our project, celery is the only blocker that prevent us to upgrade to python 3.7

one related link: https://tidelift.com/

I think sponsorship is very possible, but you need to plant the idea in people's heads. E.g. outside of this thread, nobody even thinks that this project may need support, they just download and install celery and think it just "exists". I think, if you guys had started a fundraiser, a lot of companies would donate.
I saw the donation link in the README and contributed a bit from a personal account, but it's still too obscure for companies to notice. It should be right in front of them (at least in all the social networks) and it should have a proper call to action. E.g. if you're a developer and you know your company relies on celery, ask your manager to donate.
I can say that whenever I see organized fundraisers from the open source projects I like, I always try to donate or at least spread their social media campaign. I'm sure a lot of people do the same.

I ran a fundraiser and it turned to a massive failure!! we have opencollective and tide lift option open in the readme. people interested can donate/sponsor there or ping me directly in my email.

@auvipy I saw most of the issues in the 4.3 milestone were moved or closed now.
Are we planning on doing a 4.3 release soon?

yes!!! as it's a long overdue!!

I'm on it. Expect an RC very soon.

I ran a fundraiser and it turned to a massive failure!! we have opencollective and tide lift option open in the readme. people interested can donate/sponsor there or ping me directly in my email.

Celery authors. I have become a backer from https://opencollective.com/celery. Thank you for your awesome tool. Your tools are helping me a lot :)

Anyone objecting to codename 4.3 as Rhubarb?
It's one of my favourite tracks from Selected Ambient Works II.

If you have any other suggestions, please make note of them here.

I like it :)

Kombu 4.3 is now released!

Can someone please take a look at our Windows build?
Some of the tests are failing.
Also, we're missing some Python versions there, we should add them (although that's not a blocker).

@thedrow Where are the Windows build running? Appveyor?

Can someone please take a look at our Windows build?
Some of the tests are failing.
Also, we're missing some Python versions there, we should add them (although that's not a blocker).

PR here https://github.com/celery/celery/pull/5329

So 4.3 is good to go now?

Hi! You closed this issue, but the version on pypi is still 4.2.1. Is there somewhere we can track the release on pypi? Thanks.

Hi folks,

@seirl I believe that this issue was closed accidentally due to description of #5329 contains Fixes #5180 text.

@auvipy Could you please reopen this issue until 4.3 isn't actually released?
Thanks!

My bad, I thought auto-close would just be an option for pullee

this was closed automatically by GitHub!!!

@xirdneh yup.

I'm almost done with the Changelog. I still have to elaborate on some of the items and I need to complete adding the remaining items I haven't gotten to.

The what's new section is still largely incomplete.

Feel free to help in any way possible.

We have a potential blocker for GA: https://github.com/celery/kombu/issues/1006
RCs will proceed as planned.
Can someone with access to a FreeBSD install debug this?

Hello everyone. Any ETA?

I just finished the release notes for the first RC.
It's going to be released today.

Do you have any ETA for stable release?

Once this has been tried in production for a couple of weeks we'll declare GA.
In the near future we will improve our release process and document it so that everything will be clearer.

RC1 is released. :tada:
Please try it in your staging environments.

In the meanwhile, if someone can take a closer look at https://github.com/celery/kombu/issues/1006, https://github.com/celery/kombu/issues/1007 and https://github.com/celery/kombu/issues/1004 which are all regressions from Kombu 4.3, that will help us hit GA.

I made a potential fix for https://github.com/celery/kombu/issues/1007 in https://github.com/celery/kombu/pull/1009

Since https://github.com/celery/kombu/issues/1004 seemed to be in the same area, I attempted a fix for that as well: https://github.com/celery/kombu/pull/1010

Both are now merged.

@lithammer Thank you!

Since this was so much fun I come bearing not one, but two solution proposals for the last issue (https://github.com/celery/kombu/issues/1006):

• Proposal 1: https://github.com/celery/kombu/pull/1011 (personal favourite)
• Proposal 2: https://github.com/celery/kombu/pull/1012

I took a short break from the release notes to write https://github.com/celery/py-amqp/pull/258.
I'm going to proceed with our what's new document.

I released Kombu 4.4 and Celery 4.3.0RC2.
Unless there's any objection or critical fix, this is going to be the last RC.

I just released py-amqp 2.4.3 as well.
It fixes two severe deserialization bugs. I have not heard anything about them in production.
Probably because there are no messages in the AMQP protocol with two consequent bitmaps.
The fixes are there for completeness as I guess they could crash Celery.

Thank you for the great work @thedrow.

Potential blocker for GA: https://github.com/celery/celery/issues/5371

@thedrow I might have some time this week to verify this. I'll update the issue if I'm able to.

@xirdneh Please update if you don't and I'll take care of it.

@lithammer If you feel like contributing more, I'll even grant you commit rights :)

Definite blocker for GA: #5377

I have completed our what's new document for this release.

Please review it and let me know if I missed anything.

@thedrow seems quite thorough and informative! One note though, I am not sure if it is automatically included somehow, but it seems to me like the contributor list has not been yet added?

I haven't generated the contribution list yet.
I'm going to do so right before GA.

Here's a list of our current release blockers:

• [x] #5377
• [x] #5371
• [x] #5328
• [x] #4812
• [x] #5383
• [x] #5259
• [x] #5007
• [x] https://github.com/celery/kombu/issues/1014

Potential Blockers:

• [ ] #5358
• [ ] https://github.com/celery/kombu/issues/1019
• [ ] https://github.com/celery/kombu/issues/1018

@thedrow I think https://github.com/celery/celery/issues/5383 is valid as well.

I just released Celery 4.3.0rc3.
This includes new features and bug fixes and some resolved blockers.

Do you maybe know when full version will be available?

When we will finish resolving all the blockers.

Only one blocker left and than some final documentation tasks.

@thedrow looks like celery/kombu#1014 has been closed by a revert-based PR. However, it doesn't show as complete on the checklist. Is it incomplete because the unique keyword still needs to be supported?

Yes. I'll do that and the final documentation tasks tomorrow.

Released! :tada:

Thank you everyone for your efforts, time and skills.

The next version, Celery 5 is something to be excited about.

I want to start removing python2 from master

Can I know when will the Celery 5 be released? I am really excited to use that.

maybe some times before Christmas :)

@auvipy Can you issue a blog post about 4.3?

yes of course, started kast ng=ight but got busy. will complete by today.