Gunicorn: multiprocessing.cpu_count() is dangerous in containers
Due to https://bugs.python.org/issue36054, the recommendation in these docs is dangerous when running inside a container. If your container has been allocated (say) a single core and 512 MB of RAM on a machine with 64 cores, multiprocessing.cpu_count() thinks it has 64 cores instead of just 1, and you end up spawning 129 workers. This is way more than the container has resources for, and ends up causing out of memory crashes.
twosigmajab
All 4 comments
Hello 😄! Thanks for the notification.
The default in Gunicorn is a single worker, even if the documentation has an example of detecting the number. Given that this looks likely to be a Python bug fix eventually, I'm not sure there's anything we should do in particular here. Filing it downstream at flask-gunicorn makes a lot of sense, though.
tilgovi
on 2 May 2019
it's more something that should be fixed in python not really an issue there. The documentation has been made at the time docker were not even existing but I do think it's still useful for the example since it's very explicit. Closing the issue then.
benoitc
on 23 May 2019
Adding 1 tiny line to the docs to say ("but beware of BPO-36054 if using containers!") could save many users headaches for the ~years they'll still be using versions of Python that don't have this fixed. If you're not opposed to merging that, I'll submit a PR if no one beats me to it.
twosigmajab
on 23 May 2019
@twosigmajab good idea! please go ahead :)
benoitc
on 23 May 2019
Related issues
yongzhuo
·
4Comments
leonardbinet
·
4Comments
lordmauve
·
3Comments
ttcqaq
·
4Comments
gr790
·
4Comments
Most helpful comment
@twosigmajab good idea! please go ahead :)