Gunicorn: Memory increase after service run a long time
HI :
I use flask + gunicorn to make an api service. Because I want to share a large data between worker processors, so I init the data in gunicorn.conf and use "gunicorn -c gunicorn.conf" to finish it. Any data in gunicorn.conf would be done only once, so every worker processor share a common large data. The big data in my program is ready only, no writing operation. But when I start my service and run a long time, sometime the memory increase 1G suddenly, I guess there may be some writing operation in gunicorn cause the system copy-on-write mechanism. And How do I share the big data between worker processors? I still confused by this problem, expect your reply!!! Thank you!
haolujun
All 3 comments
This is related to #1640.
The only other way to handle this would be to do something that avoids the Python memory management. Perhaps the mmap module could help.
tilgovi
on 10 Oct 2018
Thank you!!! It's very helpful!
haolujun
on 11 Oct 2018
HI:
With the information you give me, I upgrade my python from 2.7 to 3.7 and use gc.freeze() to avoid some objects collected by garbage-collector. This time, it dosen't appear that "memory increase 1G suddenly", but it casue a memory growthing slowly util each sub-processor memory increased 500M+, then stop growthing. I think gc.freeze() is work and make a different memory-appearance. I hope the work what I have done is helpful for other people. Thanke you！
haolujun
on 12 Oct 2018
Related issues
mrwillis
·
4Comments
twosigmajab
·
4Comments
lordmauve
·
3Comments
bywangxp
·
4Comments
Abraxos
·
4Comments
Most helpful comment
This is related to #1640.
The only other way to handle this would be to do something that avoids the Python memory management. Perhaps the mmap module could help.