์ฌ๋ฌ ์
๋ฌ๋ฆฌ ๋ฒ์ ์์ ๋์ผํ ๋ฐฉ์์ผ๋ก ์คํจํฉ๋๋ค.
์
๋ฌ๋ฆฌ ๋ฒ์ 3.1.7 ๋น๊ตฌ ๋ฒ์ 3.3.0.23
์
๋ฌ๋ฆฌ ๋ฒ์ 3.1.25 ๋น๊ตฌ ๋ฒ์ 3.3.0.23
celery worker -l debug --autoscale=50,4 -Ofair -Q tower_scheduler,tower_broadcast_all,tower,localhost -n celery@localhost
Ubuntu 14.04 ๋ฐ 16.04์์ ์ผ๊ฐ 8 ์๊ฐ ํตํฉ ํ ์คํธ๋ฅผ ์คํํ ๋ ์ฐ๋ฐ์ ์ผ๋ก ๋ฐ์ํฉ๋๋ค. RHEL 7์ด๋ Centos 7์์๋ ๋ฐ์ํ์ง ์์ต๋๋ค. ์๊ท๋ชจ ๋ ํฌ๋ฆฌ์์ด์ ๋จ๊ณ๋ฅผ ์งํ ์ค์ ๋๋ค.
์
๋ฌ๋ฆฌ ๋ง์คํฐ ํ๋ก์ธ์ค์ ํน์ ์์
์๋ ๋ชจ๋ ๋์ผํ ํ์ผ ์ค๋ช
์์์ ์ฐจ๋จ๋์ด read()
ํฉ๋๋ค. ์์
์๋ ์์
์ ๋ง์น๊ณ ๋ ๋ง์ ์์
์ ํ ์ค๋น๊ฐ๋์์ต๋๋ค. ์์๋ ์์
์๊ฐ ๊ณต์ ํ๋ read()
์์คํ
ํธ์ถ์ ์ฐจ๋จํ๋ "์ ๊น"์ํ์
๋๋ค.
์
๋ฌ๋ฆฌ๋ฅผ ๋ค์ ์์ํ๋ฉด ๋ฌธ์ ๊ฐ "์์ "๋ฉ๋๋ค. ์ข ๋ ์ธ๊ณผ ์ ์ผ๋ก๋ SIGUSR1์ ๋ง์คํฐ ํ๋ก์ธ์ค๋ก ์ ์กํ๋ฉด read()
์์คํ
ํธ์ถ์์ ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ฌ ๋ฌธ์ ๊ฐ "์์ "๋ฉ๋๋ค. ๊ทธ๋ฐ ๋ค์ ์์ด๋ ์ฝ๊ธฐ์์ ๋์์ ๋ณด๋ฅ์ค์ธ ๋ฉ์์ง๋ฅผ ์ฒ๋ฆฌํ๋ ๊ฒ์ฒ๋ผ ๋ณด์
๋๋ค. ๋ง์คํฐ ํ๋ก์ธ์ค๋ PIPE์์ read()
๋ฅผ ๋ค์ ์๋ํ์ง ์์ต๋๋ค.
https://github.com/celery/celery/blob/3.1/celery/concurrency/asynpool.py#L220 ์ ์ฐ๋ฆฌ๊ฐ ๋งค๋ฌ๋ ค์๋ ๊ณณ์
๋๋ค. https://github.com/celery/billiard/blob/3.3/Modules/_billiard/multiprocessing.c#L202 ์์๋ฐ์ __read__
๋ํ ํธ์ถ์
๋๋ค.
์ด๊ฒ์ ๋น ์ฐจ๋จ ๋ฐ ๋น๋๊ธฐ์
๋๋ค. ์ด ์ฝ๋๊ฐ ๊ต์ฐฉ ์ํ / ๋ฌดํ ์ฐจ๋จ ์๋๋ฆฌ์ค์ ์ด๋ป๊ฒ ์ํฅ์๋ฐ์ง ์๋์ง ์ ๋ชจ๋ฅด๊ฒ ์ต๋๋ค. read
๋ ๋น ์ฐจ๋จ ๋ฐฉ์์ด๋ฉฐ, ์์์ด ์ฃฝ์ผ๋ฉด ์์ํ ์ฐจ๋จํ ์์๋ ๋ฐฉ์์ผ๋ก ํธ์ถ๋ฉ๋๋ค.
์ด์ ๊ต์ฐฉ ์ํ๊ฐ ๋ฐ์ํ ๋ ์๊ฒฉ ๋๋ฒ๊ทธ ์ธ์
์ผ๋ก ์ด๋ํ ์ ์๋๋ก CELERY_RDBSIG="1"
์ธํธ๋ก ๋ฌธ์ ๋ฅผ ์ฌํํ๋ ค๊ณ ํฉ๋๋ค.
์ด๋ค ์กฐ์ธ์ด๋ผ๋ ๋์์ด ๋ ๊ฒ์ ๋๋ค.
OS ๊ด์ ์์ ์ด๊ฒ์ด ์ด๋ป๊ฒ ๋ฐ์ํ ์ ์๋์ง ์ถ๋ก ํ ์ ์์ต๋๋ค.
write()
๊ฐ ๊ด์ฐฐ๋์ง ์์ต๋๋ค.์ฌ๊ธฐ์๋ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค. ์์ ์๋ ์คํ์ค์ธ ๊ฒ์ฒ๋ผ ๋ณด์ด์ง๋ง ๋ฉ์์ง๋ฅผ ์ฒ๋ฆฌํ์ง ์์ต๋๋ค.
Python 3.6์์ ์คํ ์ค์ด๋ฉฐ Celery 4.1.0์ ์ฌ์ฉํ๊ณ ์์ต๋๋ค.
@codeadict ๋ ํ๋ก์ธ์ค (์์ ์ ๋ฐ ๋ถ๋ชจ)๋ฅผ
sudo lsof | grep 53r
๋ก ๋์ผํ fd์ธ์ง ํ์ธํ ์ ์์ต๋๋ค. ์ฌ๊ธฐ์ 53r
๋ ํ๋ก์ธ์ค๊ฐ ์ฐจ๋จ ๋ ํ์ผ ์ค๋ช
์์ ๋ฒํธ์
๋๋ค.
@chrismeyersfsu ๊ฐ ์ธ๊ธํ๋ฏ์ด strace
ํ๊ณ ํ์ดํ์์ ์ฝ๊ธฐ๊ฐ ๋ฉ์ถฐ ์๋์ง ํ์ธํ ์ ์์ต๋๋ค.
$ sudo strace -p <pid-of-celery-master-process> -s 100
Process X attached
read(53
Python ๊ธฐํธ์ ํจ๊ป python-dbg
์ค์น ํ ๊ฒฝ์ฐ gdb
๋ฅผ ์ฌ์ฉํ์ฌ ๋ง์คํฐ๊ฐ ๋ฉ์ถ ์์น๋ฅผ ํ์ธํ ์๋ ์์ต๋๋ค.
$ gdb python -p <pid-of-master-process>
...
(gdb) py-list
216 # header
217
218 while Hr < 4:
219 try:
220 n = __read__(
>221 fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
https://github.com/celery/celery/blob/v3.1.17/celery/concurrency/asynpool.py#L219
๊ฐ์ ๋ฌธ์ ์ ๋๋ค.
์
๋ฌ๋ฆฌ == 3.1.23, ๋น๊ตฌ == 3.3.0.23
์ต์
์ผ๋ก ์์๋ ์์
์ : -time-limit=7200 --autoscale=64,2 --maxtasksperchild=1
ํ์ด์ฌ 2.7.3
๋ช ์๊ฐ ํ ์
๋ฌ๋ฆฌ '๋๊ฒฐ'
๋ง์คํฐ์ ์ฐจ์ผ๋ ๋ฌดํ๋๋ ํ์ดํ (btw, ๋ค๋ฅธ ํ์ดํ)์์ read ()๋ฅผ ๊ธฐ๋ค๋ฆฝ๋๋ค.
๋ฐ๋ผ์ ์ด๊ฒ์ ๋ง์คํฐ ํ๋ก์ธ์ค์ ๋๋ค.
root<strong i="13">@vm1</strong>:~# strace -p 41840
Process 41840 attached - interrupt to quit
read(50, ^C <unfinished ...>
Process 41840 detached
root<strong i="14">@vm1</strong>:~# sudo lsof | grep 50r
python 41840 root 50r FIFO 0,8 0t0 548796 pipe
python 46237 root 50r CHR 1,3 0t0 7604 /dev/null
python 46304 root 50r FIFO 0,8 0t0 548796 pipe
root<strong i="15">@vm1</strong>:~# ls -lh /proc/41840/fd/50
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]
root<strong i="16">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[548796]') 2>/dev/null
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/51 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/41840/task/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/41840/task/41840/fd/51 -> pipe:[548796]
l-wx------ 1 root root 64 Oct 20 10:02 /proc/46237/fd/51 -> pipe:[548796]
l-wx------ 1 root root 64 Oct 20 10:07 /proc/46237/task/46237/fd/51 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/51 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/46304/task/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/46304/task/46304/fd/51 -> pipe:[548796]
๊ทธ๋ฆฌ๊ณ ์๋ ์ค ํ๋ :
root<strong i="5">@vm1</strong>:~# strace -p 46304
Process 46304 attached - interrupt to quit
read(16, ^C <unfinished ...>
Process 46304 detached
root<strong i="6">@vm1</strong>:~# ls -lh /proc/46304/fd/16
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]
root<strong i="7">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[482281]') 2>/dev/null
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/41840/task/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root root 64 Oct 20 10:07 /proc/41840/task/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root root 64 Oct 20 10:02 /proc/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/46237/task/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root root 64 Oct 20 10:07 /proc/46237/task/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]
lr-x------ 1 root root 64 Oct 20 10:07 /proc/46304/task/46304/fd/16 -> pipe:[482281]
root<strong i="8">@vm1</strong>:~#
๊ทธ๋ฆฌ๊ณ ๋ง์คํฐ ํ๋ก์ธ์ค์ ํ์ฌ '๊ณ ์ ๋'ํ๋ ์ :
Thread 0x7efc1c101700
('Frame: ', 63347776)
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 54, in <module>
main()
File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
main()
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
return self.execute(command, argv)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
return self(*args, **options)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
ret = self.run(*args, **kwargs)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
state_db=self.node_format(state_db, hostname), **kwargs
File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
self.blueprint.start(self)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
step.start(parent)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
return self.obj.start()
File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
blueprint.start(self)
File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
step.start(parent)
File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 838, in start
c.loop(*c.loop_args())
File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
next(loop)
File "/opt/waf/python/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
cb(*cbargs)
File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 279, in on_result_readable
next(it)
File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 221, in _recv_message
fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
()
@harlov @codeadict @chrismeyersfsu ํด๊ฒฐ์ฑ ์ด๋ ํด๊ฒฐ ๋ฐฉ๋ฒ์ ์ฐพ์์ต๋๊น?
์๋์. ์ ๋ฌ๋ฆฌ 4๋ก ์ ๊ทธ๋ ์ด๋ํ๊ณ ์์ต๋๋ค.์ด ์ค๋ฅ๋ฅผ ํธ์ถํ๋ ๋์ผํ ํ ์คํธ ์ธํธ๋ฅผ ์คํํ๊ณ ์ ๋ฌ๋ฆฌ 4์์ ๋ค์ ๋ง๋ค ์ ์๋์ง ํ์ธํฉ๋๋ค.
@korycins ์ฐ๋ฆฌ๋ ์ ๊ฑฐ - ์๋ ์ค์ผ์ผ๊ณผ - maxtasksperchild - ๋ชจ๋ ๊ด์ฐฎ ์์ง ์์ต๋๋ค.
autoscale ยฎ maxtasksperchild๋ฅผ ์ฌ์ฉํ์ฌ ์์ ์๊ฐ ๊ณ์ ๋ฆฌ์คํฐ๋๋ ๊ฒ์ด ์ด๋ฌํ ์ํฉ์ ์ ๋ฐํ ์ ์๋ค๊ณ ์๊ฐํฉ๋๋ค.
@chrismeyersfsu ์
๋ฌ๋ฆฌ 4.1๊ณผ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ผ๋ฏ๋ก v4.1๋ก ์ ํํด๋ ๋์์ด๋์ง ์์ ๊ฒ ๊ฐ์ต๋๋ค.
์ด์ ์๋ --autoscale ์ต์
๋ง ์ฌ์ฉํ์ต๋๋ค. ์์
์๋ ์์๋ก ๋งค๋ฌ๋ฆฌ์ง ๋ง (์ฃผ๋น ํ ๋ฒ ์ ๋) maxtasksperchild = 20์ ์ถ๊ฐํ๋ฉด ์์
์๋ ํ ์๊ฐ์ ํ ๋ฒ์ฉ ๋งค๋ฌ๋ฆฝ๋๋ค. maxtasksperchild์ ๋ฌธ์ ๊ฐ์์ ์ ์๋ค๊ณ ์๊ฐํ๊ธฐ ๋๋ฌธ์ ์ ๊ฑฐํ์ต๋๋ค.
๋ฌด๋ฃ ๋ฆฌ์์ค๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ๋์ ์ผ๋ก ํ์ฅ / ์ถ์ํ๊ธฐ ์ํด ์ฌ์ฉ์ ์ง์ ์๋ ํฌ๊ธฐ ์กฐ์ ์ ๋ง๋ค์์ผ๋ฉฐ ๋์ผํ ๋ฌธ์ ๊ฐ ๋ฐ์ํ๋ฏ๋ก ๋ฌธ์ ๊ฐ ์์
์ ๋ค์ ์์ฑ๊ณผ ๊ด๋ จ์ด ์๋ค๊ณ ์๊ฐํฉ๋๋ค ( @harlov๊ฐ ๋งํ๋ฏ์ด)
๋๊ตฌ๋ ์ต์ ๋ง์คํฐ ์์ ๋ฌธ์ ๋ฅผ ํ์ธํ ์ ์์ต๋๊น? ์ต์ ์ปค๋ฐ์ผ๋ก ์ด๋ฏธ ์์ ๋์๋์ง ํ์คํ์ง ์์ต๋๋ค.
@auvipy ์ด๋ค ์ปค๋ฐ์ด ์ด๊ฒ์ ํด๊ฒฐํ ์ ์๋ค๊ณ ์๊ฐํฉ๋๊น? ํ์ฌ PyPI์ celery 4.1.0, billiard 3.5.0.3์์์ด ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ต๋๋ค.
ํ์คํ์ง ์์ผ๋ฏ๋ก ๋ง์คํฐ ๋ธ๋์น๋ฅผ ์ฌ์ฉํด ๋ณผ ์ ์์ต๋๊น? 4.2 ๋ฆด๋ฆฌ์ค ํ์๋ ์ฌ์ ํ ์กด์ฌํ๋ ๊ฒฝ์ฐ ์์ ์์ ์ด ์ํ๋ฉ๋๋ค.
@auvipy ์ฐ๋ฆฌ๋ ๊ทธ๊ฒ์
@ryanpetrello max_task_per_child ๋ฅผ ์ถ๊ฐํ๊ณ 1๋ก ์ค์ ํ์ญ์์ค. http://docs.celeryproject.org/en/latest/userguide/workers.html#max -tasks-per-child-setting
์ด๊ฒ์ ๋ฌธ์ ๋ฐ์ ๊ฐ๋ฅ์ฑ์ ๋์ฌ์ผํฉ๋๋ค.
๊ฐ์ ๋ฌธ์ , ์ง๊ธ์ ์ฌํ ํ ์ ์์ต๋๋ค. ๋ค์ ์์ํ ๋ ์์ ์๋ฅผ ๋์ ๋ ๊ตฌ์ฑ ๋ฌธ์ ๋ผ๊ณ ์๊ฐํ์ง๋ง ๋๋๋ก ๋ฐ์ํ๋ ๊ฒ์ฒ๋ผ ๋ณด์ ๋๋ค.
์ฐ๋ฆฌ๋์ด ๋ฌธ์ ์ ๋ถ๋ช์ณค๋ค ๊ณ ๋งํ๊ธฐ ๋งํ๋ฉด๋ฉ๋๋ค. ์
๋ฌ๋ฆฌ 4.1.0, ๋ค์๋ง 4.1.0, ๋น๊ตฌ 3.5.0.3.
๋ช
๋ น ์ค : /usr/bin/python /usr/local/bin/celery worker --app myapp.celery -B --loglevel INFO --autoscale=5,1 --max-tasks-per-child=50 --max-memory-per-child=1000000 -O fair
ํ์ผ ์ค๋ช ์ ์ฐจ๋จ๊ณผ ๊ด๋ จํ์ฌ ์์ @harlov ์ ๋์ผํฉ๋๋ค. ๋ง์คํฐ๋ ์์ ์ ์ค ํ๋์ ๊ต์ฐฉ ์ํ์ ์์ต๋๋ค.
SIGUSR1์ ์ ๋ฌ๋ฆฌ ๋ง์คํฐ์ ์ ์กํ๋ฉด ํ๋ก์ธ์ค๊ฐ "๊ณ ์ ํด์ "๋๊ณ ์๋์ ์คํ ํธ๋ ์ด์ค๊ฐ ๋คํ๋ฉ๋๋ค. ์์ฌ์ค๋ฌ์ด ์ ์ "on_result_Readable"ํจ์์์ ์ฝ๊ธฐ๊ฐ ์ค๋จ๋๋ค๋ ๊ฒ์ ๋๋ค. ์ด๊ฒ์ ๊ณ ์ ์ ์ธ ๊ฒฝ์ ์กฐ๊ฑด์ฒ๋ผ ๋๊ปด์ง๋๋ค. poll ()์ ์ฝ๊ธฐ ๊ฐ๋ฅํ์ง๋ง read () ๋ธ๋ก์ ๋ฐํํฉ๋๋ค. ์ด๋ฐ ์ผ์ด ๋ฐ์ํ ์ ์์ต๋๋ค. ์ด๊ฒ์ด ๋ฌธ์ ๊ฐ ์๋์ง ํ์ธํ๊ธฐ ์ํด ํญ์ ๋น ๋ธ๋ก ๋ชจ๋์์ FD๋ฅผ ์ค์ ํ๋ ๊ฒ์ด ๊ถ์ฅ๋๋ ์ด์ ์ ๋๋ค.
LKML ๊ด๋ จ ๋ฉ์์ง : https://lkml.org/lkml/2011/6/18/105
๋ง์ ์ ๋ฌ๋ฆฌ ๋ง์คํฐ๊ฐ์๋ ํ๊ฒฝ์ด ์๊ณ ํญ์ ๋ช ๊ฐ์ง ๋ฌธ์ ๊ฐ ์๊ธฐ ๋๋ฌธ์ ์ค๋ช ์๋ฅผ ์ฐจ๋จํ์ง ์๋๋ก ์ค์ ํ๋ ๊ฒ์ด ์ฝ์ง ์์ต๋๋ค. ๊ฐ๋จํ ๋ณ๊ฒฝ ์ฌํญ์ด๋ฉด ์ ์ํ๊ฒ ํ ์คํธ ํ ์ ์์ต๋๋ค.
MainThread [636/660]
=================================================
File "/usr/local/bin/celery", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 14, in main
_main()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 326, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 281, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 480, in handle_argv
return self.execute(command, argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 412, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 221, in run_from_argv
return self(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 244, in __call__
ret = self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 256, in run
worker.start()
File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 370, in start
return self.obj.start()
File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 320, in start
blueprint.start(self)
File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python2.7/dist-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/usr/local/lib/python2.7/dist-packages/kombu/async/hub.py", line 354, in create_loop
cb(*cbargs)
File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 292, in on_result_readable
next(it)
File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 235, in _recv_message
fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 349, in cry_handler
safe_say(cry())
File "/usr/local/lib/python2.7/dist-packages/celery/utils/debug.py", line 193, in cry
traceback.print_stack(frame, file=out)
=================================================
LOCAL VARIABLES
=================================================
{'P': <functools.partial object at 0x7f29da0f1c58>,
'frame': <frame object at 0x37137b0>,
'out': <vine.five.WhateverIO object at 0x7f29da14e150>,
'sep': u'=================================================',
'sepchr': u'=',
'seplen': 49,
'thread': <_MainThread(MainThread, started 139818014390016)>,
'threading': <module 'threading' from '/usr/lib/python2.7/threading.pyc'>,
'tid': 139818014390016,
'tmap': {139818014390016: <_MainThread(MainThread, started 139818014390016)>}}
์ ๋ฐ์ดํธ ์ผ๋ฟ์ ๋๋ค. ์ฐ๋ฆฌ๋ ๋ฌด์ธ๊ฐ๋ฅผ ํ ์คํธํ๊ณ ์ฌํํ๋ ๋ฐ ์๊ฐ์ ๋ณด๋์ต๋๋ค.
์ ๋ฌ๋ฆฌ์์ ์ผ๋ฐ์ ์ผ๋ก ์ฌ์ฉํ๋ ํ์ผ ์ค๋ช ์๊ฐ ๋ชจ๋ ์ฌ๋ฐ๋ฅด๊ฒ ๋น ์ฐจ๋จ์ผ๋ก ํ์๋๊ณ ์๋์ง ํ์ธํ์ต๋๋ค. ๋ฐ๋ผ์ ๋ฌธ์ ๋ ๋ชจ๋ ๋์คํฌ๋ฆฝํฐ๊ฐ ๋น ์ฐจ๋จ ์ธ ๊ฒฝ์ฐ ์ฝ๋๊ฐ ์ด๋ป๊ฒ ์ฐจ๋จ๋๋์ง์ ๋๋ค.
๋ด๊ฐ ์ง๊ธ ๋ฐ๋ฅด๋ ํธ๋์ epoll () ๊ฐ์ฒด๊ฐ ์์ฒญํด์ผํ๋ ์ค๋ช ์์ ๋ํด ํผ๋์ค๋ฌ์ํ๊ณ ์๋ค๋ ๊ฒ์ ๋๋ค. ์ ๋ฌ๋ฆฌ๋ฅผ straceํ๋ฉด ๋ค์๊ณผ ๊ฐ์ ๋ง์ ์ถ๋ ฅ์ ์ป์ ์ ์์ต๋๋ค.
14659 14:32:56.665937 epoll_ctl(4, EPOLL_CTL_ADD, 43, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=43, u64=14879199376994992171}}) = -1 EEXIST (File exists)
14659 14:32:56.667084 epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=16, u64=14879199376994992144}}) = -1 EEXIST (File exists)
14659 14:32:56.667278 epoll_ctl(4, EPOLL_CTL_ADD, 26, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=26, u64=14879199376994992154}}) = -1 EEXIST (File exists)
14659 14:32:56.667440 epoll_ctl(4, EPOLL_CTL_ADD, 23, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=23, u64=14879199376994992151}}) = -1 EEXIST (File exists)
14659 14:32:56.667607 epoll_ctl(4, EPOLL_CTL_DEL, 37, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)
14659 14:32:56.667804 epoll_ctl(4, EPOLL_CTL_DEL, 38, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)
ํ ๋ฒ์ ํ ๋ฒ ์ฌ์ฉ๋์๋ ํ์ผ ์ค๋ช ์๊ฐ ๋ค๋ฅธ ์ฉ๋๋ก ์ฌ์ฉ๋๋ฉด ํ๋ธ๊ฐ ํผ๋์ค๋ฌ์์ง๊ณ ์๋ชป๋ ๋ฉ์๋์ ๋ถ์ ํธ์ถํ๋ ๊ฒ์ ์ ์ ์ผ๋ก ๊ทธ๋ด๋ฏํฉ๋๋ค!
์ข์, ๊ทธ๋์ ๋ฌธ์ ๋ฅผ ๋ฐ๊ฒฌํ๊ณ ์คํจํ strace ์ถ๋ ฅ์ด ์์ต๋๋ค. ๊ทธ๋ฆฌ๊ณ ๊ทธ๊ฒ์ ๋งค์ฐ ๋ฏธ๋ฌํ๊ณ epoll () ์ธํฐํ์ด์ค์ ๋ค์๊ณผ ๊ฐ์ ์์ ํ ๋ถ๋ช ํ ํ '๊ธฐ๋ฅ'๊ณผ ๊ด๋ จ์ด ์์ต๋๋ค. ๋งจ ํ์ด์ง์์ :
Q6
ํ์ผ ์ค๋ช ์๋ฅผ ๋ซ์ผ๋ฉด ๋ชจ๋ epoll ์ธํธ์์ ์๋์ผ๋ก ์ ๊ฑฐ๋ฉ๋๊น?
A6
์,ํ์ง๋ง ๋ค์ ์ฌํญ์ ์ ์ํ์ญ์์ค. ํ์ผ ์ค๋ช ์๋ ์ด๋ฆฐ ํ์ผ ์ค๋ช ์ ๋ํ ์ฐธ์กฐ์ ๋๋ค (open (2) ์ฐธ์กฐ). dup (2), dup2 (2), fcntl (2) F_DUPFD ๋๋ fork (2)๋ฅผ ํตํด ํ์ผ ์ค๋ช ์๊ฐ ๋ณต์ ๋ ๋๋ง๋ค ๋์ผํ ์ด๋ฆฐ ํ์ผ ์ค๋ช ์ ์ฐธ์กฐํ๋ ์ ํ์ผ ์ค๋ช ์๊ฐ ์์ฑ๋ฉ๋๋ค. ์ด๋ฆฐ ํ์ผ ์ค๋ช ์์ด๋ฅผ ์ฐธ์กฐํ๋ ๋ชจ๋ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ ๋๊น์ง ๊ณ์ ์กด์ฌํฉ๋๋ค. ํ์ผ ์ค๋ช ์๋ ๊ธฐ๋ณธ ์ด๋ฆฐ ํ์ผ ์ค๋ช ์ ์ฐธ์กฐํ๋ ๋ชจ๋ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ ํ์ ๋ง โโepoll ์ธํธ์์ ์ ๊ฑฐ๋ฉ๋๋ค (๋๋ ํ์ผ ์ค๋ช ์๊ฐ epoll_ctl (2) EPOLL_CTL_DEL์ ์ฌ์ฉํ์ฌ ๋ช ์ ์ ์ผ๋ก ์ ๊ฑฐ ๋ ๊ฒฝ์ฐ). ์ด๋ epoll ์ธํธ์ ์ผ๋ถ์ธ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ ํ์๋ ๋์ผํ ๊ธฐ๋ณธ ํ์ผ ์ค๋ช ์ ์ฐธ์กฐํ๋ ๋ค๋ฅธ ํ์ผ ์ค๋ช ์๊ฐ ์ด๋ ค์๋ ๊ฒฝ์ฐ ํด๋น ํ์ผ ์ค๋ช ์์ ๋ํด ์ด๋ฒคํธ๊ฐ๋ณด๊ณ ๋ ์ ์์์ ์๋ฏธํฉ๋๋ค.
์๋ฌ๋ฆฌ๊ฐ ํต์ ์ ์ํด ํ์ผ ์ค๋ช ์๋ฅผ ์ฌ์ฉํ๊ณ ์๊ณ , ํ์ผ ์ค๋ช ์๊ฐ ์์ ์์๊ฒ ์ ์ถ๋๊ณ , ๋ง์คํฐ๊ฐ ํ์ผ ์ค๋ช ์๋ฅผ ๋ซ์ ๋ค์ non-blocking_์ผ๋ก ํ์๋์ง ์์ ์ผํฐ๋ฌ _์ ๋ํด ๋ค๋ฅธ ํ์ผ ์ค๋ช ์๊ฐ ์ด๋ฆฝ๋๋ค. ์์ ํ๋ก์ธ์ค๊ฐ ์ฃฝ์ด ํ์ผ ๋์คํฌ๋ฆฝํฐ๊ฐ ์ฝ์ ์์๋ ๊ฒ์ผ๋ก ํ์๋๊ณ ๋ง์คํฐ๋ ๋ธ๋ก์ ์ฝ์ผ๋ ค๊ณ ํฉ๋๋ค.
์, epoll ()์์ด ํ๋ก์ธ์ค์์ ๋ ์ด์ ์ด๋ ค ์์ง ์์ ํ์ผ ์ค๋ช ์์ ๋ํ ๊ฐ๋ ์ฑ์ ๋ฐํํฉ๋๋ค.
์ฌ๊ธฐ์ fd 49์ ๋ํ ์บก์ฒ ๋ strace ์ถ๋ ฅ์ ์ฆ๊ฑฐ๋ก ๋ถ์ฌ ๋ฃ์ต๋๋ค (๋ค์ค ๋ผ์ธ ์ถ๋ ฅ์ ๋ํ ์ฌ๊ณผ, strace๊ฐ ์ฌ๋ฌ ํ๋ก์ธ์ค๋ฅผ ๋ชจ๋ํฐ๋งํ์ต๋๋ค).
7417 15:04:00.001511 pipe( <unfinished ...>
7417 15:04:00.001646 <... pipe resumed> [49, 50]) = 0
7417 15:04:00.001727 fcntl(49, F_GETFL <unfinished ...>
7417 15:04:00.002091 <... fcntl resumed> ) = 0 (flags O_RDONLY)
7417 15:04:00.002165 fcntl(49, F_SETFL, O_RDONLY|O_NONBLOCK <unfinished ...>
7417 15:04:00.002222 <... fcntl resumed> ) = 0
7417 15:04:00.003052 fcntl(49, F_GETFL <unfinished ...>
7417 15:04:00.003195 <... fcntl resumed> ) = 0x800 (flags O_RDONLY|O_NONBLOCK)
...
7417 15:04:00.237131 fcntl(49, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK)
7417 15:04:00.237363 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417 15:04:00.237478 <... epoll_ctl resumed> ) = 0
...
7417 15:04:00.274482 read(49, <unfinished ...>
7417 15:04:00.274558 <... read resumed> "\0\0\0\25", 4) = 4
7417 15:04:00.274670 read(49, <unfinished ...>
7417 15:04:00.274734 <... read resumed> "(I15\n(I9395\ntp0\ntp1\n.", 21) = 21
7417 15:04:00.274851 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417 15:04:00.274970 <... epoll_ctl resumed> ) = -1 EEXIST (File exists)
...
... from here fd 49 is used normally for a while, have not included all the output ...
...
7417 15:04:52.871646 epoll_wait(4, [{EPOLLIN, {u32=49, u64=12923050543437316145}}, {EPOLLIN, {u32=33, u64=12923050543437316129}}, {EPOLLOUT, {u32=48, u64=12923050543437316144}}, {EPOLLOUT, {u32=8, u64=12923050543437316104}}], 1023, 155) = 4
7417 15:04:52.871888 read(49, "\0\0\0\31", 4) = 4
7417 15:04:52.872079 read(49, "(I4\n(I9400\nI155\ntp0\ntp1\n.", 25) = 25
... here it uses normal poll for a once?
7417 15:04:52.891818 poll([{fd=49, events=POLLERR}], 1, 10 <unfinished ...>
7417 15:04:52.902239 <... poll resumed> ) = 0 (Timeout)
7417 15:04:52.913780 close(49 <unfinished ...>
7417 15:04:52.914020 <... close resumed> ) = 0
... here we create a new pipe for a child process
7417 15:04:53.148195 pipe([49, 50]) = 0
7417 15:04:53.148996 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f2dee7f79d0) = 9482
... the write end got closed, now we dup it. This is the sentinal fd so is not marked non-blocking
7417 15:04:53.155409 dup(49) = 50
7417 15:04:53.156030 epoll_ctl(4, EPOLL_CTL_ADD, 50, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=50, u64=12923050543437316146}}) = 0
... and here it breaks
7417 15:04:53.776974 epoll_wait(4, <unfinished ...>
7417 15:04:53.785786 <... epoll_wait resumed> [{EPOLLHUP, {u32=49, u64=12923050543437316145}}], 1023, 1878) = 1
7417 15:04:53.785876 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9481, si_uid=905, si_status=155, si_utime=4, si_stime=3} ---
7417 15:04:53.786073 read(49, <unfinished ...>
... discontinuity ...
7417 15:09:24.338109 <... read resumed> 0x23f4f10, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
7417 15:09:24.345507 epoll_ctl(4, EPOLL_CTL_DEL, 49, 0x7ffeecbf7660) = -1 ENOENT (No such file or directory)
epoll ()์ด fd 49๊ฐ ๋ ์ด์ fdset์ ์๋ค๋ ๊ฒ์ ์ด๋ป๊ฒ ์ธ์ ํ์ง๋ง ๊ทธ๋ผ์๋ ๋ถ๊ตฌํ๊ณ ์ซ์๋ฅผ ๋ฐํํ๋์ง ์ฃผ๋ชฉํ์ญ์์ค.
์ด๋ฅผ ์์ ํ๋ ๋ฐฉ๋ฒ์ ์์ ์์์ epoll ()์ ์ํด ๋ชจ๋ํฐ๋ง๋๋ ๋ชจ๋ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ ์๋์ง ํ์ธํ๋ ๊ฒ์ ๋๋ค. ํ์ฌ ์์ปค๋ ์์ ์๊ฒ ํด๋นํ์ง ์๋ ํ์ผ ๋์คํฌ๋ฆฝํฐ๋ฅผ ๋ซ์ง ๋ง _other_ ์์ปค์ ๊ด๋ จ๋ ์์ ๋ ํ์ผ ๋์คํฌ๋ฆฝํฐ๋ฅผ ๊ฑด๋๋ฆฌ์ง ์์ต๋๋ค. ํด๊ฒฐ ๋ฐฉ๋ฒ์ ์ผํฐ๋ fd๋ฅผ ๋น ์ฐจ๋จ์ผ๋ก ํ์ํ๊ฑฐ๋ ๋จ์ํ epoll () ๋์ poll ()์ ์ฌ์ฉํ๋ ๊ฒ์ ๋๋ค. ํ์๋ฅผ ๋ก์ปฌ์์ ํ ์คํธํ๊ณ ์์ต๋๋ค.
์๋ค์ํผ poll ()์ ์ฌ์ฉํ๋ฉด ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋์ง ์์ต๋๋ค. ์ค์ ๋ฌธ์ ๋ ํ๋ธ์ ๋ฑ๋ก๋์ด์๋ ๋์ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ๊ณ ์๋ค๋ ๊ฒ์ ๋๋ค. ์ด๋์์ ์๋ชป๋์๋์ง ์ ํํ๊ฒ ํ์ ํ๋ ๊ฒ์ด ์ฝ๋ค๋ ๊ฒ์ ์๊ฒ๋๋ฉด ๋ค์ ํดํน์ผ๋ก์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์ ์์ต๋๋ค. ์ด๊ฒ์ด ์์ผ๋ฉด ํ๋ ์๊ฐ ์ด๋ด์ ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ต๋๋ค. ๊ทธ๊ฒ์ผ๋ก ์ฐ๋ฆฌ๋ ๊ทธ๊ฒ์ ์์ง ์ด๋ฐ์ํฌ ์ ์์์ต๋๋ค. ํดํน์ด ์ฌ๋ฐ๋ฅธ ์์ ์ ์๋์ง๋ง ์ค์ ๋ก ์ฝ๋๋ฅผ ๋ ์ ์๋ ์ฌ๋์ด ์ข์ ์์ ์ ์์ฑํด์ผํฉ๋๋ค.
--- /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py~ 2018-03-16 01:10:04.000000000 +0000
+++ /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py 2018-03-16 12:08:58.058686003 +0000
@@ -735,6 +735,7 @@
except KeyError:
pass
self.on_inqueue_close = on_inqueue_close
+ self.hub_remove = hub_remove
def schedule_writes(ready_fds, total_write_count=[0]):
# Schedule write operation to ready file descriptor.
@@ -1241,6 +1242,7 @@
if queue:
for sock in (queue._reader, queue._writer):
if not sock.closed:
+ self.hub_remove(sock)
try:
sock.close()
except (IOError, OSError):
์ฐ๊ธฐ ํ fd๊ฐ ํ๋ธ์์ ์ ๋๋ก ์ ๊ฑฐ ๋ ๊ฒ์ฒ๋ผ ๋ณด์ด์ง๋ง (๋ช ์ค ์๋ก) ์ฝ๊ธฐ ํ fd๋ ๊ทธ๋ ์ง ์์ต๋๋ค.
์ฌ๊ธฐ์ ์ ๋ฐ์ดํธ๊ฐ ์์ต๋๊น? :(
@auvipy https://github.com/celery/celery/issues/4185#issuecomment -373755206์์ ์์ @kleptog ๋ถ์์ ๋ํ ์๊ฐ์ด ์์ต๋๊น?
์ฃ์กํฉ๋๋ค. ์ง๊ธ ๋ฌธ์ ๋ฅผ ์กฐ์ฌ ํ ์๊ฐ์ด ์ถฉ๋ถํ์ง ์์ต๋๋ค.
@kleptog ์ ๋ํ ๋ง์
on_inqueue_close
_itself_๊ฐ hub_remove(fd)
ํธ์ถํ๋ ๊ฒ ๊ฐ์ต๋๋ค.-ํนํ L744
์์ ๋ค์๊ณผ ๊ฐ์ด ์๋ฅผ ๋ค๋ฉด ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
... ํธ์ถ ๋ destroy_queues()
์ L1247
๋ง์, queues[0]
.
์ด ์ง์ ์ด ์ค์ ๋ก ์ ํํฉ๋๊น?
ํดํน์ด ์ฌ๋ฐ๋ฅธ ์์ ์ ์๋์ง๋ง ์ค์ ๋ก ์ฝ๋๋ฅผ ๋ ์ ์๋ ์ฌ๋์ด ์ข์ ์์ ์ ์์ฑํด์ผํฉ๋๋ค.
๋๋ ์
๋ฌ๋ฆฌ ๋ด๋ถ์ ๋ํด ์ถฉ๋ถํ ์์ง ๋ชปํด์ ์ฝ๋๊ฐ ํ์ฌ queues[0]
๋ํด _only_ํ๋ ์ด์ ๋ฅผ ์ ์ ์์ต๋๋ค. ์ ์ด๋ diff๊ฐ์๋ ํ ๋ฆฌํ์คํธ์ ๊ฐ์น๊ฐ์์ ์ ์์ต๋๋ค.
IIRC๋ ์ค์ ๋ก ์ ์ธ ๋ ๋ค๋ฅธ ํ ์์ผ๋ฏ๋ก ๋ค๋ฅธ ํ์์ inqueue_close๋ฅผ ์ํํ๋ ๋ฌธ์ ์ผ๊น์? ๊ทธ๋๋ ์ ์ ํ ์คํธ ํ ๊ธฐํ๊ฐ ์์ต๋๋ค.
@auvipy ์์๋ฆฌ๋ฅผํ๊ฒ๋์ด ์ ๊ฐ์ ๋๋ค. ์คํ ์์ค ํ๋ก์ ํธ์์ ์์ ์์ ์ํ๋ ๊ณ ๋ง๊ฒ ์ฌ๊ธฐ๋ ์์ ์ ๊ณต๊ฐํ ์ ์์ต๋๋ค .๐-์ฌ๊ฐ ์๊ฐ์ ์ฌ๋ฌ ์์ ์ํ๊ณ ํ๋ฃจ ์ผ๊ณผ๋ฅผ ์ํด ๋ ๋ค๋ฅธ ์์ ์ํฉ๋๋ค.
๋น์ ์ด๋ ๋ค๋ฅธ ์ ๋ฌ๋ฆฌ ๊ด๋ฆฌ์๊ฐ ์ด๊ฑธ ์ฟ๋ณผ ๊ธฐํ๊ฐ ์์๋์ง ์์ญ๋๊น? ์ง๋ ์ฌ๋ฆ์๋ณด๊ณ ๋ ์ดํ๋ก์ด ์ ์ฝ ํด์ง๋ ๋ฌธ์ ๋ฅผ ๊ฒช๊ณ ์๋ ์ ๋ฌ๋ฆฌ ์ฌ์ฉ์๊ฐ์ด ์ค๋ ๋์์ ์ฃผ๋ชฉํ ๋งํ๊ณ , ์ฌ์ง์ด ๋ฒ์ธ ์ธ ์ฝ๋ ์์ญ์ ํ์ ํ๊ณ ์๋ณ ํ ์์๋ ์์์ ์ฌ์ฉ์๋ ์์ต๋๋ค. ์ด ๋ฒ๊ทธ๋ฅผ ์ธ์ ๋ณผ ์ ์๋์ง์ ๋ํ ์ผ์ข ์ ETA๊ฐ ์์ต๋๊น?
@ryanpetrello @kleptog ์ด ์ฝ๋๋ฅผ ๊ฐ๋ตํ ์ดํด ๋ณด์์ต๋๋ค. ์ฌ๊ธฐ์ ๋ช ๊ฐ์ง ์๊ฐ์ด ์์ต๋๋ค. ์ด๊ฒ์ ์์ ํ ์๋ก์ด ๊ฒ์ด๊ธฐ ๋๋ฌธ์ ๊ฒฐ์ ์ ์ด์ง ์์ต๋๋ค. ์ฃ์กํฉ๋๋ค.
์ด ์ปจํ
์คํธ์์ queues
๋ inq, outq, synq
์์ ๋ฐํ ๋ AsynPool.create_process_queues()
inq, outq, synq
ํํ์
๋๋ค. ๋ฐ๋ผ์ queue[0]
๋ ๊ตฌ์ฒด์ ์ผ๋ก inq
ํ๋ฉฐ on_inqueue_close
๋ผ๋ ํจ์๊ฐ์ด๋ฅผ
๋ฐ๋ผ์ ํ์ฌ hub_remove()
์ ์ ๋ฌ๋๋ ์ ์ผํ ๊ฒ์ inq._writer
์ด ์๋๋ผ _reader
์๋๋ผ outq
๋๋ synq
์๋๋๋ค proc
์ธ fileno_to_inq
์ ํค์ธ ๊ฒฝ์ฐ์๋ง ํด๋น๋ฉ๋๋ค.
๋๋ ์ด๊ฒ์ ๋ํด ๋ง์ ํต์ฐฐ๋ ฅ์ด ์์ต๋๋ค. ๊ทธ๋ฌ๋ fileno_to_outq
๋ฐ fileno_to_synq
๋์
๋๋ฆฌ๋ ์์ต๋๋ค. ๋ค๋ฅธ ํธ๋ค์์ hub_remove()
๋ฅผ ํธ์ถํ๊ธฐ ์ ์ ๋ด์ฉ์ ํ์ธํด์ผํฉ๋๊น? ์๋๋ฉด ๋ชจ๋ ๊ฒ์ ๋ฌด์กฐ๊ฑด hub_remove
ํด์ผํฉ๋๊น? ์ด์ฉ๋ฉด ์ฐ๋ฆฌ๋ ์ ๊ฑฐํด์ผํฉ๋๋ค hub_remove()
์์ ์ ํ on_inqueue_close()
๊ทธ๋ฅ์์ ํธ์ถ destroy_queues()
์์ DIFF ๊ฐ์ด?
๋ฟก๋ฟก
๋๋ผ์ด ์์
์ ๊ฐ์ฌ๋๋ฆฝ๋๋ค. ํ๋์์ด ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ผ๋ฉฐ ํ๋ฆฌ ํ์น ๋ ์์
์ด ์ญ์ ๋๋ ๊ฒ์ ์์คํ
์ ์ํ๊น์ด ์ํฅ์ ๋ฏธ์ณค์ต๋๋ค. FD์ ๊ฑธ๋ ค์๋ ๋
ธ๋์์ ๋
ธ์ ๊ฐ์ ๊ทผ๋ณธ ์์ธ์ ์ฐพ์์ง๋ง ๊ต์ฐฉ ์ํ๋ฅผ ์ฐพ์ ์ ์์์ต๋๋ค. ์์ ์ฌํญ์ ์ ์ฉํ๋ฉด ์ด์ ์คํธ๋ ์ค ํ
์คํธ๊ฐ ์ค๋จ์์ด 5 ๋ง ๊ฐ์ ์์
์ ์ฝ๊ฒ ์ฒ๋ฆฌํ์ต๋๋ค.
@AlexHill์ด ๋งํ์ต๋๋ค.
๋ค๋ฅธ ํธ๋ค์์ hub_remove ()๋ฅผ ํธ์ถํ๊ธฐ ์ ์ ๋ด์ฉ์ ํ์ธํด์ผํฉ๋๊น? ์๋๋ฉด ์ฐ๋ฆฌ๋ ๋ฌด์กฐ๊ฑด ๋ชจ๋ ๊ฒ์ hub_removeingํด์ผํฉ๋๊น? on_inqueue_close ()์์ hub_remove () ํธ์ถ์ ์ ๊ฑฐํ๊ณ ์์ diff ์์์ฒ๋ผ destroy_queues ()์์ ํธ์ถํด์ผํ ๊น์?
๊ฐ๋จํ ๋งํด์, ๋ซ๊ธฐ ์ ์ ํ๋ธ์์ ํ์ผ ์ค๋ช ์๋ฅผ ์ ๊ฑฐ ํด์ผํฉ๋๋ค. ๋ซ์๋ ํ๋ธ์์ ์ ๊ฑฐ๋์ง๋ ์๊ธฐ ๋๋ฌธ์ ๋๋ค. ๊ทธ๋์ ๋น์ ์ ๋ง์ง๋ง ์ ์์ ๋ "๋ถ๋ช ํ ์ณ๊ณ "๋๋ ๊ทธ๊ฒ์ ์ด๋ ฌํ ํฌ์ด๊ธฐ ๋๋ฌธ์ ์ค์ ๋ก ๊ฝค ์ข์ต๋๋ค.
strace๋ฅผ ์ฌ์ฉํ๋ฉด ๋ค์์ด ํ์๋ฉ๋๋ค.
epoll_ctl (5, EPOLL_CTL_DEL, 16, 0x7ffe54f25410) = -1 ENOENT (ํด๋น ํ์ผ ๋๋ ๋๋ ํ ๋ฆฌ ์์)
epoll_ctl (5, EPOLL_CTL_DEL, 21, 0x7ffe54f25410) = -1 ENOENT (ํด๋น ํ์ผ ๋๋ ๋๋ ํ ๋ฆฌ ์์)
fcntl (26, F_GETFL) = 0x2 (ํ๋๊ทธ O_RDWR)
fcntl (26, F_SETFL, O_RDWR | O_NONBLOCK) = 0
recvfrom (26, 0x7f42ca4409e4, 7, 0, NULL, NULL) = -1 EAGAIN (์ผ์์ ์ผ๋ก ์ฌ์ฉํ ์์๋ ๋ฆฌ์์ค)
์ด ๋ฌธ์ ์ ๊ด๋ จ์ด ์์ต๋๊น?
๊ฐ์ฌ
๋ค์ ๋งํฌ๋ก ๋ฌธ์ ๋ฅผ ์ง๋จํ๋ ๊ฒ์ด ๋์์ด ๋ ์ ์์ต๋๋ค. http://memos.ml/
Ubuntu 16์ผ๋ก ์ ๊ทธ๋ ์ด๋ ํ ์ดํ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
4.2.1์์ ๋์ผํ ๋ฌธ์ ์ ์ง๋ฉดํ์ต๋๋ค.
๋ค์๊ณผ ๊ฐ์ด ์์ผ ์๊ฐ ์ ํ ์ต์ ์ผ๋ก ์ ๋ฌ๋ฆฌ๋ฅผ ๊ตฌ์ฑํ์ญ์์ค.
celery.config_from_object ({ 'broker_transport_options': { 'socket_timeout': 30}})
๋๋ ๊ฐ์ ๋ฌธ์ ๊ฐ ์์๋ค. ๊ทธ๋ฌ๋ ํจ์น๊ฐ ์๋จ์ ๊ฒ์ ๋ ํ ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋์์ต๋๋ค.
๋๋ ๋ํ์ด ๋ฌธ์ ๊ฐ ์์ต๋๋ค. @ kleptog ์ ์์ ์ ๋ํ ํ ์์ฒญ์ด ์์ต๋๊น? ์๋๋ฉด์ด ๋ฌธ์ ๊ฐ ํด๊ฒฐ ๋ ๋๊น์ง ์ฌ๋๋ค์ด ํฌํฌ๋ฅผ ์ฌ์ฉํ๊ณ ์์ต๋๊น?
tcp ์์ผ ์ฐ๊ฒฐ ์ ์ง ๋ฐ ์๊ฐ ์ด๊ณผ (redis ์ฌ์ฉ)๋ฅผ ์ถ๊ฐ ํ ํ ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋์์ต๋๋ค.
BROKER_TRANSPORT_OPTIONS = {'socket_timeout': 60,
'socket_connect_timeout': 120,
'socket_keepalive': True,
'socket_keepalive_options': {socket.TCP_KEEPIDLE: 60,
socket.TCP_KEEPINTVL: 60,
socket.TCP_KEEPCNT: 3}}
@ linar-jether : socket.TCP_KEEPIDLE
์๋์ด ์์ผ์ ๋ฌด์์
๋๊น?
@ nishith107 import socket
redis ๋ธ๋ก์ปค์ ๊ตฌ์ฑ์ ๋ณ๊ฒฝํ๋ ๊ฒ์์ด ์ค๋ ๋์ ๋ฌธ์ ์ ๊ฑฐ์ ๊ด๋ จ์ด ์์ต๋๋ค. ์ด๋ ๋ง์คํฐ-์์ ํต์ ํ์ดํ์์ ์ฝ๊ธฐ ์ฐจ๋จ ๋ฌธ์ ์ ๊ดํ ๊ฒ์ ๋๋ค.
๋ง์ต๋๋ค. ๋ธ๋ก์ปค ์ค์ ์ ์์ ํ๋ฉด ๊ทผ๋ณธ์ ์ธ ๋ฌธ์ ๋ฅผ _ ๋ฐฉ์ง _ํ๋ ๋ฐ ์ถฉ๋ถํ ์งํฐ๊ฐ ๋ฐ์ํ ์ ์์ง๋ง ๋ง์คํฐ-์์ IPC ์ค๋จ์ ๋๋ ทํ ๋ฒ๊ทธ์ ๋๋ค.
@auvipy ์ด๊ฒ์ ์๋ 1 ๋
์ ์ ์ค๋๋ณด๊ณ ๋์์ต๋๋ค. ์ ์ ๋ค๋ฅธ ์ฌ๋๋ค์ ์ฝ๋๊ฐ ๋ช ์ค๋ก ์ขํ์ง strace
์ถ๋ ฅ์ ํฌํจํ์ฌ ์ฝ๋๊ฐ ์ค๋จ ๋ ๊ฒ์ฒ๋ผ ๋ณด์ด๋ ์์น์ ๋ํ ์์ธํ ์ ๋ณด์ ๋ถ์์ ๋จ๊ฒผ์ต๋๋ค. ๋๊ตฐ๊ฐ๊ฐ ํจ์น์ ๋ํ ๋งํฌ (
https://github.com/celery/celery/issues/4185#issuecomment-373755206) ์ฌ๋๋ค์ด ํด๊ฒฐ ๋ฐฉ๋ฒ์ผ๋ก (ํ๋ก๋์
์์?) ์ฌ์ฉํ๊ณ ๋ฐฐํฌํ๋ ๊ฒ ๊ฐ์ต๋๋ค. ์ด ๋๊ธ ์ค๋ ๋์ ๊ด๋ จ๋ ๋ง์ ์ฌ๋๋ค์ ์ฌ์ ํ โโ๋ค์ํ ์
๋ฌ๋ฆฌ ๋ฒ์ ๋ฐ ๋ธ๋ก์ปค ์ค์ ์์์ด ๋ฌธ์ ๋ฅผ ์ ๊ธฐ์ ์ผ๋ก ๊ฒช๋ ๊ฒ ๊ฐ์ต๋๋ค.
๋๋ ์ด๊ฒ์ด ์คํ ์์ค ํ๋ก์ ํธ๋ผ๋ ๊ฒ์ ๊ฐ์ฌํ๊ณ , ๊ด๋ จ๋ ๋ชจ๋ ์ฌ๋๋ค์ด ์์ ๋ด์ฌ์์ด๊ธฐ ๋๋ฌธ์ ๋์์ด ์์๋ฆฌ๋ฅผํ๋ ค๋ ๊ฒ์ด ์๋๋๋ค. _anyone_์ด์ด ์ค์ํ ๋ฌธ์ ๋ฅผ ์กฐ์ฌํ๊ฑฐ๋ ์ง๋จํ๊ธฐ ์ํด ๋ ธ๋ ฅํ๋์ง ์๊ณ ์์ต๋๊น?
์ฐ๋ฆฌ ์ค ๋๊ตฐ๊ฐ๊ฐ ํ๋ก๋์ ์์ ์ง๋ฉดํ๋ฉด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ๊ฒ์ด ๋ ์ฌ์ธ ๊ฒ์ ๋๋ค. ์ฌ๋ฌ๋ถ ์ค ์ผ๋ถ๋์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๊ธฐ ์ํด ๋ช ๊ฐ์ง ํด๊ฒฐ ๋ฐฉ๋ฒ์ ์ฌ์ฉํ๊ธฐ ๋๋ฌธ์ ์ ์ ํ ํ ์คํธ ์ฌ๋ก์ ํจ๊ป ํ๋ณด๋ฅผ ๋ณด๋ด์ง ์์ผ์๊ฒ ์ต๋๊น? ์ ๋ฌ๋ฆฌ๋ 5 ~ 6 ๋ ๋ ๋ฌธ์ ๋ ์์ต๋๋ค.
์ ๋ฅผ ๋ฏฟ์ผ์ญ์์ค. ๋ชจ๋ ๋ฌธ์ ์ ๋์ฒํ๊ธฐ๊ฐ ์ ๋ง ์ด๋ ต์ต๋๋ค. ์ค์ ํ๊ฒฝ์์ ์ ์ ๋ ๋ณ๊ฒฝ ์ฌํญ์ ๋ ์ฝ๊ฒ ํ ์คํธ ํ ์ ์๋๋ก ์ผ๋ถ ์ฌ์ฉ์๊ฐ ํ๋ก๋์ ์์ ์ง๋ฉดํ๊ณ ์์ผ๋ฏ๋ก ์์ ์ฌํญ์ ๋ณด๋ด์ญ์์ค.
@ linar-jether ๊ฐ์ฌํฉ๋๋ค,์ด ์์ ์ ์ ์๊ฒ ํจ๊ณผ์ ์ด์์ต๋๋ค :)
์ด PR https://github.com/celery/celery/pull/4997์ ํ์ธํ ์ ์์ต๋๊น? ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋๋ฉด ๋ณํฉ ํ ์ ์์ต๋๋ค.
์ด ์ง์ ์ ์ํํด ๋ณผ๊ฒ์. ๊ฐ์ ๋ก ๋ฐ์์ํค๋ ๋ฐฉ๋ฒ์ ์ฐพ์ง ๋ชปํ๊ธฐ ๋๋ฌธ์ ์ ๋ ฌ๋์๋์ง ํ์ธํ๋ ๋ฐ ์ฝ๊ฐ์ ์๊ฐ์ด ๊ฑธ๋ฆฝ๋๋ค.
์ด ๋ฒ๊ทธ ์์ ํ ์คํธ์ ๋ํ ์์์ด ์์ต๋๊น?
์ง๊ธ๊น์ง 2 ์ฃผ ์ด์ ๋ฌธ์ ๊ฐ ์์์ต๋๋ค. ๋๋ ์์ง 100 %๋ ์๋์ง๋ง ์ ์ ๋ PR์ด ๋๋ฅผ ์ํด ๊ทธ๊ฒ์ ๊ณ ์ณค๋ค ๊ณ ์๊ฐํฉ๋๋ค.
์ด ํจ์น๊ฐ ์ ์ฉ๋ 4.2 ๋ธ๋์น๋ฅผ 3 ๊ฐ์ ํด๋ผ์ด์ธํธ ๋จธ์ ์ ์ปค์คํ ํ ๋ก ์ ์ฉํ์ต๋๋ค. ํ์ฌ ์์ฐ ์ค์ ๋๋ค. ์ด ๋ฒ๊ทธ๋ ๋ช ๋ฌ ๋์ ์ผ์ฃผ์ผ์ ํ ๋ฒ ์์ ์๋ฅผ ์ง์ฒด ์์ผฐ์ต๋๋ค. ๋ฐฐํฌ๋์์ต๋๋ค. ๊ทธ๊ฒ์ด ๋ฌธ์ ๋ฅผ ์๋ฐฉํ๋์ง ๋ณผ ๊ฒ์ ๋๋ค.
๊ทธ๋์ ๋๋ pr https://github.com/celery/celery/pull/4997์ ์ฌ๊ธฐ์ ๊ฒ์ ํด์ฃผ์๊ณ , ๋จ์ / ํตํฉ ํ ์คํธ ์ ์์ด ์์ผ์๋ฉด PR์ ์งํ ํด์ฃผ์ธ์.
@ryanpetrello ๊ถ๊ธํด,์ด ํจ์น๋ฅผ ๋ฐฐํฌํ๊ณ ํ ์คํธ ํ์ต๋๊น?
@AvnerCohen ์ด ๋ฌธ์ ์ ๋ช ๊ฐ์ง ๋ค๋ฅธ ๋ฌธ์ ๋ก ์ธํด ์ ๋ฌ๋ฆฌ๋ฅผ ์ฌ์ฉํ์ง ์๊ณ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ์ต๋๋ค (์ฐ๋ฆฌ๋ ํ๋ฒํ ์ค๋๋ ๋ค์๋ง๋ก ํ์ํ ๊ฒ์ ๋ฌ์ฑ ํ ์์์์ต๋๋ค). ํ์ง๋ง ๋ค๋ฅธ ์ฌ๋๋ค์ด์ด ํจ์น๋ก ์ด์ด ์ข์๋ ๊ฒ ๊ฐ์ต๋๋ค!
์ง๊ธ๊น์ง๋ ํ๋ก๋์ ์ ๋ฐฐํฌ ํ ์ดํ๋ก ๋ ์ด์ ๋ฌธ์ ๋ฅผ ์ธ์ํ์ง ๋ชปํ์ต๋๋ค. ๋๋ ๊ทธ๊ฒ์ ๊ณ ์ณค๋ค ๊ณ ์๊ฐํฉ๋๋ค.
๊ฐ์ฌํฉ๋๋ค!
@auvipy ์ด ํจ์น๊ฐ ํฌํจ ๋ ๊ณต์ ๋ฆด๋ฆฌ์ค๋ ์ธ์ ๊ธฐ๋ํด์ผํฉ๋๊น?
๋ด ๋ชฉํ๋ ํฌ๋ฆฌ์ค๋ง์ค ์ด์ ์ ๋๋ค
ํ๋ฅญํ ๋๋ฒ๊น ์ ์ํด @kleptog ์๊ฒ ๊ฐ์ฌ๋๋ฆฝ๋๋ค!
Celery์ ๋ง์คํฐ์ ์น๊ตฌ๋ค์ ์คํํ๊ธฐ๊น์ง ์ ์ ๋ ๋ชจ๋ ์์ ์ฌํญ์ ์๋ํ์ง๋ง ์์ ์๊ฐ ConnectionResetError๋ฅผ ๋ง๋๋ฉด ๋ ์ด์ ์์ ์ ์ฒ๋ฆฌํ์ง ์์ต๋๋ค. RabbitMQ๋ฅผ ๋ค์ ์์ํ๋ฉด ๋ฉ์ถ ์์ ์๊ฐ ์์ ์ ๋ค์ ์ฒ๋ฆฌํ๊ฒ๋ฉ๋๋ค. ๋น์ฐํ ์ด๊ฒ์ ์ข์ ํด๊ฒฐ์ฑ ์ ์๋์ง๋ง ๋ ๋๋ฒ๊น ํ ๋ฐฉํฅ์ ์๋ ค์ค๋๋ค.
์ด ํจ์น๋ฅผ ์ฌ์ฉํ๊ณ ์์ง๋ง ๊ฐ๋์ฉ์ด ๋ฌธ์ ๊ฐ ๋ฐ์ํฉ๋๋ค. ์์ ์๊ฐ ์์ ์ ์ฑ๊ณต์ ์ผ๋ก ์๋ฃํ๊ณ ์ฃผ์ ํ๋ก์ธ์ค๊ฐ ์ ๊ธฐ ๋ฉฐ ๊ฒฐ๊ณผ๋ฅผ ๊ธฐ๋ค๋ฆฌ๋ ์ค์ ๋๋ค.
๋๋ celery == 4.2.1์ ์ฌ์ฉํ๊ณ ์์ต๋๋ค; python == 3.6.7 ์ฐ๋ถํฌ 16.04. ๋ฐฑ์๋๋ก Redis
๊ฐ๋
์๋ฅผ ์ฌ์ฉํ์ฌ ๋ช
๋ น์ผ๋ก ์
๋ฌ๋ฆฌ๋ฅผ ์์ํ์ต๋๋ค.
celery-ํ๋ก์ ํธ ์์
์ -l info --autoscale = 10,2
๋๋ก๋ ์ฌ์ ํ ๋ฉ์ท์ต๋๋ค.
@mikolaje ๋๋ฅผ ์ํด ์๋ํ๋ ๊ฒ์ ๊ต์ฐฉ ์ํ๋ฅผ ํด๊ฒฐํ๊ธฐ ์ํด redis.conf์์ tcp-keepalive
์ ์ค์ ํ๋ ๊ฒ์
๋๋ค.
๊ฐํ ์์ ์์ ์ํฉ์ ๋ชจ๋ํฐ๋งํ๋ ์ข์ ๋ฐฉ๋ฒ์ ์ฐพ์ ์ฌ๋์ด ์์ต๋๊น?
์ฐ๋ฆฌ๋ ์ฌ์ ํ ๋ง์คํฐ์ ์ฝ๋๋ก์ด ๋์์๋ณด๊ณ ์์ต๋๋ค. ์ด๊ฒ์ด ๊ฐ์ ๋์๋์ง ์ฌ๋ถ๋ ๋งํ ์ ์์ง๋ง, ์ด๊ธฐ ๋๋์ ์ ์ด๋ ์ฐ๋ฆฌ ์ฌ์ฉ ์ฌ๋ก์์๋ ์ฌ์ ํ ์์ ์๊ฐ ๊ฐํ ์๋ค๋ ๊ฒ์ ๋๋ค.
์ง๋ฌธ์ ๊ฐ๋ณ ์์
์๋ฅผ ๋ชจ๋ํฐ๋งํ๋ ๋ฐฉ๋ฒ์
๋๋ค. ์ง๊ธ ์ฐ๋ฆฌ๊ฐ ํ ์์๋ ์ผ์ ๋๊ธฐ์ด ์์ค (๋ฉ์์ง๊ฐ ์๊ณ ์ ํ๋์ง ์์)์ ๋ชจ๋ํฐ๋งํ๋ ๊ฒ ๋ฟ์ด์ง ๋ง ์์
์๊ฐ ๋ ์ด์ ์์
์ ์ํํ์ง ์๋์๊ธฐ๋ฅผ ํ์ธํ์ฌ ํจ์ฌ ๋ ์ผ์ฐ ์ถ์ ํ๊ณ ์ถ์ต๋๋ค.
๋ด๊ฐ ์์ ์ฐจ๋ฆฐ ๊ฒ์ celery "status"๋ช
๋ น์ด ์๊ด์์ด OK๋ฅผ ๋ฐํํ๊ณ ๋ชจ๋ ๊ฒ์ด ์ข๋ค๋ ๊ฒ์ ๋ณด์ฌ์ฃผ๋ ๊ฝ ์ด๋ผ๋ ๊ฒ์
๋๋ค.
Celery์ ๋ง์คํฐ์ ์น๊ตฌ๋ค์ ์คํํ๊ธฐ๊น์ง ์ ์ ๋ ๋ชจ๋ ์์ ์ฌํญ์ ์๋ํ์ง๋ง ์์ ์๊ฐ ConnectionResetError๋ฅผ ๋ง๋๋ฉด ๋ ์ด์ ์์ ์ ์ฒ๋ฆฌํ์ง ์์ต๋๋ค. RabbitMQ๋ฅผ ๋ค์ ์์ํ๋ฉด ๋ฉ์ถ ์์ ์๊ฐ ์์ ์ ๋ค์ ์ฒ๋ฆฌํ๊ฒ๋ฉ๋๋ค. ๋น์ฐํ ์ด๊ฒ์ ์ข์ ํด๊ฒฐ์ฑ ์ ์๋์ง๋ง ๋ ๋๋ฒ๊น ํ ๋ฐฉํฅ์ ์๋ ค์ค๋๋ค.
@hedleyroos , ๋น์ ์ ๋ํ์ด ๋ฌธ์ ๊ฐ์์ ์ ์์ต๋๋ค # 4895? ์ด ๋ฌธ์ ๋ฅผ ํผํ๊ธฐ ์ํด Celery v4.1.x๋ฅผ ์คํํ๊ณ ์์ต๋๋ค.
@kingb ์, ๊ทธ ๋ฌธ์ ๊ฐ ์์ต๋๋ค. 4.1๋ก ๋ค์ด ๊ทธ๋ ์ด๋ํ๋ ๊ฒ์ ํผํ๊ณ ๋ธ๋ก์ปค๋ก Redis๋ก ์ ํํ์ง๋ง ์ฌ์ ํ ๋์ผํ ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ต๋๋ค. ๋คํํ Redis์๋ 60 ์ด๋ก ์ค์ ํ tcp-keepalive
์ต์
์ด ์์ต๋๋ค. ์ด๊ฒ์ Redis๊ฐ ๋ถ์ค ์ฐ๊ฒฐ๋ก ๊ฐ์ฃผ๋๋ ๊ฒ์ ์ข
๋ฃํ๊ณ ๊ฒฐ๊ณผ์ ์ผ๋ก ์์
์๊ฐ ๊ต์ฐฉ ์ํ๋ฅผ ๋ฒ์ด๋ ์์
์ ๋ค์ ์๋นํ๊ธฐ ์์ ํจ์ ์๋ฏธํฉ๋๋ค.
celery == 4.2.1, billiard == 3.5.0.5, kombu == 4.2.2.post1, amqp == 2.4.0, Python 3.6.8, RabbitMQ, MongoDB v 4์ ๋ํด ๋๊ฒฐํ์ต๋๋ค.
@tonal ๋๋ ๋น์ ๊ณผ ๊ฐ์ ๋ฒ์ ์ ์ฌ์ฉํ๊ณ ์์ผ๋ฉฐ ๋๋๋ก์ด ์ค๋ฅ๊ฐ ๋ฐ์ํฉ๋๋ค. ์์ ์ด๋ ๊ทธ์ ๋น์ทํ ๊ฒ์ ์ฌ ์๋ํ์ง ์์ต๋๋ค. ์์ ์ด RECIEVED ์ํ์์ ๋ฉ ์ถฅ๋ ๋ค.
ํญ์ ๋ฐ์ํ์ง๋ ์์ง๋ง ๋ช ์๊ฐ / ์์ผ๋ง๋ค ๋ฐ์ํ๋ฉฐ ๋ฉ์์ง๋ฅผ ์ ๊ฑฐํ๊ณ ์๋น์ค๋ฅผ ๋ค์ ์์ํฉ๋๋ค.
4.3? 4.4rc2?
์๋์, celery==4.2.1, billiard==3.5.0.5, kombu==4.2.2.post1, amqp==2.4.0,
@auvipy๋ ์ฌ์ ํ 4.3์์ 1 ์๊ฐ ๋์ ์ฃฝ๋ ๊ฒ์๋ณด๊ณ ์์ต๋๋ค. ๋์์ด๋๋ค๋ฉด ์ฑ๊ณต / ์คํจ ์์ ์ ์ค๋จํ๊ณ ๊ณ์ ์์ ํ ๋ค์ ์กฐ์ฉํ ์คํจํ์ง๋ง ์ ๋ฌ๋ฆฌ ํ๋ก์ธ์ค๋ ๊ณ์ ์คํ๋ฉ๋๋ค.
/code # celery status
celery@celery-worker: OK
1 node online.
๋๊ตฐ๊ฐ 4.3์์ ์ค์ ๋ก ํด๊ฒฐ๋์๋ค๊ณ ํ์ ํ ์ ์์ต๋๊น? ์ด๋ฅผ ๋ฐํ์ผ๋ก ์ ๊ทธ๋ ์ด๋ ๊ฒฐ์ ์ ๋ด๋ ค์ผํฉ๋๋ค.
์๋ ํ์ธ์,
์ด๊ฒ์ 4.3์์ ์์ ๋์ง ์์์ต๋๋ค. ํ๋ก์ ํธ์์์ด ๋ฒ์ ์ ์ฌ์ฉํ๊ณ ์๋๋ฐ ์์ ์๊ฐ ์ฌ์ ํ ๋ฉ ์ถฅ๋ ๋ค ( "ํํธ ๋นํธ ๋๋ฝ"๋ฉ์์ง ํ์ ํ).
4.3์์์ด ๋ฌธ์ ๋ฅผ ์ฐํ ํ ์ฌ๋์ด ์์ต๋๊น? ์ฝ๋๊ฐ ์์ ํด๊ฒฐ ๋ฐฉ๋ฒ / ํจ์น์ ํฌ๊ฒ ๋ค๋ฅด๋ค๋ ๊ฒ์ ์์์ผ๋ฏ๋ก ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ค๋ ์๋๊ฐ ์์์ง๋ง ํด๊ฒฐ๋์ง ์์ ๊ฒ ๊ฐ์ต๋๋ค.
๊ฐ์ฌ!
Celery 4.3 (Python 3.7.3)์์ ๋์ผํ ๋ฌธ์ ๊ฐ ์์์ ํ์ธํ ์ ์์ต๋๋ค.
celery == 4.4.0rc3๋ฅผ ์ฌ์ฉํด ๋ณผ ์ ์์ต๋๊น?
@auvipy๋ ์๋ํ์ง๋ง ๋์์ด๋์ง ์์์ต๋๋ค. ์ฐธ๊ณ ๋ก gevent
์ด ๋ฌธ์ ๊ฐ ๋ค์ ์ด๋ฆด ์ ์์ต๋๊น? ์ฌ๋๋ค์ด ์ ๋ฌ๋ฆฌ์ ์ต์ ๋ฒ์ ์์ ์ฌ์ ํ ๊ทธ๊ฒ์ ์ ํ๋ ๊ฒ์ฒ๋ผ ๋ค๋ฆฝ๋๋ค.
๋๋ ๊ฐ์ ๋ฌธ์ ์ ์ง๋ฉดํ๊ณ ์์ต๋๋ค. Python2.7, Celery4.1.0. SQS ๋ธ๋ก์ปค๋ฅผ ์ฌ์ฉํฉ๋๋ค. ์ ๋ฌ๋ฆฌ๋ ์ผ์ ์๊ฐ ๋์ ์์ ์ ์ฒ๋ฆฌํ๊ณ ์ ์ ํ ๊ฐ์๊ธฐ ์ฐจ๋จ๋ฉ๋๋ค.
๋๋ ๊ฐ์ ๋ฌธ์ ์ ์ง๋ฉดํ๊ณ ์์ต๋๋ค. Python2.7, Celery4.1.0. SQS ๋ธ๋ก์ปค๋ฅผ ์ฌ์ฉํฉ๋๋ค. ์ ๋ฌ๋ฆฌ๋ ์ผ์ ์๊ฐ ๋์ ์์ ์ ์ฒ๋ฆฌํ๊ณ ์ ์ ํ ๊ฐ์๊ธฐ ์ฐจ๋จ๋ฉ๋๋ค.
์ต์ ๋ฆด๋ฆฌ์ค๋ฅผ ์๋ํด์ผํฉ๋๋ค.
์ฐ๋ฆฌ๋์ด ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ผ๋ฏ๋ก ๋๊ตฐ๊ฐ ์ฐ๋ฆฌ๋ฅผ ์ํด ๋๋ฒ๊น ํด์ผํ๋ฉฐ ๊ฐ๋ฅํ ๊ฒฝ์ฐ์ด ๋ฌธ์ ์ ๋ํ ํด๊ฒฐ์ฑ ์ ์ ๊ณตํด์ผํฉ๋๋ค.
์ฐ๋ฆฌ๋ ์ด๊ฒ์ ๋งค์ผ ์์ฐํ๊ณ ์์ต๋๋ค. ์ฐ๋ฆฌ๋ ๊ฐ๋ฐ ๋๋ ๋๋ฒ๊ทธ ์ง์ ํ๊ฒฝ์์ ๋ค์ ๋ง๋ค์ง ๋ชปํ์ต๋๋ค.
์ด๋ค ์ข
๋ฅ์ ์ ๋ณด๋ฅผ ์ ๊ณต ํ ์ ์๋์ง์ ๋ํ ์ ์์ด ์์ต๋๊น?
python-hunter ๋ฐ / ๋๋ python-manhole ๊ณผ ๊ฐ์ ๊ฒ์ ์ฌ์ฉํ๋ ๊ฒ์ด ์ข์ต๋๋ค.
๋ง์คํฐ ํ๋ก์ธ์ค์ ๋ฌธ์ ๊ฐ์์ ์ ์์ต๋๋ค.
๋๋ strace๋ฅผ ์ฌ์ฉํ์ฌ ์ฝ๊ฑฐ๋ ์ฐ๋ ์์ผ / ํ์ดํ๋ฅผ ํ์ธํ๊ณ ๊ฑฐ๊ธฐ์์ ๊ณต์ ํด๋ณด์ญ์์ค.
์ํ์๋ฉด ๋ค์ ์ฃผ์ ์ฌ๋ฌด์ค์ ์ฌ ์ ์์ต๋๋ค. ์ ํ๋ก ์๊ธฐํฉ์๋ค.
๊ฐ์ฌํฉ๋๋ค. ์ผ๋ฐ์ ์ธ ์์ ๋ง์ฐฌ๊ฐ์ง๋ก PS์ ๊ด์ ์์ ํ๋ก์ธ์ค๊ฐ ๋ฉ์ถฐ์๋ ๋ชจ์ต์ ๋๋ค.
username 14182 1 1 03:12 ? 00:02:28 [celeryd: workername<strong i="6">@hostname</strong>:MainProcess] -active- (moshe)
username 15487 14182 0 03:15 ? 00:00:12 [celeryd: workername<strong i="7">@hostname</strong>:ForkPoolWorker-5]
username 15766 14182 0 03:18 ? 00:00:12 [celeryd: workername<strong i="8">@hostname</strong>:ForkPoolWorker-14]
username 16666 14182 0 03:52 ? 00:00:10 [celeryd: workername<strong i="9">@hostname</strong>:ForkPoolWorker-43]
username 20253 14182 65 06:45 ? 00:00:24 [celeryd: workername<strong i="10">@hostname</strong>:ForkPoolWorker-110]
๋์์ฑ์ 4๋ก ์ค์ ๋๊ณ "max-tasks-per-child"๊ฐ ์ค์ ๋์ด X ๋ฉ์์ง๋ง๋ค ๋ค์ ์์๋ฉ๋๋ค.
๋ถ๋ช
ํ ์ ์ ์๋ฏ์ด 20253
์ (๋) ์ฌ์ ํ ํ์ฑ ์ํ์ด๋ฉฐ ์์
์ ์นด์ดํฐ๋ ๊ณ์ ์ฆ๊ฐํ๊ณ ์์ต๋๋ค. 15487
์ (๋) ์ด๊ธฐ์ ์ฌ๋งํ์ผ๋ฉฐ ๋ค์๊ณผ ๊ฐ์ ๋ฌธ์ ๋ฅผ ๊ฒช๊ณ ์์ต๋๋ค.
$ strace -p 15487
strace: Process 15487 attached
futex(0x2b36980, FUTEX_WAIT_PRIVATE, 0, NULL
๋ง์คํฐ๋ ์ฌ์ ํ ํ์ฑ ์ํ์ด๊ณ ๋ฉ์์ง๋ ์ฌ์ ํ ํ์ฑ ์ํ ์ธ ํ ์์ ์์๊ฒ ์ ์ก๋ฉ๋๋ค.
๋ค๋ฅธ ์ฌ๋์๊ฒ ๋์์ด ๋ ์ ์๋๋ก ์ฌ๊ธฐ์ ๋จ๊ฒจ ๋์ธ์. ๋๊ฐ์ ๋ฌธ์ ๊ฐ ์๋์ง ํ์คํ์ง ์์ง๋ง ์ง์๋ค๋ ์์ญ๋ง ๊ฐ์ ์์ ์ ์ฒ๋ฆฌ ํ ํ ์ค๋จ๋์์ต๋๋ค. ๋ค์์ ์ฌ์ฉํ์ฌ Celery๋ฅผ ์๋น์ค๋ก ์ถ์ํ์ต๋๋ค.
celery multi start w1 w2 -A app.celery --time-limit=21600 --concurrency=4
์์ ์๊ฐ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ์ง ๋ชปํ ํ ๋ช ๋ น์ ์ผ๋ฐ์ผ๋ก ๋ณ๊ฒฝํ์ต๋๋ค.
celery worker -A app.celery --time-limit=21600 --concurrency=4
๊ทธ๋ฆฌ๊ณ ์ด๊ฒ์ ๋๋ฅผ ์ํด ๊ทธ๊ฒ์ ๊ณ ์ณค์ต๋๋ค! ์ด ๋ฐฉ๋ฒ์ผ๋ก ์ ๋ฌ๋ฆฌ๋ฅผ ์๋์ผ๋ก ์คํํ์ ๋ ์์ ์ด ๋ค์ด์ค๊ณ ์คํ๋๊ณ ๊ฐ๋ ์์ ์ด ์ค๋จ๋๋ ๊ฒ์ ๋ณผ ์์์์ต๋๋ค (๊ทธ๋ฆฌ๊ณ ์์ ์๊ฐ ์ ๋๋ก ๋ค์ ์์๋จ). ์ด์ ๋ฐฉ์์ ์์ ์ด ์คํจํ ํ ์์ ์๋ฅผ ์ ๋๋ก ๋ค์ ์์ํ์ง ์์ ๊ฒ ๊ฐ์ต๋๋ค. ๋ด๊ฐ ๋ง๋ ๋ ๋ค๋ฅธ ๋ณ๊ฒฝ ์ฌํญ์ ์๋ฐฑ / ์์ฒ ๋ฒ ์คํ๋๋ ์์ ์ ๋ํด ๋ ๋ฎ์ ์๊ฐ ์ ํ์ ์ค์ ํ๋ ๊ฒ์ด ์์ต๋๋ค. ๋ฐ๋ผ์ ์์ ์๊ฐ ๋ถ๊ฐํผํ๊ฒ ๋ฉ์ท์ ๋ ์ฃผ ์์ ์ ํ๋ก์ธ์ค๋์ด๋ฅผ ์ฃฝ์ด๊ณ ์ ์์ ์ ์๋นํ ๋น ๋ฅด๊ฒ ๋ค์ ์์ํ ์ ์์ต๋๋ค. ๊ทธ๋ค ๋ชจ๋๊ฐ ๊ฐํ๊ธฐ ์ ์.
์ด ๋ฌธ์ ๋ฅผ ํจ๊ณผ์ ์ผ๋ก ๋๋ฒ๊น ํ ์์๋ ์ ๋ฌ๋ฆฌ ๋ด๋ถ์ ๋ํด ์ถฉ๋ถํ ์์ง ๋ชปํฉ๋๋ค.ํ์ง๋ง ๋๊ตฐ๊ฐ๊ฐ ์ง์นจ์ด๋ ๋ณผ ์์๋ ์ฅ์๋ฅผ ์ ๊ณต ํ ์ ์๋ค๋ฉด ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ ๋ Google ์ธก์์ ์กฐ์ฌ ํ ์ ์์ต๋๋ค.
์์ฒญ์์ด ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์๋ ์์ง๋ง ์๊ฐ์ด ์ง๋จ์ ๋ฐ๋ผ ๋น์ฌ ์ฌ์ดํธ ์ค ํ๋์์ ๋ฐ์ํฉ๋๋ค. ์์ ์๊ฐ ๋ถํ๋ฅผ๋ฐ๋ ๋์ ์์ ์์ RabbitMQ ์๋ฒ ์ฌ์ด์ ๋คํธ์ํฌ ๋๋กญ์ด์์ ๋ ๋ฌธ์ ๋ฅผ ์ถ์ ํ ์์์์ต๋๋ค. ์คํจ๋ ๊ธฐ๋ณธ์ ์ผ๋ก ์๋ก์ด ์์ ์ ์ํํ์ง ๋ชปํ๋๋ก ๊ธฐ๋ณธ ํ๋ก์ธ์ค๋ฅผ ์ ๊ธ๋๋ค.
์์คํ ์ ๋ถํ๊ฐ ๊ฑธ๋ฆฌ์ง ์์ผ๋ฉด ๋ชจ๋ ๊ฒ์ด ๊ณ์ ์ ๋๋ก ์๋ํฉ๋๋ค. ์์ ์์ ๋ํ ๋ถํ์ ๋์์ ๋คํธ์ํฌ ์ ํ์ ์กฐํฉ์ด์ด์ผํฉ๋๋ค. ์ด๋ฅผ ํตํด ๋ ๊ฐ์ง ๋ฐฉ๋ฒ์ด ํ์ธ๋์์ต๋๋ค. ํ๋๋ ๋น์๋ก๋๊ฐ์๋ ๋ณด์กฐ ์คํ ์ด์ง ์์คํ ์ด ํผํฉ ์คํจ๋ฅผ ํ์ธํ์ง๋ง ์ดํ์๋ ๊ณ์ ์ ๋๋ก ์๋ํฉ๋๋ค. ๋์งธ, ์ฐ๋ฆฌ๋ ์์ ๋น ์ฌ๋ฌ ์์ ์๋ฅผ ์คํํฉ๋๋ค. ์์ ์ ์คํ์ค์ธ ์์ ์ ๋ง ์คํจํฉ๋๋ค. ๋ถํ๊ฐ ์ ๊ณ ๋ชจ๋ ์ฃผ์ ํ๋ก์ธ์ค๊ฐ ์์ ์ ์ฒ๋ฆฌํ์ง ์๋ ๊ฒฝ์ฐ ์ผ๋ถ ์ฃผ์ ํ๋ก์ธ์ค ๋ง ์คํจํฉ๋๋ค.
์์ ๋ช
๋ น :
celery worker -n XXX-%I.${CELERY_HOST_NAME} \
-A XXX.app --pidfile=/run/celery/XXX-%I.pid \
--autoscale=10,3 \
--max-tasks-per-child 1\
-Q YYYl \
--loglevel=INFO --logfile=/var/log/XXX/XXX-%I${CELERY_PROCESS_INDEX_SEPERATOR}.log --soft-time-limit=7200 --time-limit
๋ก๊ทธ ์ถ๋ ฅ :
{"event": "mingle: sync with 54 nodes", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: processing reply from celery@XXXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "mingle: processing reply from celery@XXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: sync complete", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Gossip", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 2", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Tasks", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Control", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 3", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting event loop", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Autoscaler...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Pool...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "basic.qos: prefetch_count->3", "logger": "kombu.common", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1] ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', {'lang': 'py', 'task': 'XXX', 'id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"xxx}\", 'origin': 'gen49433<strong i="17">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09] ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', {'lang': 'py', 'task': 'XXX', 'id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{'xxx}\", 'origin': 'gen49438<strong i="18">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020] ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', {'lang': 'py', 'task': 'XXX', 'id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{xxx}\", 'origin': 'gen49438<strong i="19">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "celery<strong i="20">@XXX</strong> left", "logger": "celery.worker.consumer.gossip", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09] pid:111546", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1] pid:111544", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020] pid:111538", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "missed heartbeat from celery@XXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "missed heartbeat from celery@XXXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec22d68>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c5f8>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c860>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : Prev sent/recv: None/None, now - 31/490, monotonic - 4090172.991839465, last_heartbeat_sent - 4090172.991832662, heartbeat int. - 60 for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
py-bt ์คํ ์ถ์ ๋๋ ๋ง์คํฐ ํ๋ก์ธ์ค :
<built-in method read of module object at remote 0x7fb3e61f33b8>
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 64, in __read__
chunk = read(fd, size)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 241, in _recv_message
fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
<built-in method next of module object at remote 0x7fb3e627ac28>
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 298, in on_result_readable
next(it)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
cb(*cbargs)
<built-in method next of module object at remote 0x7fb3e627ac28>
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
worker.start()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
ret = self.run(*args, **kwargs)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
return self(*args, **options)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
return self.execute(command, argv)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
_main()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
sys.exit(main())
๋ง์คํฐ ํ๋ก์ธ์ค์ bt ์คํ ์ถ์
#0 0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x0000000000542a85 in _Py_read (fd=fd@entry=56, buf=0x7fb3cecbad70, count=count@entry=4) at Python/fileutils.c:1407
#2 0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=56) at ./Modules/posixmodule.c:8043
#3 os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x7fb3ceb7f3b8, nargs=<optimized out>)
at ./Modules/clinic/posixmodule.c.h:3625
#4 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x7fb3ceb7f3b8,
self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5 _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>,
args=args@entry=0x7fb3ceb7f3b8, nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6 0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3ceb7f218, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 64, in __read__ (fd=56, buf=<_io.BytesIO at remote 0x7fb3ceca0f10>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>)) at Python/ceval.c:547
#9 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf537420>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x347a2b8, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf3074f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='__read__',
qualname='__read__') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#11 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#13 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cebfb138) at Objects/genobject.c:221
#14 gen_iternext (gen=0x7fb3cebfb138) at Objects/genobject.c:542
#15 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
nargs=<optimized out>) at Python/bltinmodule.c:1426
#16 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3481f10,
self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#17 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#18 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#20 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x3481d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 298, in on_result_readable (fileno=56, it=<generator at remote 0x7fb3cebfb138>))
at Python/ceval.c:547
#21 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2ed300>,
globals=globals@entry={'__name__': 'celery.concurrency.asynpool', '__doc__': "Version of multiprocessing.Pool using Async I/O.\n\n.. note::\n\n This module will be moved soon, so don't use it directly.\n\nThis is a non-blocking version of :class:`multiprocessing.Pool`.\n\nThis code deals with three major challenges:\n\n#. Starting up child processes and keeping them running.\n#. Sending jobs to the processes and receiving results back.\n#. Safely shutting down this system.\n", '__package__': 'celery.concurrency', '__loader__': <SourceFileLoader(name='celery.concurrency.asynpool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py') at remote 0x7fb3cf2db828>, '__spec__': <ModuleSpec(name='celery.concurrency.asynpool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py', loader_state=None, submodule_search_locations=N...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7fb3cec173a8, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
closure=(<cell at remote 0x7fb3ceb5f9a8>, <cell at remote 0x7fb3ceb5f9d8>, <cell at remote 0x7fb3ceb5fa08>, <cell at remote 0x7fb3ceb5fa38>, <cell at remote 0x7fb3ceb5fb58>), name='on_result_readable',
qualname='ResultHandler._make_process_result.<locals>.on_result_readable') at Python/ceval.c:3930
#22 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec260d0>, args=args@entry=0x7fb3cec173a8,
nargs=1, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#23 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec260d0>, args=args@entry=(56,),
kwargs=kwargs@entry=0x0) at Objects/call.c:226
#24 0x0000000000422479 in do_call_core (kwdict=0x0, callargs=(56,), func=<function at remote 0x7fb3cec260d0>) at Python/ceval.c:4645
#25 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#26 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#27 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#28 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
nargs=<optimized out>) at Python/bltinmodule.c:1426
#29 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#30 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#31 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#32 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#33 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#34 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#35 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#36 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.Celery---Type <return> to continue, or q <return> to quit---
Config', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
kwargs=kwargs@entry=0x0) at Objects/call.c:226
#37 0x0000000000422479 in do_call_core (kwdict=0x0,
callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#38 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#39 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#40 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#41 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#42 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#43 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#44 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#45 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#46 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#47 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#48 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#49 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#64 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n Number of child processes processing the queue. The default\n is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n Pool implementation:\n\n prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n Set custom hostname (e.g., 'w1@%%h'). Expands: %%h (hostname),\n %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n Also run the `celery beat` periodic task scheduler. Please note that\n there must only be one instance of this service.\n\n .. note::\n\n ``-B`` is meant to be used for development purposes. For production\n environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n L...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#65 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#66 0x000000000043e749 in _PyObject_FastCallDict (
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#67 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'b---Type <return> to continue, or q <return> to quit---
gl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
at Objects/call.c:904
#68 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#69 0x0000000000422479 in do_call_core (
kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#70 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#71 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#72 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#73 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,
args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#74 0x000000000043e749 in _PyObject_FastCallDict (
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98
#75 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904
#76 0x0000000000494665 in slot_tp_call (
self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_
worker.ta...(truncated), args=(),
kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': N---Type <return> to continue, or q <return> to quit---
one, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
at Objects/typeobject.c:6379
#77 0x000000000043c638 in PyObject_Call (
callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#78 0x0000000000422479 in do_call_core (
kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#80 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#81 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#82 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#83 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
at Python/ceval.c:4616
#84 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#85 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#86 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
qualname='CeleryCommand.execute') at Python/ceval.c:3930
#87 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#88 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#89 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#90 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#91 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#92 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#93 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#94 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#95 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'pr---Type <return> to continue, or q <return> to quit---
eload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#96 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#97 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#98 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#99 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#100 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#101 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#102 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#103 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#104 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#105 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#106 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#107 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#108 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#109 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
at Objects/call.c:283
#110 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#111 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#112 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#113 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#114 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#115 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
args=0x0,
locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#116 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#117 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
at Python/pythonrun.c:1035
#118 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#119 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
at Python/pythonrun.c:429
#120 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
fp=0x26f05c0) at Modules/main.c:427
#121 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#122 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
---Type <return> to continue, or q <return> to quit---
#123 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#124 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#125 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#126 0x0000000000429df4 in _start ()
py-bt ์คํ ์ถ์ ๋๋ ๋ถ๊ธฐ ๋ ํ๋ก์ธ์ค :
Traceback (most recent call first):
<built-in method read of module object at remote 0x7fb3e61f33b8>
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 422, in _recv
chunk = read(handle, remaining)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 456, in _recv_bytes
buf = self._recv(4)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 243, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/queues.py", line 355, in get_payload
return self._reader.recv_bytes()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 445, in _recv
return True, loads(get_payload())
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 473, in receive
ready, req = _receive(1.0)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 351, in workloop
req = wait_for_job()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 292, in __call__
sys.exit(self.workloop(pid=pid))
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 327, in _bootstrap
self.run()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 79, in _launch
code = process_obj._bootstrap()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 24, in __init__
self._launch(process_obj)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/context.py", line 333, in _Popen
return Popen(process_obj)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 124, in start
self._popen = self._Popen(self)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1158, in _create_worker_process
w.start()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 445, in _create_worker_process
return super(AsynPool, self)._create_worker_process(i)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1328, in _repopulate_pool
self._create_worker_process(self._avail_index())
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1343, in _maintain_pool
self._repopulate_pool(joined)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1351, in maintain_pool
self._maintain_pool()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 130, in _reschedules
return fun(*args, **kwargs)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
return self.fun(*self.args, **self.kwargs)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
entry()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
<built-in method next of module object at remote 0x7fb3e627ac28>
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
worker.start()
---Type <return> to continue, or q <return> to quit---
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
ret = self.run(*args, **kwargs)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
return self(*args, **options)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
return self.execute(command, argv)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
_main()
File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
sys.exit(main())
๋ถ๊ธฐ ๋ ํ๋ก์ธ์ค์ bt ์คํ ์ถ์
#0 0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x0000000000542a85 in _Py_read (fd=fd@entry=12, buf=0x7fb3cf30b410, count=count@entry=4) at Python/fileutils.c:1407
#2 0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=12) at ./Modules/posixmodule.c:8043
#3 os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0, nargs=<optimized out>)
at ./Modules/clinic/posixmodule.c.h:3625
#4 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3873df0,
self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5 _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0,
nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6 0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x3873c38, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 422, in _recv (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>, buf=<_io.BytesIO at remote 0x7fb3cecccba0>, handle=12, remaining=4)) at Python/ceval.c:547
#9 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53ff60>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec4f1e8, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf54e488, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='_recv',
qualname='Connection._recv') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#11 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#13 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3cec4f048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 456, in _recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxsize=None)) at Python/ceval.c:547
#14 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5420c0>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec87a78, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf543108, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='_recv_bytes', qualname='Connection._recv_bytes') at Python/ceval.c:3930
#15 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#16 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#17 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#18 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3cec878e0, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 243, in recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxlength=None)) at Python/ceval.c:547
#19 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53f300>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec9e9e8, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf538bf8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='recv_bytes', qualname='_ConnectionBase.recv_bytes') at Python/ceval.c:3930
#20 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#21 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#22 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#23 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#24 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#25 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#26 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#27 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3cec87728, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 445, in _recv (timeout=<float at remote 0x7fb3d01417c8>, loads=<function at remote 0x7fb3cf5d72f0>)) at Python/ceval.c:547
#28 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598780>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x3346fa0, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3ced2c878, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
closure=closure@entry=(<cell at remote 0x7fb3cec534f8>,), name='_recv', qualname='Worker._make_recv_method.<locals>._recv')
at Python/ceval.c:3930
#29 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#30 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#31 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#32 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x3346df8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 473, in receive (debug=<function at remote 0x7fb3d970b0d0>)) at Python/ceval.c:547
#33 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598a50>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=0, kwnames=0x0, kwargs=kwargs@entry=0x386d2c0, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cec222c8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
closure=closure@entry=(<cell at remote 0x7fb3ceb62708>, <cell at remote 0x7fb3ceb62fd8>), name='receive',
qualname='Worker._make_protected_receive.<locals>.receive') at Python/ceval.c:3930
#34 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#35 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#37 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x386d058, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 351, in workloop (debug=<function at remote 0x7fb3d970b0d0>, now=<built-in method monotonic of module object at remote 0x7fb3de980d68>, pid=111793, put=<method at remote 0x7fb3cecf55c8>, inqW_fd=13, synqW_fd=None, maxtasks=1, max_memory_per_child=0, prepare_result=<method at remote 0x7fb3cec86548>, wait_for_job=<function at remote 0x7fb3cec7b268>, wait_for_syn=<function at remote 0x7fb3cec3ff28>, completed=0)) at Python/ceval.c:547
#38 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5985d0>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=1, kwnames=0x7fb3cf599060, kwargs=kwargs@entry=0x3490068, kwcount=1, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf536f90, defcount=defcount@entry=3, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='workloop',
qualname='Worker.workloop') at Python/ceval.c:3930
#39 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#40 0x0000000000428c25 in call_function (kwnames=('pid',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
at Python/ceval.c:4616
#41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#42 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x348feb8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/si---Type <return> to continue, or q <return> to quit---
te-packages/billiard/pool.py, line 292, in __call__ (self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898...(truncated)) at Python/ceval.c:547
#43 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf598270>,
globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7ffdbea23e20, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0,
name='__call__', qualname='Worker.__call__') at Python/ceval.c:3930
#44 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf53db70>,
args=args@entry=0x7ffdbea23e20, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:376
#45 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea23e20,
callable=<function at remote 0x7fb3cf53db70>) at Objects/call.c:98
#46 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf53db70>,
obj=obj@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:904
#47 0x0000000000494665 in slot_tp_call (
self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated), args=(), kwds={})
at Objects/typeobject.c:6379
#48 0x000000000043c638 in PyObject_Call (
callable=callable@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:245
#49 0x0000000000422479 in do_call_core (kwdict={}, callargs=(),
func=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated)) at Python/ceval.c:4645
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf1f5b70, args=<optimized out>, args@entry=0x7ffdbea24610,
nargs=nargs@entry=2,
globals=globals@entry={'__name__': 'billiard.popen_fork', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.popen_fork', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py') at remote 0x7fb3cf201748>, '__spec__': <ModuleSpec(name='billiard.popen_fork', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/popen_fork.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf201780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated))
at Objects/call.c:283
#64 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf205c80>,
args=args@entry=0x7ffdbea24610, nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#65 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffdbea24610,
callable=<function at remote 0x7fb3cf205c80>) at Objects/call.c:98
---Type <return> to continue, or q <return> to quit---
#66 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf205c80>,
obj=obj@entry=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
kwargs=kwargs@entry=0x0) at Objects/call.c:904
#67 0x000000000048f059 in slot_tp_init (self=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
args=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated), kwds=0x0)
at Objects/typeobject.c:6613
#68 0x000000000048a332 in type_call (type=<optimized out>, type@entry=0x336fdb8,
args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
kwds=kwds@entry=0x0) at Objects/typeobject.c:949
#69 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x336fdb8>, stack=<optimized out>,
nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#70 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#71 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#72 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#73 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#74 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#75 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#76 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#77 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#78 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#80 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#81 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#82 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#83 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#84 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3cf1f0570, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 445, in _create_worker_process (self=<AsynPool(sched_strategy=4, synack=False, _queues={(<_SimpleQueue(_reader=<Connection(_handle=31, _readable=True, _writable=False) at remote 0x7fb3cecdecf8>, _writer=<Connection(_handle=32, _readable=False, _writable=True) at remote 0x7fb3cecde2b0>, _poll=<method at remote 0x7fb3cf23cec8>, _rlock=None, _wlock=None) at remote 0x7fb3ced47748>, <_SimpleQueue(_reader=<Connection(_handle=33, _readable=True, _writable=False) at remote 0x7fb3cecde2e8>, _writer=<Connection(_handle=34, _readable=False, _writable=True) at remote 0x7fb3cecde828>, _poll=<method at remote 0x7fb3ceb4a188>, _rlock=None, _wlock=None) at remote 0x7fb3cecde160>, None): <ForkProcess(_identity=(3731,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _pope...(truncated)) at Python/ceval.c:547
#85 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf2ed8a0>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3ced34988, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0,
closure=closure@entry=(<cell at remote 0x7fb3cf2e5eb8>,), name='_create_worker_process',
qualname='AsynPool._create_worker_process') at Python/ceval.c:3930
#86 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#87 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#88 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#89 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#90 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#91 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#92 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#93 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#94 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#95 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#96 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#97 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf5299c0, args=<optimized out>, args@entry=0x7ffdbea25400,
nargs=nargs@entry=1,
globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated))
at Objects/call.c:283
#98 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2d61e0>,
args=args@entry=0x7ffdbea25400, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:322
---Type <return> to continue, or q <return> to quit---
#99 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea25400,
callable=<function at remote 0x7fb3cf2d61e0>) at Objects/call.c:98
#100 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3cf2d61e0>, obj=<optimized out>, args=(), kwargs={})
at Objects/call.c:904
#101 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3cecd5048>, args=args@entry=(),
kwargs=kwargs@entry={}) at Objects/call.c:245
#102 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<method at remote 0x7fb3cecd5048>) at Python/ceval.c:4645
#103 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#104 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x347e4f8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py, line 130, in _reschedules (args=(), kwargs={}, last=None, now=<float at remote 0x7fb3ceb52108>, lsince=<float at remote 0x7fb3d01419f0>)) at Python/ceval.c:547
#105 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2f05d0>,
globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7fb3e6246060, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
closure=(<cell at remote 0x7fb3cec65be8>, <cell at remote 0x7fb3cec30d08>, <cell at remote 0x7fb3cec30d38>, <cell at remote 0x7fb3cec30d98>, <cell at remote 0x7fb3cec30e28>), name='maintain_pool', qualname='Pool.maintain_pool') at Python/ceval.c:3930
#106 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec6a620>, args=args@entry=0x7fb3e6246060,
nargs=0, kwargs=kwargs@entry={}) at Objects/call.c:376
#107 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec6a620>, args=args@entry=(),
kwargs=kwargs@entry={}) at Objects/call.c:226
#108 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<function at remote 0x7fb3cec6a620>) at Python/ceval.c:4645
#109 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#110 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf2e69c0, args=<optimized out>, args@entry=0x7ffdbea259c0,
nargs=nargs@entry=1,
globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated))
at Objects/call.c:283
#111 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2ee730>,
args=args@entry=0x7ffdbea259c0, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#112 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=1, args=0x7ffdbea259c0,
callable=<function at remote 0x7fb3cf2ee730>) at Objects/call.c:98
#113 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf2ee730>,
obj=obj@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwargs=kwargs@entry=0x0) at Objects/call.c:904
#114 0x0000000000494665 in slot_tp_call (self=self@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwds=kwds@entry=0x0)
at Objects/typeobject.c:6379
#115 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<Entry at remote 0x7fb3cebb3768>,
stack=<optimized out>, nargs=nargs@entry=0, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#116 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#117 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#118 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x3480618, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py, line 145, in fire_timers (self=<Hub(timer=<Timer(max_interval=<float at remote 0x7fb3d0141a08>, on_error=None, _queue=[<scheduled at remote 0x7fb3cebf0d18>, <scheduled at remote 0x7fb3cebf0a48>, <scheduled at remote 0x7fb3cecea3b8>, <scheduled at remote 0x7fb3ceceabd8>, <scheduled at remote 0x7fb3ceb60368>, <scheduled at remote 0x7fb3cec05d18>, <scheduled at remote 0x7fb3ceb606d8>, <scheduled at remote 0x7fb3cec8f7c8>, <scheduled at remote 0x7fb3cec8f688>]) at remote 0x7fb3cf2b04e0>, readers={71: (<method at remote 0x7fb3cec80b48>, (<Connection(_connection_id='ed843206bfeb424292688d00071ac41e', authentication=(<AMQPLAIN(username='aep', password='F8P7uYpFbxyWQSJF4psFc52P6') at remote 0x7fb3ceb98a20>,), client_properties={'product': 'py-amqp', 'product_version': '2.5.1', 'capabilities': {'consumer_cancel_notify': True, 'connection.blocked': True, ...(truncated)) at Python/ceval.c:547
#119 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf550780>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=1, kwnames=0x7fb3cf2dfd80, kwargs=kwargs@entry=0x347c960, kwcount=1, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3cf549ab0, defcount=defcount@entry=4, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='fire_timers', qualname='Hub.fire_timers') at Python/ceval.c:3930
#120 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#121 0x0000000000428c25 in call_function (kwnames=('propagate',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
at Python/ceval.c:4616
#122 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#123 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#124 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#125 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
nargs=<optimized out>) at Python/bltinmodule.c:1426
#126 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#127 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>,
args=args@entry=0x347abd0, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#128 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#129 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#130 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#131 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-em---Type <return> to continue, or q <return> to quit---
IVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#132 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#133 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
kwargs=kwargs@entry=0x0) at Objects/call.c:226
#134 0x0000000000422479 in do_call_core (kwdict=0x0,
callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#135 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#136 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#137 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#138 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#139 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#140 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#141 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#142 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#143 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#144 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#145 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#146 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#147 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#148 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#149 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#150 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#151 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#152 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
at Objects/call.c:283
#153 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#154 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#155 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#156 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#157 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#158 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#159 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#160 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#161 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n Number of child processes processing the queue. The default\n is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n Pool implementation:\n\n prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n Set custom hostname (e.g., 'w1@%%h'). Expands: %%h (hostname),\n %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n Also run the `celery beat` periodic task scheduler. Please note that\n there must only be one instance of this service.\n\n .. note::\n\n ``-B`` is meant to be used for development purposes. For production\n environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n L...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#162 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'schedu---Type <return> to continue, or q <return> to quit---
ler': None}) at Objects/call.c:376
#163 0x000000000043e749 in _PyObject_FastCallDict (
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#164 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
at Objects/call.c:904
#165 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#166 0x0000000000422479 in do_call_core (
kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#167 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#168 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#169 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#170 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,
args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#171 0x000000000043e749 in _PyObject_FastCallDict (
kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98
#172 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904
---Type <return> to continue, or q <return> to quit---
#173 0x0000000000494665 in slot_tp_call (
self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated), args=(),
kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
at Objects/typeobject.c:6379
#174 0x000000000043c638 in PyObject_Call (
callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
args=args@entry=(),
kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#175 0x0000000000422479 in do_call_core (
kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#176 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#177 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#178 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#179 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#180 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
at Python/ceval.c:4616
#181 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#182 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#183 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
qualname='CeleryCommand.execute') at Python/ceval.c:3930
#184 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#185 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#186 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#187 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_per---Type <return> to continue, or q <return> to quit---
iodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#188 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#189 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#190 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#191 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#192 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#193 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#194 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#195 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#196 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#197 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#198 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#199 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:433
#200 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#201 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#202 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
at Objects/call.c:283
#203 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#204 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#205 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#206 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
at Objects/call.c:283
#207 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
kwnames=<optimized out>) at Objects/call.c:415
#208 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#209 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#210 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#211 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#212 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
args=0x0,
locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#213 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#214 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
at Python/pythonrun.c:1035
#215 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': ---Type <return> to continue, or q <return> to quit---
None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#216 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
at Python/pythonrun.c:429
#217 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
fp=0x26f05c0) at Modules/main.c:427
#218 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#219 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
#220 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#221 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#222 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#223 0x0000000000429df4 in _start ()
์ข ์์ฑ :
celery==4.3.0
kombu==4.6.5
billard==3.6.1.0
amqp==2.5.1
python==3.7.3
celery == 4.4.0rc4๋ฅผ ์๋ํด ๋ณผ ์ ์์ต๋๊น?
์คํ ์ด์ง์ ํ์ธํ๋ ๋ฐ ์ผ์ฃผ์ผ์ด ๊ฑธ๋ฆฌ์ง ๋ง ํ๋ก๋์ ์์ ๋ถํ๋ฅผ ์ฒ๋ฆฌํ๋ ค๊ณ ๋ ธ๋ ฅํ ๊ฒ์ ๋๋ค.
@ wakemaster39 : ๋์์ด ๋์๋์?
์ด๊ฒ์ด ๊ณ ์ณ์ง ๊ฒ์ฒ๋ผ ๋ณด์ด๋์ง ๊ถ๊ธํฉ๋๋ค @ wakemaster39
์ ๋ฌ๋ฆฌ == 4.4.0rc5?
์ง๋ ๋ช ์ฃผ ๋์ ๋ค๋ฅธ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํด์ผํ๊ณ ๋ฌธ์ ๊ฐ์๋ ์ฌ์ดํธ๊ฐ ํด์ผ ๋์ ์คํ๋ผ์ธ ์ํ๊ฐ๋๋ฏ๋ก ๋ณ๊ฒฝํ๊ณ ์ถ์ง ์์ต๋๋ค. ๋๋ ์ํด๋ฅผ ๋ง์ดํ์ฌ ๋ถํ๋ฅผ ๋ฐ๊ธฐ ์ํด ๋ฐฐ์นํ๊ธฐ๋ฅผ ํฌ๋งํ๋ฉด์ ๋ค์ ์ฃผ ๋ฐ ๋์ ์คํ ์ด์ง์ ๊ฒํ ํ๊ณ ์์ต๋๋ค.
์ด ๋ฌธ์ ๊ฐ beat
์ ๊ฐ์ ์
๋ฌ๋ฆฌ์ ๋ค๋ฅธ ๋ถ๋ถ์ด ์๋๋ผ ํนํ ์
๋ฌ๋ฆฌ ์์
์์๊ฒ ๋ฌธ์ ๊ฐ๋๋์ง ์๊ณ ์์ต๋๊น? ์ธ์ฒด ๊ณตํ์ ๋ด๊ฐ ์ข์ํ๋ ๋ญ๊ฐ๋ฅผ ์ ํํ๋ ๊ฒฝ์ฐ go-celery
์ ๊ณ์ celery.beat
์ฆ ์ด๋ก ์ ์ผ๋ก ๋ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํฉ๋๊น? ์๋๋ฉด ์ฌ๋๋ค์ด ์ด๋ค ์ด์ ๋ก ๋ beat
์ค๋จ ๋ฌธ์ ๋ฅผ ๊ฒฝํ ํ ์ ์ด ์์ต๋๊น?
์ ๊ฒฝ์ฐ์๋ ์ฃฝ๋ ๋นํธ์กฐ์ฐจ๋ ์ ๋ฌ๋ฆฌ ์์ ์๊ฐ ๊ต์ฐฉ ์ํ๋ฅผ ์ ์งํฉ๋๋ค. ์๊ฐ ์ ํ์ ์ถ๊ฐํ์ฌ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ์ง๋ง ๊ฐ๋์ฉ ์ค๋ ๋๊ฐ ๋ฉ์ถ์์ต๋๋ค. ์์ ์ ์ด ์ง ์์ ๋ฐ์ดํฐ๋ฒ ์ด์ค ํธ๋์ญ์ ์ด ์์์ ๋ ํจ์ฌ ๋ ๋ง์ด ๋ฐ์ํ์ง๋ง ๊ทธ๊ฒ์ ์์ ํ๋ฉด ๋ํต์ด ํด๊ฒฐ๋์์ต๋๋ค.
tldr : ๋นํธ๊ฐ ์๋๋ผ "์์๋ก IPC์ ๋งค๋ฌ๋ฆฌ๋"์์ ์
celery = "== 4.4.0"์ ์คํํ๋ ์ ์ฌํ ๊ฒ๋ค์ ๋ณด์์ต๋๋ค. Beat๋ ์ ๋์๊ฐ์ง๋ง ์ด๋ค ์ด์ ๋ก ์ ๋ฌ๋ฆฌ๊ฐ ๋ถ์ด ์์ต๋๋ค. ๊ต์ฐฉ ์ํ ์ผ ์ ์์ง๋ง ๋งํ๊ธฐ๋ ๋งค์ฐ ์ด๋ ต์ต๋๋ค.
@ vincent31337 ๊ณผ ๋น์ทํ ๊ฒฝํ์ celery multi start
๋ช
๋ น์ ์ฌ์ฉํ์ง ์์ผ๋ฉด ์ด๊ฒ์ด ํผํ ์ ์๋ค๊ณ ์๊ฐํฉ๋๋ค.
๋๋ ๊ฐ์ ๋ฌธ์ ์ ์ง๋ฉดํ๊ณ ์์ต๋๋ค. Python2.7, Celery4.1.0. SQS ๋ธ๋ก์ปค๋ฅผ ์ฌ์ฉํฉ๋๋ค. ์ ๋ฌ๋ฆฌ๋ ์ผ์ ์๊ฐ ๋์ ์์ ์ ์ฒ๋ฆฌํ๊ณ ์ ์ ํ ๊ฐ์๊ธฐ ์ฐจ๋จ๋ฉ๋๋ค.
์ต์ ๋ฆด๋ฆฌ์ค๋ฅผ ์๋ํด์ผํฉ๋๋ค.
์ ๊ทธ๋ ์ด๋ ํ ์์ ๋์์ต๋๋ค.
celery => 4.4.x์ ๊ฒฝ์ฐ์๋ ์ฌ์ ํ ํด๋น๋๋์ง ์๊ณ ์ถ์ต๋๋ค.
์ด ๊ฐ์ ๋ฌธ์ ๋ฅผ ๋ค๋ฃจ๊ณ ์์ต๋๋ค.
ํ์ด์ฌ 3.8
์ฅ๊ณ 3.0.3
์
๋ฌ๋ฆฌ 4.4.0
redis 3.4.1 (๋ธ๋ก์ปค ๋ฐ ๊ฒฐ๊ณผ ์ฉ)
์ ๋ 5 ๋ช
์ ์์
์๋ฅผ ์ด์ํ๊ณ ์์ต๋๋ค. ์๋ฃ๋ 5 ๊ฐ์ ์์
์ ์คํํ๋ฉด ์์
์๊ฐ read([different pipe value for each]
์์ ์ ํ๋ฅผ ๋์ต๋๋ค. ๊ทธ ํ ์์
์ด ์๋ฝ๋๊ณ Inspect(app=app).active()
๋์ง๋ง worker_pid
๋ ์์
์์ pid ์ค ํ๋๊ฐ ์๋๋๋ค. ์ข
์ข
pid๋ 50๊ณผ ๊ฐ์ด ๋ฎ์ ๊ฐ์
๋๋ค. ps -p [low pid]
์ฌ์ฉํ๋ฉด ์ค์ ๋ก ์คํ์ค์ธ pid๋ฅผ ๋ณผ ์ ์์ต๋๋ค. ์ด ์ค๋ ๋์์ ์์ง ์ธ๊ธ ํ ๊ฒ์ ๋ณด์ง ๋ชปํ์ต๋๋ค.
์ ๋ฌ๋ฆฌ == 4.4.2๋ก๋ ๊ทธ๋ ์ฃ ?
์ฐ๋ฆฌ๋ celery == 4.3๊ณผ ์ต์ celery = 4.4.2๋ฅผ ์ฌ์ฉํด ๋๊ฐ์ ๋ฌธ์ ์ ์ง๋ฉดํ๊ณ ์์ต๋๋ค.
๋ฐฑ์๋ (๋ธ๋ก์ปค ๋ฐ ๊ฒฐ๊ณผ)๋ก rabbitmq๋ฅผ ์ฌ์ฉํ๊ณ ์์ต๋๋ค.
kombu == 4.6.8
์
๋ฌ๋ฆฌ == 4.4.2
amqp == 2.5.2
๋น๊ตฌ == 3.6.3.0
librabbitmq == 2.0.0
ํ์ด์ฌ 3.7.4
์ ๋ฌ๋ฆฌ๋ ์ฝ๊ธฐ์ ๋ฉ์ถฐ ์๊ณ ๋ค๋ฅธ ์์ ์ ์ฒ๋ฆฌ ํ ์ โโ์์ต๋๋ค.
$ sudo strace -p 8553
strace: Process 8553 attached
read(10, ^Cstrace: Process 8553 detached
<detached ...>
$ sudo strace -p 8557
strace: Process 8557 attached
read(14, ^Cstrace: Process 8557 detached
<detached ...>
$ sudo strace -p 8561
strace: Process 8561 attached
read(18, ^Cstrace: Process 8561 detached
<detached ...>
ํ๋ก์ธ์ค 8553
๋ํ lsof ์์
celery 8553 ubuntu 10r FIFO 0,10 0t0 43059 pipe
FWIW ์ฐ๋ฆฌ๋ billiard==3.6.3.0
(์
๋ฌ๋ฆฌ์๋ ๋
๋ฆฝ์ ์ผ๋ก)์ ์์ ์ฌ์ฉ์ผ๋ก ๋์ผํ ๋ฌธ์ ๋ฅผ ๊ฒช์ ๊ฒ ๊ฐ์ต๋๋ค.
๋ถ๋ชจ์:
(gdb) py-bt
Traceback (most recent call first):
<built-in method acquire of _thread.lock object at remote 0x7f838895e620>
File "/usr/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()
File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1957, in next
self._cond.wait(timeout)
File "xxx.py", line 1063, in main
for res in self.pool.imap_unordered(_partial, itertools.islice(a_generator, nb_processes)):
...
์ค๋จ ๋ ์์ ํ๋ก์ธ์ค :
0x00007f8393cc11c4 in __GI___libc_read (fd=21, buf=0x7f838892c578, nbytes=4) at ../sysdeps/unix/sysv/linux/read.c:27
Traceback (most recent call first):
<built-in method read of module object at remote 0x7f8392d3de58>
File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 422, in _recv
chunk = read(handle, remaining)
File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 456, in _recv_bytes
buf = self._recv(4)
File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 243, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 395, in get_payload
return self._reader.recv_bytes()
File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
return True, loads(get_payload())
...
์ด๊ฒ์ https://github.com/celery/celery/issues/4185#issuecomment -554828564 py-bt stack trace or forked process
์์์ ๋์ผํ ์คํ์
๋๋ค.
๋ค๋ฅธ ๋ชจ๋ ์์ ํ๋ก์ธ์ค๋ semlock์ ์์ต๋๋ค.
Traceback (most recent call first):
<built-in method __enter__ of _multiprocessing.SemLock object at remote 0x7f838896ff10>
File "/usr/local/lib/python3.6/dist-packages/billiard/synchronize.py", line 116, in __enter__
return self._semlock.__enter__()
File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 394, in get_payload
with self._rlock:
File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
return True, loads(get_payload())
...
OpenWISP์ ์ฌ๋ฌ ์ธ์คํด์ค์์ ์์ ์๊ฐ ๋ถ์ค ํด์ง๋ ๋ฌธ์ ๋ฅผ ๊ด์ฐฐํ์ต๋๋ค.
celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0
๋ช ๋ น:
celery -A openwisp2 worker -l info --pool=gevent --concurrency=10 -Ofair
์์
์ค์ธ ์์
์์ ์๋ํ์ง ์๋ ์์
์ ์ฌ์ด์ ์ ์ผํ ์ฐจ์ด์ ์ ์์
์ ์ฒ๋ฆฌํ์ง ์๋ ์์
์๊ฐ strace
์ถ๋ ฅ์์ด๋ฅผ ํ์ํ๋ค๋ ๊ฒ์
๋๋ค.
select(22, [21], [], [21], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
์์ ์๊ฐ ์์ ์ ์ฒ๋ฆฌํ๋ ๊ฒฝ์ฐ์๋ strace ์ถ๋ ฅ์ ๋ค์ ๋ ๊ฐ์ง ์ ํ์ ์ค์ด ํ์๋ฉ๋๋ค.
epoll_ctl(4, EPOLL_CTL_ADD, 20, {EPOLLIN, {u32=20, u64=12884901908}}) = -1 EEXIST (File exists)
recvfrom(20, 0x377a460, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
๋ฐ๋ผ์ ๋ง์ง๋ง ๋ ๊ฐ์ ๋ก๊ทธ ๋ผ์ธ์ด ๊ทผ๋ณธ์ ์ธ ๋ฌธ์ ๋ฅผ ๋ํ๋ด๊ฑฐ๋ ์ ์ด๋ ์ด๊ฒ์ด ๋ฌธ์ ์ ์ฆ์์ด ์๋๋ผ๊ณ ์๊ฐํ์ง ์์ต๋๋ค.
# strace -p 81
strace: Process 81 attached
read(24, ^Cstrace: Process 81 detached
<detached ...>
# lsof -p 81 | grep 24
celery 81 root mem REG 252,1 535124 /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (path dev=0,112)
celery 81 root 24r FIFO 0,13 0t0 311875006 pipe
md5-dbb5b56c5b0426df6f75353710061685
python version: 3.6
celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0
์๋ ํ์ธ์,
๋๋ ์ฌ๊ธฐ์ ๊ฐ์ ๋ฌธ์ ๋ฅผ ๋ค๋ฃจ๊ณ ์์ต๋๋ค.
์ ๋ฌ๋ฆฌ ๊ฒ์ฌ ๋ณด๊ณ ์ :
software -> celery:4.3.0 (rhubarb) kombu:4.6.4 py:3.6.6
billiard:3.6.1.0 py-amqp:2.5.1
platform -> system:Linux arch:64bit
kernel version:3.10.0-1062.18.1.el7.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:amqp
broker_url: 'amqp://nextgen:********@*:5672/ng_vhost'
result_backend: 'amqp'
task_queues:
(<unbound Queue default -> <unbound Exchange default(direct)> -> default>,
<unbound Queue runForecasts -> <unbound Exchange runForecasts(direct)> -> runForecasts>,
<unbound Queue runEnsembles -> <unbound Exchange runEnsembles(direct)> -> runEnsembles>)
task_default_queue: 'default'
task_default_exchange: 'default'
task_default_routing_key: '********'
broker_heartbeat: None
task_acks_late: True
worker_prefetch_multiplier: 1
CELERYD_OPTS :
--time-limit=3600 -c:worker1 3 -Q:worker1 runForecasts -c:worker2 2 -Q:worker2 runEnsembles -c:worker3 2 -Q:worker3 default
runForecasts ํ๋ง ๋ฉ์ถ ๊ฒ์ฒ๋ผ ๋ณด์ด๋ฉฐ ๋ค์์ ํ์ ํ๋ก์ธ์ค์ ๋ํ strace ์ถ๋ ฅ์ ๋๋ค.
$ sudo strace -p 17426
strace: Process 17426 attached
read(28, ^Cstrace: Process 17426 detached
<detached ...>
$ sudo ls -l /proc/17426/fd/28
lr-x------. 1 wind wind 64 Nov 10 22:45 /proc/17426/fd/28 -> pipe:[284399927]
๋ถ์ด์๋ ํ์ดํ๋ฅผ ์ฐพ์์ง๋ง ๋ ์ด์ ๋๋ฒ๊น ํ๋ ๋ฐฉ๋ฒ์ ๋ชจ๋ฆ ๋๋ค. ๋๊ตฐ๊ฐ๊ฐ ์ด๊ฒ์ ๋ํด ๋ฐํ ์ ์๋ค๋ฉด ๊ฐ์ฌํ๊ฒ ์ต๋๋ค.
# strace -p 32
strace: Process 32 attached
epoll_wait(7, [], 1023, 787) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 4050) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [{EPOLLIN, {u32=15, u64=139972984176655}}], 1023, 275) = 1
ioctl(15, FIONBIO, [1]) = 0
recvfrom(15, "\10\0\0\0\0\0\0", 7, 0, NULL, NULL) = 7
recvfrom(15, "\316", 1, 0, NULL, NULL) = 1
ioctl(15, FIONBIO, [0]) = 0
ioctl(15, FIONBIO, [1]) = 0
recvfrom(15, 0x7f4e96911aa0, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
ioctl(15, FIONBIO, [0]) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 19) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 670) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000) = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000) = 0
lrwx------. 1 root root 64 Dec 3 08:21 7 -> 'anon_inode:[eventpoll]'
์ ๋ฌ๋ฆฌ == 4.4.7
์์ ์ ์ธ์ : --autoscale ๋ฐ --max-tasks-per-child
๋๋ ์ด๊ฒ์ ์ํ ํ ๊ฒ์ด๋ค.
--autoscale 20,1
๋ฐ --max-tasks-per-child 1
์ง๋ฌด
<strong i="10">@share_task</strong>
def echo():
print(1)
# wait for processes scaling down and run next
for i in range(100):
echo.delay()
# once large call
for i in range(10000):
echo.delay()
๋ ๋ค๋ฅธ ์ค๋ฅ๊ฐ ์์ต๋๋ค.
[2020-12-03 18:03:42,137: CRITICAL/MainProcess] Unrecoverable error: AssertionError()
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
next(loop)
File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 303, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
entry()
File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
return self.fun(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 636, in verify_process_alive
assert proc.outqR_fd in hub.readers
AssertionError
https://github.com/celery/celery/issues/4185#issuecomment -694179347์์
billiard / pool.py ๋ผ์ธ 1957์ ๊ฒํ ํ ๊ฒฐ๊ณผ ๋ค์์ ๋ฐ๊ฒฌํ์ต๋๋ค.
self._cond = threading.Condition(threading.Lock())
......
self._lost_worker_timeout = lost_worker_timeout
......
def next(self, timeout=None):
with self._cond:
......
self._cond.wait(timeout)
......
self._cond.wait๊ฐ ์๊ฐ ์ด๊ณผ๋ฅผ ์ค์ ํ์ง ์์์ต๋๋ค. ์ฌ๊ธฐ์ ๊ฐํ ์์ต๋๊น?
์ด ๋ฌธ์ ์ ์ง๋ฉด ํ ์ฌ๋๋ค์ ์ํด https://github.com/celery/billiard/pull/325 ๋ฅผ ์๋ํด ์ฃผ
https://github.com/celery/billiard/pull/325 ๊ฐ ์๋ํ์ง ์์ ์ ์์ต๋๋ค.
๊ทธ๋ฆฌ๊ณ ํฌ๊ธฐํฉ๋๋ค. ์ด์ ์ ๋ --pool threads
๊ฐ์ฅ ์ ์ฉํ ๋๊ธ
์๋ค์ํผ poll ()์ ์ฌ์ฉํ๋ฉด ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋์ง ์์ต๋๋ค. ์ค์ ๋ฌธ์ ๋ ํ๋ธ์ ๋ฑ๋ก๋์ด์๋ ๋์ ํ์ผ ์ค๋ช ์๊ฐ ๋ซํ๊ณ ์๋ค๋ ๊ฒ์ ๋๋ค. ์ด๋์์ ์๋ชป๋์๋์ง ์ ํํ๊ฒ ํ์ ํ๋ ๊ฒ์ด ์ฝ๋ค๋ ๊ฒ์ ์๊ฒ๋๋ฉด ๋ค์ ํดํน์ผ๋ก์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์ ์์ต๋๋ค. ์ด๊ฒ์ด ์์ผ๋ฉด ํ๋ ์๊ฐ ์ด๋ด์ ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ต๋๋ค. ๊ทธ๊ฒ์ผ๋ก ์ฐ๋ฆฌ๋ ๊ทธ๊ฒ์ ์์ง ์ด๋ฐ์ํฌ ์ ์์์ต๋๋ค. ํดํน์ด ์ฌ๋ฐ๋ฅธ ์์ ์ ์๋์ง๋ง ์ค์ ๋ก ์ฝ๋๋ฅผ ๋ ์ ์๋ ์ฌ๋์ด ์ข์ ์์ ์ ์์ฑํด์ผํฉ๋๋ค.
์ฐ๊ธฐ ํ fd๊ฐ ํ๋ธ์์ ์ ๋๋ก ์ ๊ฑฐ ๋ ๊ฒ์ฒ๋ผ ๋ณด์ด์ง๋ง (๋ช ์ค ์๋ก) ์ฝ๊ธฐ ํ fd๋ ๊ทธ๋ ์ง ์์ต๋๋ค.