<p>يتم تعليق عمال الكرفس بشكل عشوائي على IPC (مما يتسبب في وضع جميع المهام في قائمة الانتظار ، ولكن لم تعد تعمل)</p>

تم إنشاؤها على ٧ أغسطس ٢٠١٧  ·  109تعليقات  ·  مصدر: celery/celery

فشل بنفس الطريقة في إصدارات الكرفس المتعددة:
إصدار الكرفس 3.1.7 للبلياردو إصدار 3.3.0.23
إصدار الكرفس 3.1.25 البلياردو الإصدار 3.3.0.23

celery worker -l debug --autoscale=50,4 -Ofair -Q tower_scheduler,tower_broadcast_all,tower,localhost -n celery@localhost

خطوات التكاثر

يحدث بشكل متقطع عند إجراء اختبارات التكامل الليلية لمدة 8 ساعات على Ubuntu 14.04 و 16.04. لا يحدث في RHEL 7 أو Centos 7. نحن نعمل على مجموعة من خطوات الترفيه الأصغر.

سلوك متوقع

  • ليواصل العمال تجهيز العمل.

السلوك الفعلي

تقوم كل من عملية الكرفس الرئيسية وعامل معين بكتلة على نفس واصف الملف ، مما يؤدي إلى تنفيذ read() . أنهى العامل عمله وهو جاهز لمزيد من العمل. الوالد "مغلق" ، يحظر مكالمة النظام read() التي يشاركها العامل.

إعادة تشغيل الكرفس "يصلح" المشكلة. بشكل أكثر جراحياً ، فإن إرسال SIGUSR1 إلى العملية الرئيسية "يصلح" المشكلة عن طريق كسرها من استدعاء النظام read() . ثم يعود الطفل من القراءة ويبدو أنه يعالج / الرسالة المعلقة. لا تحاول العملية الرئيسية الحصول على read() من الأنابيب مرة أخرى.

https://github.com/celery/celery/blob/3.1/celery/concurrency/asynpool.py#L220 هو المكان الذي نتوقف فيه. هذه دعوة إلى __read__ عليها من https://github.com/celery/billiard/blob/3.3/Modules/_billiard/multiprocessing.c#L202
هذا غير محظور وغير متزامن. أنا لا أفهم حقًا كيف أن هذا الرمز ليس عرضة لسيناريو طريق مسدود / حظر لانهائي. read غير محظور ويتم استدعاؤه بطريقة يمكن أن تمنع إلى الأبد إذا مات الطفل.

نحاول الآن إعادة إنشاء المشكلة مع مجموعة CELERY_RDBSIG="1" حتى نتمكن من الانتقال إلى جلسة تصحيح الأخطاء عن بُعد عند حدوث حالة الجمود.

ومن شأن أي نصيحة تكون مفيدة.

من منظور نظام التشغيل ، لا يمكنني التفكير في كيفية حدوث ذلك.

  • لم يتم ملاحظة write() بعد SIGUSR1.
  • تم حظر عمليتين على نفس PIPE عبر قراءة (). هذا يعني أنه إما لا توجد بيانات في الأنبوب أو لا توجد بيانات كافية.
  • عملية واحدة "تستسلم" (لأننا أرسلنا SIGUSR1)
  • ثم تتابع العمليات الأخرى قراءة () مع البيانات ... دون ملاحظة الكتابة ().
Billiard Bug Report Critical Needs Test Coverage ✘ Worker Hangs
مصدر
picture chrismeyersfsu
👍22

التعليق الأكثر فائدة

حسنًا ، كما اتضح ، فإن استخدام الاستطلاع () لا يحل المشكلة. نظرًا لأن المشكلة الفعلية هي إغلاق واصف الملف بينما لا يزال مسجلاً في لوحة الوصل. بمجرد أن تدرك أنه من السهل معرفة أين يحدث الخطأ بالضبط وأن الاختراق التالي يعمل على إصلاح هذه المشكلة بالنسبة لنا. بدون هذا يمكننا إعادة إظهار المشكلة في غضون ساعة أو ساعتين. مع ذلك لم نتمكن من تشغيله بعد. الاختراق ليس هو الإصلاح الصحيح ، لكنه يتطلب حقًا شخصًا يعرف الكود بشكل أفضل لكتابة إصلاح جيد.

--- /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py~  2018-03-16 01:10:04.000000000 +0000
+++ /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py   2018-03-16 12:08:58.058686003 +0000
@@ -735,6 +735,7 @@
             except KeyError:
                 pass
         self.on_inqueue_close = on_inqueue_close
+        self.hub_remove = hub_remove

         def schedule_writes(ready_fds, total_write_count=[0]):
             # Schedule write operation to ready file descriptor.
@@ -1241,6 +1242,7 @@
             if queue:
                 for sock in (queue._reader, queue._writer):
                     if not sock.closed:
+                        self.hub_remove(sock)
                         try:
                             sock.close()
                         except (IOError, OSError):

يبدو أن قائمة انتظار الكتابة fd تمت إزالتها بشكل صحيح من المحور (بضعة أسطر لأعلى) ، لكن قائمة انتظار القراءة fd ليست كذلك.

picture kleptog في ١٦ مارس ٢٠١٨
👍12 ❤6 🎉2 👀1 😕1 😄1

ال 109 كومينتر

لدي نفس المشكلة هنا ، يبدو العمال وكأنهم يعملون لكنهم لا يعالجون الرسائل.

screen shot 2017-08-08 at 6 55 12 pm

codeadict picture codeadict في ٩ أغسطس ٢٠١٧

أنا أعمل على Python 3.6 وأستخدم Celery 4.1.0

codeadict picture codeadict في ٩ أغسطس ٢٠١٧
👍1

codeadict هل أنت قادر على متابعة العمليتين (العامل والوالد) لمعرفة ما إذا كان كلاهما عالقًا في نفس واصف الملف؟

يمكنك التحقق من أنه نفس fd باستخدام sudo lsof | grep 53r حيث 53r هو رقم واصف الملف الذي تم حظر العمليات عليه.

chrismeyersfsu picture chrismeyersfsu في ٩ أغسطس ٢٠١٧

codeadict كما ذكر chrismeyersfsu ، يمكنك strace عملية الكرفس الرئيسية ومعرفة ما إذا كانت عالقة في القراءة على أنبوب:

$ sudo strace -p <pid-of-celery-master-process> -s 100
Process X attached
read(53
ryanpetrello picture ryanpetrello في ١١ أغسطس ٢٠١٧

إذا كان لديك رموز python-dbg مثبتة مع رموز Python ، فيمكنك أيضًا استخدام gdb لمعرفة مكان توقف المعلم:

$ gdb python -p <pid-of-master-process>
...
(gdb) py-list
 216            # header
 217
 218            while Hr < 4:
 219                try:
 220                    n = __read__(
>221                        fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,

https://github.com/celery/celery/blob/v3.1.17/celery/concurrency/asynpool.py#L219

ryanpetrello picture ryanpetrello في ١١ أغسطس ٢٠١٧

نفس المشكلة.

الكرفس = 3.1.23 ، البلياردو == 3.3.0.23
بدأ العمال بالخيارات: -time-limit=7200 --autoscale=64,2 --maxtasksperchild=1
بيثون 2.7.3

بعد ساعات قليلة من تجميد الكرفس
أرى أن السيد والطفل اللانهائي ينتظران القراءة () على الأنابيب (راجع للشغل ، أنابيب مختلفة).

إذن ، هذه هي العملية الرئيسية:

root<strong i="13">@vm1</strong>:~# strace -p 41840
Process 41840 attached - interrupt to quit
read(50, ^C <unfinished ...>
Process 41840 detached

root<strong i="14">@vm1</strong>:~# sudo lsof | grep 50r
python     41840                 root   50r     FIFO                0,8        0t0     548796 pipe
python     46237                 root   50r      CHR                1,3        0t0       7604 /dev/null
python     46304                 root   50r     FIFO                0,8        0t0     548796 pipe


root<strong i="15">@vm1</strong>:~# ls -lh /proc/41840/fd/50
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]

root<strong i="16">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[548796]') 2>/dev/null
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/51 -> pipe:[548796]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/51 -> pipe:[548796]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/51 -> pipe:[548796]

وأحد الأبناء:

root<strong i="5">@vm1</strong>:~# strace -p 46304
Process 46304 attached - interrupt to quit
read(16, ^C <unfinished ...>
Process 46304 detached

root<strong i="6">@vm1</strong>:~# ls -lh /proc/46304/fd/16
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]


root<strong i="7">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[482281]') 2>/dev/null
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/16 -> pipe:[482281]
root<strong i="8">@vm1</strong>:~#

وهذا الإطار "المجمد" الحالي في العملية الرئيسية:

Thread 0x7efc1c101700
('Frame: ', 63347776)
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
    blueprint.start(self)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 838, in start
    c.loop(*c.loop_args())
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
    next(loop)
  File "/opt/waf/python/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
    cb(*cbargs)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 279, in on_result_readable
    next(it)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 221, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
()
harlov picture harlov في ٢٠ أكتوبر ٢٠١٧

harlovcodeadictchrismeyersfsu هل وجدت أي حل أو الحل؟

korycins picture korycins في ٢١ نوفمبر ٢٠١٧

لا ، نحن نقوم بالترقية إلى الكرفس 4. سنجري نفس مجموعة الاختبارات التي تستدعي هذا الخطأ ومعرفة ما إذا كان بإمكاننا إعادة الإنشاء على الكرفس 4.

chrismeyersfsu picture chrismeyersfsu في ٢١ نوفمبر ٢٠١٧

korycins أزلنا - autoscale و- maxtasksperchild - جميع بخير حتى الان.

أعتقد أن عمال إعادة النشر المستمر باستخدام مقياس تلقائي о maxtasksperchild قد يتسبب في هذا الموقف.

harlov picture harlov في ٢١ نوفمبر ٢٠١٧

chrismeyersfsu لدي نفس المشكلة مع الكرفس 4.1 لذا أعتقد أن التحول إلى الإصدار 4.1 لن يساعدك.
في السابق كنت أستخدم خيار مقياس تلقائي فقط. يتشبث العمال من وقت لآخر (مرة واحدة في الأسبوع تقريبًا) ولكن عندما أضفت maxtasksperchild = 20 ، يتسكع العمال مرة واحدة في الساعة. اعتقدت أن المشكلة يمكن أن تكون مع maxtasksperchild لذلك قمت بإزالتها.
لقد أنشأت مقياسًا تلقائيًا مخصصًا لزيادة / خفض مستوى ديناميكيًا استنادًا إلى الموارد المجانية وتحدث نفس المشكلة ، لذا أعتقد أن المشكلة مرتبطة بإعادة تفريخ العمال (كما قال harlov )

korycins picture korycins في ٢٢ نوفمبر ٢٠١٧

يمكن لأي شخص التحقق من القضايا على رأس أحدث ماجستير؟ لست متأكدًا مما إذا كان قد تم إصلاحه بالفعل بأحدث الالتزام.

auvipy picture auvipy في ٢١ ديسمبر ٢٠١٧

auvipy ما هي الالتزامات التي تشك في أنها قد تحل هذه المشكلة؟ يمكنني حاليًا إعادة إنتاج هذه المشكلة على الكرفس 4.1.0 ، البلياردو 3.5.0.3 من PyPI.

ryanpetrello picture ryanpetrello في ٣ يناير ٢٠١٨

لست متأكدًا من ذلك ، هل يمكنك تجربة الفرع الرئيسي؟ إذا كان لا يزال موجودًا بعد الإصدار 4.2 ، فسيتم تنفيذ العمل لإصلاحه

auvipy picture auvipy في ١٥ يناير ٢٠١٨

auvipy يمكننا أن نعطيه _try_ ، ولكن قد يكون من الصعب الجزم - _ شخصيًا_ ، يبدو أننا نواجه هذا التعليق مرة واحدة فقط كل بضعة أسابيع _ ، لذلك من الصعب جدًا تضييق نطاقه.

ryanpetrello picture ryanpetrello في ١٥ يناير ٢٠١٨

ryanpetrello حاول إضافة max_task_per_child وضبطها على 1. http://docs.celeryproject.org/en/latest/userguide/workers.html#max -tasks-per-child-setting
يجب أن يزيد هذا من إمكانية حدوث المشكلة

korycins picture korycins في ١٥ يناير ٢٠١٨
👍3 👀1

نفس المشكلة ، لا يمكن إعادة إنتاجها الآن ، اعتقدت أنها مشكلة في التكوين ، عند إعادة التشغيل فقط قم بإلغاء تعليق العامل ، ولكن يبدو أنه يحدث من وقت لآخر.

trianglesis picture trianglesis في ٤ مارس ٢٠١٨

مجرد رنين للقول إننا واجهنا هذه المشكلة أيضًا. كرفس 4.1.0 ، كومبو 4.1.0 ، بلياردو 3.5.0.3.
سطر الأوامر: /usr/bin/python /usr/local/bin/celery worker --app myapp.celery -B --loglevel INFO --autoscale=5,1 --max-tasks-per-child=50 --max-memory-per-child=1000000 -O fair

فيما يتعلق بالحظر على واصفات الملفات ، نرى نفس harlov أعلاه: السيد في مأزق مع أحد عماله.

إرسال SIGUSR1 إلى سيد الكرفس "يفتح" العملية ويخرج تتبع المكدس أدناه. الأمر المريب هو أنه في دالة "on_result_readable" ، تتعطل أثناء القراءة. يبدو هذا وكأنه حالة سباق كلاسيكية: يعرض الاستطلاع () أنه قابل للقراءة ولكن كتل القراءة (). يمكن أن يحدث هذا ، ولهذا السبب يوصى دائمًا بتعيين FDs في وضع عدم الحظر للتأكد من عدم حدوث أي خطأ.

الرسالة ذات الصلة على LKML: https://lkml.org/lkml/2011/6/18/105

لست متأكدًا من أنه من السهل تعيين الواصفات دون حظر ، لأن لدينا بيئة بها العديد من أساتذة الكرفس وهناك دائمًا عدد قليل عالق ، إذا كان تغييرًا بسيطًا ، فقد نتمكن من اختباره بسرعة.

MainThread                                                                                                                                                                                                                            [636/660]
=================================================                                                                                                                                                    
  File "/usr/local/bin/celery", line 11, in <module>                                                                                                                                                 
    sys.exit(main())                                                                                                                                                                                 
  File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 14, in main                                                                                                             
    _main()                                                                                                                                                                                          
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 326, in main                                                                                                              
    cmd.execute_from_commandline(argv)                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline                                                                                                            
    super(CeleryCommand, self).execute_from_commandline(argv)))                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 281, in execute_from_commandline                                                                                        
    return self.handle_argv(self.prog_name, argv[1:])                                                                                                              
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 480, in handle_argv                                                                                                       
    return self.execute(command, argv)                                                                                                                                                           
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 412, in execute                                                                                                           
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])                                                                                                                                   
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 221, in run_from_argv                                                                                                                       
    return self(*args, **options)                                                                                                                                                                    
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 244, in __call__                                                                                                            
    ret = self.run(*args, **kwargs)                                                                                                                                                                  
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 256, in run                                                                                                          
    worker.start()                                                                                                                                                                                   
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 203, in start                                                                                                     
    self.blueprint.start(self)                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start                                                                                                                                
    step.start(parent)                                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 370, in start                                                                                                              
    return self.obj.start()                                                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 320, in start                                                                                               
    blueprint.start(self)                                                                                                                                                                            
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start                                                                                                             
    step.start(parent)                                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 596, in start                                                                                               
    c.loop(*c.loop_args())                                                                                                                                                                           
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/loops.py", line 88, in asynloop                                                                                                         
    next(loop)                                                                                                                                                                                       
  File "/usr/local/lib/python2.7/dist-packages/kombu/async/hub.py", line 354, in create_loop                                                                        
    cb(*cbargs)                                                                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 292, in on_result_readable                                                                                      
    next(it)
  File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 235, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
  File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 349, in cry_handler
    safe_say(cry())
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/debug.py", line 193, in cry
    traceback.print_stack(frame, file=out)
=================================================
LOCAL VARIABLES
=================================================
{'P': <functools.partial object at 0x7f29da0f1c58>,
 'frame': <frame object at 0x37137b0>,
 'out': <vine.five.WhateverIO object at 0x7f29da14e150>,
 'sep': u'=================================================',
 'sepchr': u'=',
 'seplen': 49,
 'thread': <_MainThread(MainThread, started 139818014390016)>,
 'threading': <module 'threading' from '/usr/lib/python2.7/threading.pyc'>,
 'tid': 139818014390016,
 'tmap': {139818014390016: <_MainThread(MainThread, started 139818014390016)>}}
kleptog picture kleptog في ٧ مارس ٢٠١٨

مجرد تحديث. لقد أمضينا بعض الوقت في اختبار الأشياء ومحاولة إعادة إنتاجها.

  • تعيين max_tasks_per_child على 1 يجعل الأمر أسوأ
  • إزالة -O عادل لا فرق
  • لا تساعد إزالة المقياس التلقائي أيضًا
  • لقد قمنا حتى الآن بإعادة إنتاجه فقط على الأجهزة التي تحتوي على kernel 4.9 (Stretch) ، وليس على الأجهزة ذات 3.16 (Jessie)

لقد تحققت من أن واصفات الملفات المستخدمة عادةً بواسطة الكرفس تم تمييزها جميعًا بشكل صحيح بأنها غير محظورة وهي كذلك. لذا فإن السؤال هو ، كيف يتم حظر الكود إذا كان كل واصف غير محظور.

المسار الذي أتبعه الآن هو أن الكائن epoll () أصبح مرتبكًا بشأن الواصفات التي من المفترض أن يشاهدها. إذا قمت بتدبيس الكرفس ، فستحصل على الكثير من المخرجات مثل:

14659 14:32:56.665937 epoll_ctl(4, EPOLL_CTL_ADD, 43, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=43, u64=14879199376994992171}}) = -1 EEXIST (File exists)
14659 14:32:56.667084 epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=16, u64=14879199376994992144}}) = -1 EEXIST (File exists)
14659 14:32:56.667278 epoll_ctl(4, EPOLL_CTL_ADD, 26, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=26, u64=14879199376994992154}}) = -1 EEXIST (File exists)
14659 14:32:56.667440 epoll_ctl(4, EPOLL_CTL_ADD, 23, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=23, u64=14879199376994992151}}) = -1 EEXIST (File exists)
14659 14:32:56.667607 epoll_ctl(4, EPOLL_CTL_DEL, 37, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)
14659 14:32:56.667804 epoll_ctl(4, EPOLL_CTL_DEL, 38, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)

من المعقول تمامًا أنه في مرحلة ما يتم استخدام واصف الملف الذي تم استخدامه مرة واحدة لشيء واحد لشيء آخر ، يتم الخلط بين المحور واستدعاء الطريقة الخاطئة والازدهار!

kleptog picture kleptog في ١٣ مارس ٢٠١٨
👍2

حسنًا ، لقد وجدت المشكلة ، ولدي إخراج strace يلتقطها فشل. وهو دقيق للغاية ويتعلق بـ "الميزة" التالية غير الواضحة تمامًا لواجهة epoll (). من manpage:

س 6
هل سيؤدي إغلاق واصف ملف إلى إزالته من جميع مجموعات epoll تلقائيًا؟
أ 6
نعم ، لكن كن على علم بالنقطة التالية. واصف الملف هو مرجع لوصف الملف المفتوح (انظر فتح (2)). عندما يتم تكرار واصف الملف عبر dup (2) أو dup2 (2) أو fcntl (2) F_DUPFD أو fork (2) ، يتم إنشاء واصف ملف جديد يشير إلى نفس وصف الملف المفتوح. يظل وصف الملف المفتوح موجودًا حتى يتم إغلاق جميع واصفات الملف التي تشير إليه. تتم إزالة واصف الملف من مجموعة epoll فقط بعد إغلاق جميع واصفات الملف التي تشير إلى وصف الملف المفتوح الأساسي (أو قبل ذلك إذا تمت إزالة وصف الملف صراحة باستخدام epoll_ctl (2) EPOLL_CTL_DEL). هذا يعني أنه حتى بعد إغلاق واصف الملف الذي يعد جزءًا من مجموعة epoll ، فقد يتم الإبلاغ عن الأحداث الخاصة بموصف الملف هذا إذا ظلت واصفات الملفات الأخرى التي تشير إلى نفس وصف الملف الأساسي مفتوحة.

ما يحدث هو أنه يتم استخدام واصف ملف بواسطة الكرفس للتواصل ، ويتسرب واصف الملف إلى العامل ، ويغلق السيد واصف الملف ثم يفتح آخر للحارس _ الذي لم يتم وضع علامة عدم حظره_. تموت العملية الفرعية مما يؤدي إلى وضع علامة على واصف الملف على أنه مقروء ، ويحاول المعلم قراءته على شكل كتل.

نعم ، تُرجع الدالة epoll () قابلية القراءة لوصف ملف لم يعد مفتوحًا في هذه العملية.

هنا أقوم بلصق إخراج دعامة تم التقاطه مقابل fd 49 فقط كدليل (نعتذر عن الإخراج متعدد الخطوط ، كان strace يراقب عمليات متعددة):

7417  15:04:00.001511 pipe( <unfinished ...>
7417  15:04:00.001646 <... pipe resumed> [49, 50]) = 0
7417  15:04:00.001727 fcntl(49, F_GETFL <unfinished ...>
7417  15:04:00.002091 <... fcntl resumed> ) = 0 (flags O_RDONLY)
7417  15:04:00.002165 fcntl(49, F_SETFL, O_RDONLY|O_NONBLOCK <unfinished ...>
7417  15:04:00.002222 <... fcntl resumed> ) = 0
7417  15:04:00.003052 fcntl(49, F_GETFL <unfinished ...>
7417  15:04:00.003195 <... fcntl resumed> ) = 0x800 (flags O_RDONLY|O_NONBLOCK)
...
7417  15:04:00.237131 fcntl(49, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK)
7417  15:04:00.237363 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417  15:04:00.237478 <... epoll_ctl resumed> ) = 0
...
7417  15:04:00.274482 read(49,  <unfinished ...>
7417  15:04:00.274558 <... read resumed> "\0\0\0\25", 4) = 4
7417  15:04:00.274670 read(49,  <unfinished ...>
7417  15:04:00.274734 <... read resumed> "(I15\n(I9395\ntp0\ntp1\n.", 21) = 21
7417  15:04:00.274851 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417  15:04:00.274970 <... epoll_ctl resumed> ) = -1 EEXIST (File exists)
...
... from here fd 49 is used normally for a while,  have not included all the output ...
...
7417  15:04:52.871646 epoll_wait(4, [{EPOLLIN, {u32=49, u64=12923050543437316145}}, {EPOLLIN, {u32=33, u64=12923050543437316129}}, {EPOLLOUT, {u32=48, u64=12923050543437316144}}, {EPOLLOUT, {u32=8, u64=12923050543437316104}}], 1023, 155) = 4
7417  15:04:52.871888 read(49, "\0\0\0\31", 4) = 4
7417  15:04:52.872079 read(49, "(I4\n(I9400\nI155\ntp0\ntp1\n.", 25) = 25
... here it uses normal poll for a once?
7417  15:04:52.891818 poll([{fd=49, events=POLLERR}], 1, 10 <unfinished ...>
7417  15:04:52.902239 <... poll resumed> ) = 0 (Timeout)
7417  15:04:52.913780 close(49 <unfinished ...>
7417  15:04:52.914020 <... close resumed> ) = 0
... here we create a new pipe for a child process
7417  15:04:53.148195 pipe([49, 50])    = 0
7417  15:04:53.148996 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f2dee7f79d0) = 9482
... the write end got closed, now we dup it. This is the sentinal fd so is not marked non-blocking
7417  15:04:53.155409 dup(49)           = 50
7417  15:04:53.156030 epoll_ctl(4, EPOLL_CTL_ADD, 50, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=50, u64=12923050543437316146}}) = 0
... and here it breaks
7417  15:04:53.776974 epoll_wait(4,  <unfinished ...>
7417  15:04:53.785786 <... epoll_wait resumed> [{EPOLLHUP, {u32=49, u64=12923050543437316145}}], 1023, 1878) = 1
7417  15:04:53.785876 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9481, si_uid=905, si_status=155, si_utime=4, si_stime=3} ---
7417  15:04:53.786073 read(49,  <unfinished ...>
... discontinuity ...
7417  15:09:24.338109 <... read resumed> 0x23f4f10, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
7417  15:09:24.345507 epoll_ctl(4, EPOLL_CTL_DEL, 49, 0x7ffeecbf7660) = -1 ENOENT (No such file or directory)

لاحظ كيف يعترف epoll () بأن fd 49 لم يعد في مجموعة fdset الخاصة به ، لكنه أعاد الرقم مع ذلك.

طريقة إصلاح ذلك هي التأكد من إغلاق كل واصف ملف في العامل تتم مراقبته بواسطة epoll (). يقوم العمال حاليًا بإغلاق واصفات الملفات التي لا يريدون أن تتوافق مع أنفسهم ، لكنهم لا يلمسوا واصفات الملفات التي يرثونها والتي تتعلق بالعاملين الآخرين. الحلول هي تحديد fd الظاهر أيضًا على أنه غير محجوب ، أو ببساطة باستخدام الاستطلاع () بدلاً من epoll (). أنا أختبر الأخير محليًا.

kleptog picture kleptog في ١٤ مارس ٢٠١٨
👍7 👀1

حسنًا ، كما اتضح ، فإن استخدام الاستطلاع () لا يحل المشكلة. نظرًا لأن المشكلة الفعلية هي إغلاق واصف الملف بينما لا يزال مسجلاً في لوحة الوصل. بمجرد أن تدرك أنه من السهل معرفة أين يحدث الخطأ بالضبط وأن الاختراق التالي يعمل على إصلاح هذه المشكلة بالنسبة لنا. بدون هذا يمكننا إعادة إظهار المشكلة في غضون ساعة أو ساعتين. مع ذلك لم نتمكن من تشغيله بعد. الاختراق ليس هو الإصلاح الصحيح ، لكنه يتطلب حقًا شخصًا يعرف الكود بشكل أفضل لكتابة إصلاح جيد.

--- /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py~  2018-03-16 01:10:04.000000000 +0000
+++ /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py   2018-03-16 12:08:58.058686003 +0000
@@ -735,6 +735,7 @@
             except KeyError:
                 pass
         self.on_inqueue_close = on_inqueue_close
+        self.hub_remove = hub_remove

         def schedule_writes(ready_fds, total_write_count=[0]):
             # Schedule write operation to ready file descriptor.
@@ -1241,6 +1242,7 @@
             if queue:
                 for sock in (queue._reader, queue._writer):
                     if not sock.closed:
+                        self.hub_remove(sock)
                         try:
                             sock.close()
                         except (IOError, OSError):

يبدو أن قائمة انتظار الكتابة fd تمت إزالتها بشكل صحيح من المحور (بضعة أسطر لأعلى) ، لكن قائمة انتظار القراءة fd ليست كذلك.

kleptog picture kleptog في ١٦ مارس ٢٠١٨
👍12 ❤6 🎉2 👀1 😕1 😄1

لا توجد تحديثات هنا؟ :(

Olerdrive picture Olerdrive في ٢٦ مارس ٢٠١٨

auvipy أية أفكار حولkleptog الصورة التحليل أعلاه في https://github.com/celery/celery/issues/4185#issuecomment -373755206؟

ryanpetrello picture ryanpetrello في ٢٦ مارس ٢٠١٨

آسف ، ليس لديك الوقت الكافي الآن للتحقيق في المشكلة

auvipy picture auvipy في ٢٦ مارس ٢٠١٨

kleptog شكرا على الكثير من العمل في هذا الشأن.

يبدو أن on_inqueue_close _itself_ مكالمات hub_remove(fd) - تحديدًا على L744 ، على غرار المثال الخاص بك:

https://github.com/celery/celery/blob/b46bea25539cc26a76f0a491b95f4899f5b32c34/celery/concurrency/asynpool.py#L735 -L745

... والتي يتم استدعاؤها في destroy_queues() على L1247 ، ولكن فقط مقابل queues[0] .

https://github.com/celery/celery/blob/b46bea25539cc26a76f0a491b95f4899f5b32c34/celery/concurrency/asynpool.py#L1245 -L1249

فهل هذا البيان دقيق بالفعل؟

الاختراق ليس هو الإصلاح الصحيح ، لكنه يتطلب حقًا شخصًا يعرف الكود بشكل أفضل لكتابة إصلاح جيد.

لست على دراية كافية بأجزاء الكرفس الداخلية لأعرف سبب قيام الكود حاليًا _ فقط_ بعمل هذا مقابل queues[0] . قد يكون من المفيد طلب سحب مع فرقك ، على الأقل.

ryanpetrello picture ryanpetrello في ٢٦ مارس ٢٠١٨

IIRC لقد كانت بالفعل قائمة الانتظار الأخرى التي تم استبعادها ، لذلك ربما يتعلق الأمر بإجراء inqueue_close في قائمة الانتظار الأخرى؟ لن تتاح لي الفرصة لاختباره لفترة من الوقت.

kleptog picture kleptog في ٣٠ مارس ٢٠١٨

auvipy ، أنا آسف

هل تعلم ما إذا كنت أنت أو أي من المشرفين الآخرين على الكرفس قد أتيحت لهم الفرصة لإلقاء نظرة خاطفة على هذا الكرفس؟ لدينا عدد ملحوظ من مستخدمي الكرفس في هذا الموضوع الذين واجهوا هذه المشكلة المنهكة منذ أن تم الإبلاغ عنها في الصيف الماضي ، وحتى عدد قليل ممن شاركوا وربما حددوا مجال الكود الذي هو الجاني. هل هناك أي نوع من الوقت المقدر للوصول عند النظر إلى هذا الخطأ؟

ryanpetrello picture ryanpetrello في ١٩ أبريل ٢٠١٨
👍3

ryanpetrellokleptog لقد ألقيت نظرة سريعة على هذا الرمز ، وإليك بعض الأفكار - هذا كل شيء جديد لذا لا شيء نهائي أنا آسف.

queues في هذا السياق عبارة عن مجموعة inq, outq, synq كما تم إرجاعها بواسطة AsynPool.create_process_queues() . لذا يشير queue[0] خاص إلى inq ، ومن المنطقي أن يتم استدعاء دالة تسمى on_inqueue_close الغرض فقط.

لذا في الوقت الحالي ، الشيء الوحيد الذي يتم تمريره إلى hub_remove() هو inq._writer ، وليس _reader ، وليس outq أو synq الكتاب أو القراء fileno_to_inq بقيمة proc .

ليس لدي الكثير من التبصر في هذا. ولكن هناك أيضًا fileno_to_outq و fileno_to_synq Dts. هل يجب أن نتحقق من محتوياتها قبل استدعاء hub_remove() على تلك المقابض الأخرى؟ أم يجب علينا أن نكون دون قيد أو شرط hub_remove ing كل شيء؟ ربما يجب علينا إزالة المكالمة hub_remove() من on_inqueue_close() واستدعائها بـ destroy_queues() كما في الفرق أعلاه؟

AlexHill picture AlexHill في ٢٠ أبريل ٢٠١٨

تضمين التغريدة
شكرا لك على العمل الرائع. لقد عانينا من هذه المشكلة لفترة من الوقت ، وكان لها تأثير مؤسف على نظامنا حيث تم إسقاط المهام التي تم إحضارها مسبقًا. لقد وجدت نفس السبب الجذري - العمال والعبيد معلقون على FDs ، لكن لم أتمكن من العثور على حالة الجمود. نجح تطبيق الإصلاح الخاص بك وأصبح اختبار الإجهاد الآن يعالج 50 ألف مهمة بسهولة دون توقف.

msta picture msta في ٢٠ أبريل ٢٠١٨
👍2 👀1

AlexHill قال:

هل يجب أن نتحقق من محتوياتها قبل استدعاء hub_remove () على تلك المقابض الأخرى؟ أم يجب علينا إزالة كل شيء دون قيد أو شرط؟ ربما يجب أن نزيل استدعاء hub_remove () من on_inqueue_close () ونسميه فقط في destruction_queues () كما في الفرق أعلاه؟

بكل بساطة ، يجب عليك إزالة واصف ملف من الموزع قبل إغلاقه ، لأن الإغلاق لن يزيله من المحور نيابة عنك. لذا فإن اقتراحك الأخير هو في الواقع جيد جدًا ، لأنه "صحيح بشكل واضح" وأنا من أشد المعجبين به.

kleptog picture kleptog في ٤ مايو ٢٠١٨

باستخدام الدعامة ، أرى ما يلي:

epoll_ctl (5، EPOLL_CTL_DEL، 16، 0x7ffe54f25410) = -1 ENOENT (لا يوجد مثل هذا الملف أو الدليل)
epoll_ctl (5، EPOLL_CTL_DEL، 21، 0x7ffe54f25410) = -1 ENOENT (لا يوجد مثل هذا الملف أو الدليل)
fcntl (26، F_GETFL) = 0x2 (أعلام O_RDWR)
fcntl (26، F_SETFL، O_RDWR | O_NONBLOCK) = 0
recvfrom (26، 0x7f42ca4409e4، 7، 0، NULL، NULL) = -1 EAGAIN (المورد غير متاح مؤقتًا)

هل له علاقة بهذه القضية؟

شكرا

davidmir picture davidmir في ٢١ يونيو ٢٠١٨

ربما يساعد في تشخيص المشكلة بالرابط التالي: http://memos.ml/

wilderchen picture wilderchen في ٢١ يوليو ٢٠١٨

أواجه نفس المشكلة منذ الترقية إلى Ubuntu 16

simonweil picture simonweil في ٢٦ يوليو ٢٠١٨

أنا أواجه نفس المشكلة في 4.2.1.

nishith107 picture nishith107 في ٧ أغسطس ٢٠١٨

حاول تكوين الكرفس باستخدام خيار مهلة المقبس على النحو التالي:

celery.config_from_object ({'broker_transport_options': {'socket_timeout': 30}})

wilderchen picture wilderchen في ٧ أغسطس ٢٠١٨

كان لي نفس القضية. ومع ذلك ، بعد نشر التصحيح العلوي يتم حل المشكلة.

monakhv picture monakhv في ٧ أغسطس ٢٠١٨

لدي أيضًا هذه المشكلة ، هل هناك طلب سحب مع إصلاح kleptog أم أن الأشخاص يستخدمون

DomHudson picture DomHudson في ٧ أغسطس ٢٠١٨

تم حل مشكلتنا بعد إضافة tcp socket keepalive and timeout (باستخدام redis)

BROKER_TRANSPORT_OPTIONS = {'socket_timeout': 60,
                            'socket_connect_timeout': 120,
                            'socket_keepalive': True,
                            'socket_keepalive_options': {socket.TCP_KEEPIDLE: 60,
                                 socket.TCP_KEEPINTVL: 60,
                                 socket.TCP_KEEPCNT: 3}}
linar-jether picture linar-jether في ٧ أغسطس ٢٠١٨
👍1

@ linar-jether: ما هو هذا المقبس في socket.TCP_KEEPIDLE ؟

nishith107 picture nishith107 في ٧ أغسطس ٢٠١٨

@ nishith107 import socket

linar-jether picture linar-jether في ٧ أغسطس ٢٠١٨

يكاد يكون من المؤكد أن تغيير تهيئة وسيط redis غير مرتبط بالمشكلة في هذا الموضوع ، والتي تتعلق بالتعثر في قراءة حظر في أنبوب الاتصال الرئيسي-الطفل.

raylu picture raylu في ٧ أغسطس ٢٠١٨
👍1

حسنًا ، من الممكن أن تؤدي إعدادات الترقيع مع الوسيط إلى عدم استقرار كافٍ لتفادي المشكلة الأساسية ، ولكن تعليق IPC الرئيسي-الفرعي هو خطأ واضح.

ryanpetrello picture ryanpetrello في ٧ أغسطس ٢٠١٨

auvipy تم الإبلاغ عن هذا في الأصل منذ عام مضى اليوم. لقد تركت أنا والآخرون معلومات تفصيلية وتحليلاً للمكان الذي يبدو أن الكود معلق فيه ، بما في ذلك إخراج strace حيث قمنا بتضييقه إلى بضعة أسطر من التعليمات البرمجية. حتى أن شخصًا ما نشر رابطًا إلى ملف تصحيح (
https://github.com/celery/celery/issues/4185#issuecomment-373755206) يبدو أن الأشخاص يستخدمون وينشرون (في الإنتاج؟) كحل بديل. يبدو أن العديد من الأشخاص المشاركين في سلسلة التعليقات هذه يواجهون هذا بانتظام عبر عدد كبير من إصدارات الكرفس وإعدادات الوسيط.

أقدر أن هذا مشروع مفتوح المصدر ، وكل من يشارك فيه متطوع ، لذلك لا أقصد التذمر إلى ما لا نهاية هل تعرف ما إذا كان _ أي شخص _ قد بذل جهدًا في التحقيق أو تشخيص هذه المشكلة الخطيرة؟

ryanpetrello picture ryanpetrello في ٧ أغسطس ٢٠١٨
👍3

سيكون من الأسهل بالنسبة لنا إصلاح المشكلات إذا واجهها شخص منا في الإنتاج. نظرًا لأن بعضكم استخدم بعض الحلول للتخلص من هذه المشكلة ، فلماذا لا ترسل العلاقات العامة مع حالات الاختبار المناسبة؟ الكرفس له مشاكل عمرها 5-6 سنوات أيضًا :(

ثق بي ، من الصعب حقًا التعامل مع جميع المشكلات التي ظهرت. يرجى تحمل المسؤولية لإرسال إصلاح حيث يواجهه البعض منكم في الإنتاج ، لذا سيكون من الأسهل بالنسبة لك اختبار التغييرات المقترحة في البيئة الحقيقية.

auvipy picture auvipy في ١٢ أغسطس ٢٠١٨

@ linar-jether شكرًا ، نجح هذا الإصلاح بالنسبة لي :)

nishith107 picture nishith107 في ٢١ أغسطس ٢٠١٨
🎉1

هل يمكنكم يا رفاق التحقق من هذا العلاقات العامة https://github.com/celery/celery/pull/4997 ؟ إذا أدى هذا إلى إصلاح مشكلتك ، فقد نقوم بدمج ذلك.

auvipy picture auvipy في ٢١ سبتمبر ٢٠١٨

سأحاول هذا الفرع. سوف يستغرق الأمر بعض الوقت حتى أعرف ما إذا كان قد تم فرزها لأنني لم أجد طريقة لإجبارها على الحدوث.

DomHudson picture DomHudson في ٢١ سبتمبر ٢٠١٨
👍1

أي أخبار عن اختبار هذا الخطأ الإصلاح؟

kingb picture kingb في ١١ أكتوبر ٢٠١٨

لم أواجه المشكلة منذ أكثر من أسبوعين حتى الآن. أنا لست 100٪ حتى الآن ولكني أعتقد أن العلاقات العامة المقترحة قد أصلحتها لي

DomHudson picture DomHudson في ١١ أكتوبر ٢٠١٨
👍1

لقد نشرنا للتو الفرع 4.2 مع هذا التصحيح المطبق كعجلة مخصصة على 3 أجهزة عميل ... إنه قيد الإنتاج الآن. كان هذا الخطأ يوقف العمال مرة واحدة في الأسبوع لعدة أشهر الآن. تم نشره. سنرى ما إذا كان يمنع المشكلة.

kingb picture kingb في ١١ أكتوبر ٢٠١٨

لذلك أقوم بدمج العلاقات العامة https://github.com/celery/celery/pull/4997 ، يرجى إطلاعنا على مشكلات الإنتاج الخاصة بك هنا ، وإذا كان لدى أي شخص أي اقتراح بشأن اختبارات الوحدة / التكامل ، فيرجى المتابعة مع العلاقات العامة.

auvipy picture auvipy في ١٣ أكتوبر ٢٠١٨
🎉3 👍1

ryanpetrello يتساءل ، هل قمت بنشر هذا التصحيح واختباره؟

AvnerCohen picture AvnerCohen في ١٣ نوفمبر ٢٠١٨

AvnerCohen بسبب هذا وبعض المشكلات الأخرى ، انتهى بنا الأمر فقط إلى حل مشكلتنا دون استخدام الكرفس (تمكنا من تحقيق ما كنا بحاجة إليه باستخدام كومبو القديم البسيط). يبدو أن الآخرين قد حالفهم الحظ مع هذا التصحيح ، رغم ذلك!

ryanpetrello picture ryanpetrello في ١٣ نوفمبر ٢٠١٨
👍1

حتى الآن لم نعد على علم بأي مشاكل أخرى منذ أن بدأنا الإنتاج. أعتقد أنه أصلحها.

kingb picture kingb في ٢٢ نوفمبر ٢٠١٨

رائع شكرا!

auvipy picture auvipy في ٢٢ نوفمبر ٢٠١٨
🎉1 👍1

auvipy متى يجب أن نتوقع

AvnerCohen picture AvnerCohen في ٢٢ نوفمبر ٢٠١٨

هدفي قبل عيد الميلاد

auvipy picture auvipy في ٢٢ نوفمبر ٢٠١٨
👍2

شكرا kleptog لبعض التصحيح الرائع!

لقد جربت جميع الإصلاحات المقترحة ، حتى تشغيل الكرفس والأصدقاء من السيد ، ولكن بمجرد أن يواجه العامل خطأ ConnectionResetError ، فإنه يرفض معالجة أي مهام أخرى. تؤدي إعادة تشغيل RabbitMQ إلى قيام العمال العالقين بمعالجة المهام مرة أخرى. بطبيعة الحال ، هذا ليس حلاً جيدًا ولكنه يوجهني في اتجاه مزيد من التصحيح.

hedleyroos picture hedleyroos في ٤ ديسمبر ٢٠١٨

نحن نستخدم هذا التصحيح ولكننا ما زلنا نواجه هذه المشكلة من حين لآخر - ينهي العامل وظيفته بنجاح ويتم قفل العملية الرئيسية معلقة في انتظار النتائج

hexvolt picture hexvolt في ٧ ديسمبر ٢٠١٨

أنا أستخدم الكرفس == 4.2.1 ؛ بيثون == 3.6.7 أوبونتو 16.04. وريديس كخلفية
لقد استخدمت المشرف لبدء الكرفس بالأمر:
الكرفس - عامل المشروع - المعلومات - المقياس التلقائي = 10.2

لا يزال يتعطل في بعض الأحيان.

mikolaje picture mikolaje في ٩ ديسمبر ٢٠١٨

mikolaje ما يناسبني هو تعيين tcp-keepalive في redis.conf لحل المأزق.

hedleyroos picture hedleyroos في ١٤ ديسمبر ٢٠١٨

هل وجد أحد طريقة جيدة لمراقبة أوضاع العمال العالقين؟

ما زلنا نرى هذا السلوك مع الكود من السيد ، لا يمكننا القول ما إذا كان قد تم تحسينه أم لا ، ولكن الإحساس الأولي هو أنه على الأقل بالنسبة لحالة الاستخدام الخاصة بنا ، ما زلنا نتعثر في العمال.

السؤال هو كيفية مراقبة العاملين الفرديين؟ في الوقت الحالي ، كل ما يمكننا فعله هو مراقبة مستوى قائمة الانتظار (الرسائل موجودة ولم يتم التقاطها) ، لكني أرغب في تتبع ذلك قبل ذلك بكثير من خلال معرفة متى توقف العامل عن العمل.
ما لاحظته هو أن أمر "الحالة" من الكرفس سيعود جيدًا بغض النظر ، كما أن الزهرة تظهر أن كل شيء على ما يرام.

AvnerCohen picture AvnerCohen في ١٧ ديسمبر ٢٠١٨

لقد جربت جميع الإصلاحات المقترحة ، حتى تشغيل الكرفس والأصدقاء من السيد ، ولكن بمجرد أن يواجه العامل خطأ ConnectionResetError ، فإنه يرفض معالجة أي مهام أخرى. تؤدي إعادة تشغيل RabbitMQ إلى قيام العمال العالقين بمعالجة المهام مرة أخرى. بطبيعة الحال ، هذا ليس حلاً جيدًا ولكنه يوجهني في اتجاه مزيد من التصحيح.

hedleyroos ، قد تواجه هذه المشكلة أيضًا # 4895؟ نحن نقوم بتشغيل Celery v4.1.x لتجنب هذه المشكلة.

kingb picture kingb في ٨ يناير ٢٠١٩

kingb نعم لدي هذه المشكلة. كنت أرغب في تجنب الرجوع إلى الإصدار 4.1 وتحولت إلى Redis كوسيط ولكن ما زلت أواجه نفس المشكلة. لحسن الحظ ، لدى Redis خيار tcp-keepalive قمت بتعيينه على 60 ثانية. هذا يعني أن Redis سينهي ما يعتبره اتصالات قديمة ، وهذا بدوره يؤدي إلى خروج العامل من المأزق والبدء في استهلاك المهام مرة أخرى.

hedleyroos picture hedleyroos في ١٠ يناير ٢٠١٩

لدي تجميد للكرفس == 4.2.1 ، البلياردو == 3.5.0.5 ، kombu == 4.2.2.post1 ، amqp == 2.4.0 ، Python 3.6.8 ، RabbitMQ ، MongoDB v 4

tonal picture tonal في ١٨ يناير ٢٠١٩

tonal أستخدم نفس الإصدارات التي تستخدمها وأحيانًا أحصل على هذا الخطأ. عدم إعادة محاولة المهام أو أي شيء من هذا القبيل. تتعطل المهام في حالة الاسترداد.

fergyfresh picture fergyfresh في ٢٧ يونيو ٢٠١٩

لا يحدث ذلك طوال الوقت ، ولكنه يحدث كل بضع ساعات / كل بضعة أيام وقم بمسح الرسالة وإعادة تشغيل الخدمة.

fergyfresh picture fergyfresh في ٢٧ يونيو ٢٠١٩
👍1

4.3؟ 4.4rc2؟

auvipy picture auvipy في ٢٨ يونيو ٢٠١٩

لا ، celery==4.2.1, billiard==3.5.0.5, kombu==4.2.2.post1, amqp==2.4.0,

fergyfresh picture fergyfresh في ٢٨ يونيو ٢٠١٩

auvipy لا يزال يرى أنه يموت ~ 1 ساعة مع 4.3. إذا كان هناك أي مساعدة ، فإنه يتوقف عن النجاح / الفشل في المهام ويستمر في تلقيها ثم يفشل بصمت ، ولكن تظل عملية الكرفس قيد التشغيل.

/code # celery status
celery@celery-worker: OK

1 node online.
fergyfresh picture fergyfresh في ٩ يوليو ٢٠١٩

هل يمكن لأي شخص أن يؤكد لي أنه تم حلها بالفعل في 4.3؟ يجب أن أتخذ قرار الترقية بناءً على هذا.

partha117 picture partha117 في ٦ أغسطس ٢٠١٩
👍6

أهلا بكم،

لم يتم إصلاح هذا في 4.3. أنا أستخدم هذا الإصدار في مشروع ، ولا يزال عمالي عالقين (بعد إظهار رسائل "نبضات القلب المفقودة").

هل تحايل أي شخص على هذه المشكلة بـ 4.3؟ لقد لاحظت أن الشفرة تختلف اختلافًا كبيرًا عن الحل البديل / التصحيح أعلاه ، لذلك أعتقد أنه كانت هناك محاولة لإصلاح المشكلة ، ولكن لا يبدو أنها تم إصلاحها.

شكرا!

diogobaeder picture diogobaeder في ١٧ سبتمبر ٢٠١٩
👍2

أستطيع أن أؤكد أن لدي نفس المشكلة في Celery 4.3 (Python 3.7.3).

afshinm picture afshinm في ١٧ سبتمبر ٢٠١٩
👍3

هل يمكنك تجربة الكرفس == 4.4.0rc3؟

auvipy picture auvipy في ١٨ سبتمبر ٢٠١٩

auvipy حاول ذلك أيضًا ، ولم يساعد. فقط لمعلوماتك أنا أستخدم gevent

afshinm picture afshinm في ١٨ سبتمبر ٢٠١٩

هل هناك فرصة لإعادة فتح هذه القضية؟ يبدو أن الناس لا يزالون يواجهونها على أحدث إصدارات الكرفس.

ryanpetrello picture ryanpetrello في ١٦ أكتوبر ٢٠١٩

أنا أواجه نفس المشكلة. بيثون 2.7 ، كرفس 4.1.0. يمكنني استخدام وسيط SQS. يعالج الكرفس المهام لبعض الوقت ويتوقف فجأة بعد فترة.

praghavan039 picture praghavan039 في ١٨ أكتوبر ٢٠١٩

أنا أواجه نفس المشكلة. بيثون 2.7 ، كرفس 4.1.0. يمكنني استخدام وسيط SQS. يعالج الكرفس المهام لبعض الوقت ويتوقف فجأة بعد فترة.

يجب أن تجرب الإصدارات الأحدث.

auvipy picture auvipy في ٢٣ أكتوبر ٢٠١٩

لم نتمكن من إعادة إنتاج هذه المشكلة ، لذا سيتعين على شخص ما تصحيحها لنا وإذا كان بإمكانهم توفير حل لهذه المشكلة.

thedrow picture thedrow في ٢٣ أكتوبر ٢٠١٩

نرى هذا يوميا في الإنتاج. لم نتمكن أبدًا من إعادة الإنشاء في بيئة تمكين التطوير أو التصحيح.
أي اقتراحات حول نوع المعلومات التي يمكننا تقديمها والتي ستساعد؟

AvnerCohen picture AvnerCohen في ٢٣ أكتوبر ٢٠١٩

أقترح استخدام شيء مثل python-hunter و / أو python-manhole .
من المحتمل أن تكون المشكلة في العملية الرئيسية.
بدلاً من ذلك ، استخدم دعامة لمعرفة المقبس / الأنبوب الذي تتم قراءته أو كتابته وحاول الخصم من هناك.
يمكنني القدوم إلى مكاتبكم الأسبوع المقبل إذا كنتم ترغبون في ذلك. دعونا نناقش هذا عبر الهاتف.

thedrow picture thedrow في ٢٣ أكتوبر ٢٠١٩
👍1

شكرًا ، سنحاول هذا ، كمثال عام ، هكذا تبدو العمليات العالقة من وجهة نظر PS.

username 14182     1  1 03:12 ?        00:02:28 [celeryd:  workername<strong i="6">@hostname</strong>:MainProcess] -active- (moshe)
username 15487 14182  0 03:15 ?        00:00:12 [celeryd:  workername<strong i="7">@hostname</strong>:ForkPoolWorker-5]
username 15766 14182  0 03:18 ?        00:00:12 [celeryd:  workername<strong i="8">@hostname</strong>:ForkPoolWorker-14]
username 16666 14182  0 03:52 ?        00:00:10 [celeryd:  workername<strong i="9">@hostname</strong>:ForkPoolWorker-43]
username 20253 14182 65 06:45 ?        00:00:24 [celeryd:  workername<strong i="10">@hostname</strong>:ForkPoolWorker-110]

تم ضبط التزامن على 4 ، مع تعيين "max -asks-per-child" ، حتى نحصل على إعادة تشغيل كل رسائل X.
كما يمكن رؤيته بوضوح ، بينما لا يزال 20253 نشطًا (وعداد العامل يستمر في الزيادة) توفي 15487 مبكرًا وظل عالقًا مع:

$ strace -p 15487
strace: Process 15487 attached
futex(0x2b36980, FUTEX_WAIT_PRIVATE, 0, NULL

لا يزال المعلم نشطًا ولا يزال يتم إرسال الرسائل إلى العامل الوحيد الذي لا يزال نشطًا.

AvnerCohen picture AvnerCohen في ٢٤ أكتوبر ٢٠١٩
👍1

مجرد ترك هذا هنا في حال كان يساعد شخصًا آخر. ليس من الواضح ما إذا كان لدي نفس المشكلة بالضبط ، لكن العاملين لدي كانوا معلقين أيضًا بعد معالجة عدة مئات / آلاف المهام. كنت أقوم بإطلاق الكرفس كخدمة باستخدام:

celery multi start w1 w2 -A app.celery --time-limit=21600 --concurrency=4

بعد تحرّي الخلل وإصلاحه للعمال دون جدوى ، قمت بتغيير الأمر إلى عادي:

celery worker -A app.celery --time-limit=21600 --concurrency=4

وهذا أصلحه لي! عندما قمت بتشغيل الكرفس بهذه الطريقة يدويًا ، كان بإمكاني رؤية المهام القادمة وتشغيلها والمهام تتعطل أحيانًا (ثم يتم إعادة تشغيل العامل بشكل صحيح). أعتقد أن الطريقة السابقة لم تكن إعادة تشغيل العمال بشكل صحيح بعد فشل المهمة. كان التغيير الآخر الذي أجريته هو وضع حد زمني أقل لتلك المهام التي سيتم تشغيلها مئات / آلاف المرات ، لذلك عندما يتعطل العامل لا محالة ، ستكون عملية العامل الرئيسية قادرة على إنهاء المهمة وإعادة تشغيل واحدة جديدة بسرعة إلى حد ما قبل أن تتعثر كل منهم.

vincent31337 picture vincent31337 في ٦ نوفمبر ٢٠١٩

لا أعرف ما يكفي عن الأجزاء الداخلية للكرفس لتصحيح هذه المشكلة بشكل فعال ، ولكن إذا كان بإمكان شخص ما تقديم بعض الإرشادات أو الأماكن للبحث عنها ، فيمكنني التحقيق في الأمر من جانبنا عند حدوث المشكلة.

لا يمكنني إعادة إنشاء هذه المشكلة عند الطلب ، ولكنها تحدث بمرور الوقت في أحد مواقعنا. من خلال ما تمكنت من تعقبه ، تحدث المشكلة عندما يكون هناك انخفاض في الشبكة بين العامل وخادم RabbitMQ أثناء وجود العمال تحت الحمل. يؤدي الفشل بشكل أساسي إلى منع العملية الرئيسية من تولي أي وظائف جديدة.

إذا لم يكن النظام تحت الحمل ، فكل شيء يستمر في العمل بشكل صحيح. يجب أن يكون مزيجًا من الحمل على العامل وانخفاض الشبكة في نفس الوقت. وقد أكد هذا طريقتين ، أحدهما نظام مرحلي ثانوي بدون تحميل في ذلك الوقت يرى حالات الفشل المختلطة ولكنه يستمر في العمل بشكل صحيح بعد ذلك. ثانيًا ، نقوم بتشغيل عدة عمال لكل صندوق. فقط العمال الذين يديرون وظائف يفشلون. إذا كان الحمل خفيفًا ولم تكن جميع العمليات الرئيسية هي وظائف معالجة ، فستفشل بعض العمليات الرئيسية فقط.

أمر الإطلاق لدينا:

celery worker -n XXX-%I.${CELERY_HOST_NAME} \
    -A XXX.app --pidfile=/run/celery/XXX-%I.pid \
    --autoscale=10,3 \
    --max-tasks-per-child 1\
            -Q YYYl \
    --loglevel=INFO --logfile=/var/log/XXX/XXX-%I${CELERY_PROCESS_INDEX_SEPERATOR}.log --soft-time-limit=7200 --time-limit

إخراج السجل:

{"event": "mingle: sync with 54 nodes", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: processing reply from celery@XXXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "mingle: processing reply from celery@XXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: sync complete", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Gossip", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 2", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Tasks", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Control", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 3", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting event loop", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Autoscaler...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Pool...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "basic.qos: prefetch_count->3", "logger": "kombu.common", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', {'lang': 'py', 'task': 'XXX', 'id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"xxx}\", 'origin': 'gen49433<strong i="17">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', {'lang': 'py', 'task': 'XXX', 'id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{'xxx}\", 'origin': 'gen49438<strong i="18">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', {'lang': 'py', 'task': 'XXX', 'id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{xxx}\", 'origin': 'gen49438<strong i="19">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "celery<strong i="20">@XXX</strong> left", "logger": "celery.worker.consumer.gossip", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09] pid:111546", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1] pid:111544", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020] pid:111538", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "missed heartbeat from celery@XXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "missed heartbeat from celery@XXXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec22d68>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c5f8>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c860>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : Prev sent/recv: None/None, now - 31/490, monotonic - 4090172.991839465, last_heartbeat_sent - 4090172.991832662, heartbeat int. - 60 for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}

تتبع مكدس py-bt أو العملية الرئيسية:

 <built-in method read of module object at remote 0x7fb3e61f33b8>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 64, in __read__
    chunk = read(fd, size)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 241, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 298, in on_result_readable
    next(it)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
    worker.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
    ret = self.run(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
    return self(*args, **options)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
    return self.execute(command, argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
    sys.exit(main())

تتبع مكدس bt للعملية الرئيسية

#0  0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x0000000000542a85 in _Py_read (fd=fd@entry=56, buf=0x7fb3cecbad70, count=count@entry=4) at Python/fileutils.c:1407
#2  0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=56) at ./Modules/posixmodule.c:8043
#3  os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x7fb3ceb7f3b8, nargs=<optimized out>)
    at ./Modules/clinic/posixmodule.c.h:3625
#4  0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x7fb3ceb7f3b8,
    self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5  _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>,
    args=args@entry=0x7fb3ceb7f3b8, nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6  0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8  0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3ceb7f218, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 64, in __read__ (fd=56, buf=<_io.BytesIO at remote 0x7fb3ceca0f10>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>)) at Python/ceval.c:547
#9  _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf537420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x347a2b8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf3074f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='__read__',
    qualname='__read__') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#11 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#13 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cebfb138) at Objects/genobject.c:221
#14 gen_iternext (gen=0x7fb3cebfb138) at Objects/genobject.c:542
#15 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#16 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3481f10,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#17 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#18 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#20 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3481d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 298, in on_result_readable (fileno=56, it=<generator at remote 0x7fb3cebfb138>))
    at Python/ceval.c:547
#21 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2ed300>,
    globals=globals@entry={'__name__': 'celery.concurrency.asynpool', '__doc__': "Version of multiprocessing.Pool using Async I/O.\n\n.. note::\n\n    This module will be moved soon, so don't use it directly.\n\nThis is a non-blocking version of :class:`multiprocessing.Pool`.\n\nThis code deals with three major challenges:\n\n#. Starting up child processes and keeping them running.\n#. Sending jobs to the processes and receiving results back.\n#. Safely shutting down this system.\n", '__package__': 'celery.concurrency', '__loader__': <SourceFileLoader(name='celery.concurrency.asynpool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py') at remote 0x7fb3cf2db828>, '__spec__': <ModuleSpec(name='celery.concurrency.asynpool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py', loader_state=None, submodule_search_locations=N...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cec173a8, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=(<cell at remote 0x7fb3ceb5f9a8>, <cell at remote 0x7fb3ceb5f9d8>, <cell at remote 0x7fb3ceb5fa08>, <cell at remote 0x7fb3ceb5fa38>, <cell at remote 0x7fb3ceb5fb58>), name='on_result_readable',
    qualname='ResultHandler._make_process_result.<locals>.on_result_readable') at Python/ceval.c:3930
#22 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec260d0>, args=args@entry=0x7fb3cec173a8,
    nargs=1, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#23 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec260d0>, args=args@entry=(56,),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#24 0x0000000000422479 in do_call_core (kwdict=0x0, callargs=(56,), func=<function at remote 0x7fb3cec260d0>) at Python/ceval.c:4645
#25 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#26 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#27 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#28 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#29 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#30 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#31 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#32 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#33 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#34 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
    globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
    closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#35 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
    nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#36 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
    args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.Celery---Type <return> to continue, or q <return> to quit---
Config', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#37 0x0000000000422479 in do_call_core (kwdict=0x0,
    callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#38 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#39 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#40 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#41 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#42 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#43 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#44 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#45 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#46 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#47 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#48 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#49 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#64 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
    globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n    See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n    Number of child processes processing the queue.  The default\n    is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n    Pool implementation:\n\n    prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n    Set custom hostname (e.g., 'w1@%%h').  Expands: %%h (hostname),\n    %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n    Also run the `celery beat` periodic task scheduler.  Please note that\n    there must only be one instance of this service.\n\n    .. note::\n\n        ``-B`` is meant to be used for development purposes. For production\n        environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n    L...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
    kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
    closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#65 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
    args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#66 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#67 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'b---Type <return> to continue, or q <return> to quit---
gl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/call.c:904
#68 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245

#69 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#70 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191

#71 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#72 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
    globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
    kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#73 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,

    args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376

#74 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98

#75 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
    obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904

#76 0x0000000000494665 in slot_tp_call (
    self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_
worker.ta...(truncated), args=(),
    kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': N---Type <return> to continue, or q <return> to quit---
one, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/typeobject.c:6379

#77 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245

#78 0x0000000000422479 in do_call_core (

    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
    func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#80 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#81 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#82 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#83 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#84 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139

#85 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#86 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
    qualname='CeleryCommand.execute') at Python/ceval.c:3930
#87 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#88 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#89 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#90 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#91 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
    qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#92 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#93 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#94 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#95 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'pr---Type <return> to continue, or q <return> to quit---
eload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#96 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#97 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#98 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#99 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#100 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#101 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
    qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#102 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#103 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#104 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#105 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#106 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#107 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#108 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#109 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
    at Objects/call.c:283
#110 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#111 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#112 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#113 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#114 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
    kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#115 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
    args=0x0,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#116 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#117 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
    at Python/pythonrun.c:1035
#118 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#119 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
    at Python/pythonrun.c:429
#120 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
    filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
    fp=0x26f05c0) at Modules/main.c:427
#121 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#122 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
---Type <return> to continue, or q <return> to quit---
#123 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#124 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#125 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#126 0x0000000000429df4 in _start ()

تتبع مكدس py-bt أو عملية متشعبة:

 Traceback (most recent call first):
  <built-in method read of module object at remote 0x7fb3e61f33b8>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 422, in _recv
    chunk = read(handle, remaining)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 456, in _recv_bytes
    buf = self._recv(4)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 243, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/queues.py", line 355, in get_payload
    return self._reader.recv_bytes()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 473, in receive
    ready, req = _receive(1.0)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 351, in workloop
    req = wait_for_job()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 292, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 79, in _launch
    code = process_obj._bootstrap()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1158, in _create_worker_process
    w.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 445, in _create_worker_process
    return super(AsynPool, self)._create_worker_process(i)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1328, in _repopulate_pool
    self._create_worker_process(self._avail_index())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1343, in _maintain_pool
    self._repopulate_pool(joined)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1351, in maintain_pool
    self._maintain_pool()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 130, in _reschedules
    return fun(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
    entry()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
    worker.start()
---Type <return> to continue, or q <return> to quit---
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
    ret = self.run(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
    return self(*args, **options)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
    return self.execute(command, argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
    sys.exit(main())

تتبع مكدس bt لعملية متشعبة

#0  0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x0000000000542a85 in _Py_read (fd=fd@entry=12, buf=0x7fb3cf30b410, count=count@entry=4) at Python/fileutils.c:1407
#2  0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=12) at ./Modules/posixmodule.c:8043
#3  os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0, nargs=<optimized out>)
    at ./Modules/clinic/posixmodule.c.h:3625
#4  0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3873df0,
    self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5  _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0,
    nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6  0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8  0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3873c38, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 422, in _recv (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>, buf=<_io.BytesIO at remote 0x7fb3cecccba0>, handle=12, remaining=4)) at Python/ceval.c:547
#9  _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53ff60>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec4f1e8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf54e488, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='_recv',
    qualname='Connection._recv') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#11 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#13 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec4f048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 456, in _recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxsize=None)) at Python/ceval.c:547
#14 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5420c0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec87a78, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf543108, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='_recv_bytes', qualname='Connection._recv_bytes') at Python/ceval.c:3930
#15 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#16 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#17 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#18 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec878e0, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 243, in recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxlength=None)) at Python/ceval.c:547
#19 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53f300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec9e9e8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf538bf8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='recv_bytes', qualname='_ConnectionBase.recv_bytes') at Python/ceval.c:3930
#20 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#21 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#22 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#23 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#24 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#25 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#26 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#27 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec87728, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 445, in _recv (timeout=<float at remote 0x7fb3d01417c8>, loads=<function at remote 0x7fb3cf5d72f0>)) at Python/ceval.c:547
#28 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x3346fa0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3ced2c878, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3cec534f8>,), name='_recv', qualname='Worker._make_recv_method.<locals>._recv')
    at Python/ceval.c:3930
#29 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#30 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#31 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#32 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3346df8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 473, in receive (debug=<function at remote 0x7fb3d970b0d0>)) at Python/ceval.c:547
#33 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598a50>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=0, kwnames=0x0, kwargs=kwargs@entry=0x386d2c0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cec222c8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3ceb62708>, <cell at remote 0x7fb3ceb62fd8>), name='receive',
    qualname='Worker._make_protected_receive.<locals>.receive') at Python/ceval.c:3930
#34 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#35 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#37 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x386d058, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 351, in workloop (debug=<function at remote 0x7fb3d970b0d0>, now=<built-in method monotonic of module object at remote 0x7fb3de980d68>, pid=111793, put=<method at remote 0x7fb3cecf55c8>, inqW_fd=13, synqW_fd=None, maxtasks=1, max_memory_per_child=0, prepare_result=<method at remote 0x7fb3cec86548>, wait_for_job=<function at remote 0x7fb3cec7b268>, wait_for_syn=<function at remote 0x7fb3cec3ff28>, completed=0)) at Python/ceval.c:547
#38 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5985d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x7fb3cf599060, kwargs=kwargs@entry=0x3490068, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf536f90, defcount=defcount@entry=3, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='workloop',
    qualname='Worker.workloop') at Python/ceval.c:3930
#39 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#40 0x0000000000428c25 in call_function (kwnames=('pid',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#42 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x348feb8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/si---Type <return> to continue, or q <return> to quit---
te-packages/billiard/pool.py, line 292, in __call__ (self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898...(truncated)) at Python/ceval.c:547
#43 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf598270>,
    globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea23e20, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0,
    name='__call__', qualname='Worker.__call__') at Python/ceval.c:3930
#44 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf53db70>,
    args=args@entry=0x7ffdbea23e20, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:376
#45 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea23e20,
    callable=<function at remote 0x7fb3cf53db70>) at Objects/call.c:98
#46 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf53db70>,
    obj=obj@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
    args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:904
#47 0x0000000000494665 in slot_tp_call (
    self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated), args=(), kwds={})
    at Objects/typeobject.c:6379
#48 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
    args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:245
#49 0x0000000000422479 in do_call_core (kwdict={}, callargs=(),
    func=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated)) at Python/ceval.c:4645
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf1f5b70, args=<optimized out>, args@entry=0x7ffdbea24610,
    nargs=nargs@entry=2,
    globals=globals@entry={'__name__': 'billiard.popen_fork', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.popen_fork', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py') at remote 0x7fb3cf201748>, '__spec__': <ModuleSpec(name='billiard.popen_fork', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/popen_fork.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf201780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated))
    at Objects/call.c:283
#64 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf205c80>,
    args=args@entry=0x7ffdbea24610, nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#65 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffdbea24610,
    callable=<function at remote 0x7fb3cf205c80>) at Objects/call.c:98
---Type <return> to continue, or q <return> to quit---
#66 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf205c80>,
    obj=obj@entry=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
    args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:904
#67 0x000000000048f059 in slot_tp_init (self=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
    args=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated), kwds=0x0)
    at Objects/typeobject.c:6613
#68 0x000000000048a332 in type_call (type=<optimized out>, type@entry=0x336fdb8,
    args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
    kwds=kwds@entry=0x0) at Objects/typeobject.c:949
#69 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x336fdb8>, stack=<optimized out>,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#70 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#71 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#72 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#73 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#74 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#75 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#76 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#77 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#78 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#80 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#81 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#82 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#83 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#84 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cf1f0570, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 445, in _create_worker_process (self=<AsynPool(sched_strategy=4, synack=False, _queues={(<_SimpleQueue(_reader=<Connection(_handle=31, _readable=True, _writable=False) at remote 0x7fb3cecdecf8>, _writer=<Connection(_handle=32, _readable=False, _writable=True) at remote 0x7fb3cecde2b0>, _poll=<method at remote 0x7fb3cf23cec8>, _rlock=None, _wlock=None) at remote 0x7fb3ced47748>, <_SimpleQueue(_reader=<Connection(_handle=33, _readable=True, _writable=False) at remote 0x7fb3cecde2e8>, _writer=<Connection(_handle=34, _readable=False, _writable=True) at remote 0x7fb3cecde828>, _poll=<method at remote 0x7fb3ceb4a188>, _rlock=None, _wlock=None) at remote 0x7fb3cecde160>, None): <ForkProcess(_identity=(3731,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _pope...(truncated)) at Python/ceval.c:547
#85 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf2ed8a0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3ced34988, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3cf2e5eb8>,), name='_create_worker_process',
    qualname='AsynPool._create_worker_process') at Python/ceval.c:3930
#86 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#87 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#88 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#89 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#90 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#91 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#92 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#93 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#94 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#95 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#96 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#97 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf5299c0, args=<optimized out>, args@entry=0x7ffdbea25400,
    nargs=nargs@entry=1,
    globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated))
    at Objects/call.c:283
#98 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2d61e0>,
    args=args@entry=0x7ffdbea25400, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:322
---Type <return> to continue, or q <return> to quit---
#99 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea25400,
    callable=<function at remote 0x7fb3cf2d61e0>) at Objects/call.c:98
#100 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3cf2d61e0>, obj=<optimized out>, args=(), kwargs={})
    at Objects/call.c:904
#101 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3cecd5048>, args=args@entry=(),
    kwargs=kwargs@entry={}) at Objects/call.c:245
#102 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<method at remote 0x7fb3cecd5048>) at Python/ceval.c:4645
#103 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#104 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347e4f8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py, line 130, in _reschedules (args=(), kwargs={}, last=None, now=<float at remote 0x7fb3ceb52108>, lsince=<float at remote 0x7fb3d01419f0>)) at Python/ceval.c:547
#105 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2f05d0>,
    globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3e6246060, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=(<cell at remote 0x7fb3cec65be8>, <cell at remote 0x7fb3cec30d08>, <cell at remote 0x7fb3cec30d38>, <cell at remote 0x7fb3cec30d98>, <cell at remote 0x7fb3cec30e28>), name='maintain_pool', qualname='Pool.maintain_pool') at Python/ceval.c:3930
#106 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec6a620>, args=args@entry=0x7fb3e6246060,
    nargs=0, kwargs=kwargs@entry={}) at Objects/call.c:376
#107 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec6a620>, args=args@entry=(),
    kwargs=kwargs@entry={}) at Objects/call.c:226
#108 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<function at remote 0x7fb3cec6a620>) at Python/ceval.c:4645
#109 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#110 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf2e69c0, args=<optimized out>, args@entry=0x7ffdbea259c0,
    nargs=nargs@entry=1,
    globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated))
    at Objects/call.c:283
#111 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2ee730>,
    args=args@entry=0x7ffdbea259c0, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#112 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=1, args=0x7ffdbea259c0,
    callable=<function at remote 0x7fb3cf2ee730>) at Objects/call.c:98
#113 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf2ee730>,
    obj=obj@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwargs=kwargs@entry=0x0) at Objects/call.c:904
#114 0x0000000000494665 in slot_tp_call (self=self@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwds=kwds@entry=0x0)
    at Objects/typeobject.c:6379
#115 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<Entry at remote 0x7fb3cebb3768>,
    stack=<optimized out>, nargs=nargs@entry=0, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#116 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#117 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#118 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3480618, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py, line 145, in fire_timers (self=<Hub(timer=<Timer(max_interval=<float at remote 0x7fb3d0141a08>, on_error=None, _queue=[<scheduled at remote 0x7fb3cebf0d18>, <scheduled at remote 0x7fb3cebf0a48>, <scheduled at remote 0x7fb3cecea3b8>, <scheduled at remote 0x7fb3ceceabd8>, <scheduled at remote 0x7fb3ceb60368>, <scheduled at remote 0x7fb3cec05d18>, <scheduled at remote 0x7fb3ceb606d8>, <scheduled at remote 0x7fb3cec8f7c8>, <scheduled at remote 0x7fb3cec8f688>]) at remote 0x7fb3cf2b04e0>, readers={71: (<method at remote 0x7fb3cec80b48>, (<Connection(_connection_id='ed843206bfeb424292688d00071ac41e', authentication=(<AMQPLAIN(username='aep', password='F8P7uYpFbxyWQSJF4psFc52P6') at remote 0x7fb3ceb98a20>,), client_properties={'product': 'py-amqp', 'product_version': '2.5.1', 'capabilities': {'consumer_cancel_notify': True, 'connection.blocked': True, ...(truncated)) at Python/ceval.c:547
#119 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf550780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x7fb3cf2dfd80, kwargs=kwargs@entry=0x347c960, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf549ab0, defcount=defcount@entry=4, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='fire_timers', qualname='Hub.fire_timers') at Python/ceval.c:3930
#120 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#121 0x0000000000428c25 in call_function (kwnames=('propagate',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#122 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#123 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#124 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#125 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#126 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#127 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>,
    args=args@entry=0x347abd0, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#128 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#129 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#130 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#131 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
    globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-em---Type <return> to continue, or q <return> to quit---
IVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
    closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#132 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
    nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#133 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
    args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#134 0x0000000000422479 in do_call_core (kwdict=0x0,
    callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#135 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#136 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#137 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#138 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#139 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#140 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#141 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#142 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#143 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#144 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#145 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#146 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#147 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#148 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#149 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#150 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#151 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#152 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#153 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#154 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#155 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#156 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#157 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#158 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#159 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#160 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#161 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
    globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n    See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n    Number of child processes processing the queue.  The default\n    is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n    Pool implementation:\n\n    prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n    Set custom hostname (e.g., 'w1@%%h').  Expands: %%h (hostname),\n    %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n    Also run the `celery beat` periodic task scheduler.  Please note that\n    there must only be one instance of this service.\n\n    .. note::\n\n        ``-B`` is meant to be used for development purposes. For production\n        environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n    L...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
    kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
    closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#162 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
    args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'schedu---Type <return> to continue, or q <return> to quit---
ler': None}) at Objects/call.c:376
#163 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#164 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/call.c:904
#165 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#166 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#167 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#168 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#169 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
    globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
    kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#170 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,
    args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#171 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98
#172 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
    obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904
---Type <return> to continue, or q <return> to quit---
#173 0x0000000000494665 in slot_tp_call (
    self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated), args=(),
    kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/typeobject.c:6379
#174 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#175 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
    func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#176 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#177 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#178 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#179 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#180 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#181 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#182 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#183 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
    qualname='CeleryCommand.execute') at Python/ceval.c:3930
#184 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#185 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#186 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#187 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_per---Type <return> to continue, or q <return> to quit---
iodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#188 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
    qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#189 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#190 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#191 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#192 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#193 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#194 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#195 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#196 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#197 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#198 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
    qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#199 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#200 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#201 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#202 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#203 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#204 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#205 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#206 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
    at Objects/call.c:283
#207 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#208 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#209 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#210 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#211 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
    kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#212 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
    args=0x0,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#213 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#214 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
    at Python/pythonrun.c:1035
#215 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': ---Type <return> to continue, or q <return> to quit---
None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#216 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
    at Python/pythonrun.c:429
#217 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
    filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
    fp=0x26f05c0) at Modules/main.c:427
#218 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#219 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
#220 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#221 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#222 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#223 0x0000000000429df4 in _start ()

التبعيات:

celery==4.3.0
kombu==4.6.5
billard==3.6.1.0
amqp==2.5.1
python==3.7.3
wakemaster39 picture wakemaster39 في ١٨ نوفمبر ٢٠١٩

هل يمكنك تجربة الكرفس == 4.4.0rc4؟

auvipy picture auvipy في ١٨ نوفمبر ٢٠١٩

سوف يستغرق الأمر مني أسبوعًا لفحصها في مرحلة الإعداد ، لكنني سأحاول الحصول على بعض العبء الذي يعمل ضده في الإنتاج.

wakemaster39 picture wakemaster39 في ١٨ نوفمبر ٢٠١٩
👍2

@ wakemaster39 : هل ساعد هذا؟

zeeshan2709 picture zeeshan2709 في ٢٧ نوفمبر ٢٠١٩

من الغريب أيضًا أن يكون هذا قد أصلحه @ wakemaster39

aaronn picture aaronn في ١١ ديسمبر ٢٠١٩

الكرفس == 4.4.0rc5؟

auvipy picture auvipy في ١١ ديسمبر ٢٠١٩

لقد انجذبت إلى الاضطرار إلى حل مشكلات مختلفة في الأسبوعين الماضيين ، وأصبح الموقع الذي نواجه مشكلات معه غير متصل بالإنترنت خلال العطلات ، لذلك لا أريد إجراء أي تغييرات هناك. أقوم بتدقيقها في الإعداد للأسبوع المقبل في النصف على أمل أن يتم نشرها لتحفيز العام الجديد للحصول على بعض العبء ضدها.

wakemaster39 picture wakemaster39 في ١١ ديسمبر ٢٠١٩

هل نعلم ما إذا كانت هذه المشكلة تتعلق بعمال الكرفس على وجه التحديد وليس أجزاء أخرى من الكرفس مثل beat ؟ Ergo إذا قمت بالتبديل إلى شيء مثل go-celery واستمررت في استخدام celery.beat هل هذا من الناحية النظرية يحل مشكلتي؟ أو هل عانى الأشخاص من مشاكل مع توقف beat لأي سبب كان أيضًا؟

aaronn picture aaronn في ١١ ديسمبر ٢٠١٩

في حالتي ، لم يكن الأمر كذلك حتى يموت ، وخيوط عامل الكرفس في طريق مسدود. لقد أصلحته لي عن طريق إضافة مهلة ، لكن الخيط يتوقف بين الحين والآخر. لقد حدث الأمر أكثر عندما كان لدي بعض معاملات قاعدة البيانات التي لم تكن ذرية ، ولكن إصلاح ذلك أدى إلى إصلاح بعض الصداع.

tldr: إنه ليس فوزًا ، ولكن العامل الذي "يتعطل عشوائيًا على IPC"

fergyfresh picture fergyfresh في ١١ ديسمبر ٢٠١٩
👍4

رأيت أشياء مماثلة تعمل على الكرفس = "== 4.4.0". يعمل الخفق بشكل جيد ولكن الكرفس عالق لسبب ما. من المحتمل أن يكون هناك طريق مسدود ولكن من الصعب جدا تحديد ذلك

njfix6 picture njfix6 في ٩ يناير ٢٠٢٠

كانت لي تجربة مماثلة لـ @ vincent31337 . أعتقد أنه تم تجنب ذلك من خلال عدم استخدام الأمر celery multi start .

DomHudson picture DomHudson في ٩ يناير ٢٠٢٠

أنا أواجه نفس المشكلة. بيثون 2.7 ، كرفس 4.1.0. يمكنني استخدام وسيط SQS. يعالج الكرفس المهام لبعض الوقت ويتوقف فجأة بعد فترة.

يجب أن تجرب الإصدارات الأحدث.

تم إصلاحه بعد الترقية.

praghavan039 picture praghavan039 في ٢٨ يناير ٢٠٢٠

أود أن أعرف ما إذا كان لا يزال الأمر كذلك مع الكرفس => 4.4.x

auvipy picture auvipy في ٣١ يناير ٢٠٢٠

أنا أتعامل مع هذه المشكلة نفسها:

بيثون 3.8.0
جانغو 3.0.3
الكرفس 4.4.0
redis 3.4.1 (للوسيط والنتائج)

أنا أدير خمسة عمال. يمكنني تشغيل خمس مهام مكتملة ، ثم يتوقف العامل عن العمل عند read([different pipe value for each] . بعد أن يتم قبول المهام وظهورها في Inspect(app=app).active() ، ومع ذلك ، فإن worker_pid الخاص بهم ليس أحد المصفوفات الخاصة بالعاملين. غالبًا ما تكون المصفوفات ذات قيم منخفضة مثل 50. باستخدام ps -p [low pid] لا أرى هذه المربعات قيد التشغيل بالفعل. لم أر ذلك مذكورًا حتى الآن في هذا الموضوع.

JosiahDub picture JosiahDub في ٢٤ فبراير ٢٠٢٠

حتى مع الكرفس == 4.4.2 أليس كذلك؟

auvipy picture auvipy في ٢٠ مارس ٢٠٢٠

نحن نواجه نفس المشكلة بالضبط - مع الكرفس == 4.3 وكذلك تجربة أحدث الكرفس = 4.4.2
نحن نستخدم rabbitmq كخلفية (وسيط ونتائج)

كومبو == 4.6.8
الكرفس == 4.4.2
amqp == 2.5.2
البلياردو == 3.6.3.0
librabbitmq == 2.0.0
بايثون 3.7.4

الكرفس عالق في القراءة ولا يمكنه معالجة أي مهام أخرى.

$ sudo strace -p 8553
strace: Process 8553 attached
read(10, ^Cstrace: Process 8553 detached
 <detached ...>

$ sudo strace -p 8557
strace: Process 8557 attached
read(14, ^Cstrace: Process 8557 detached
 <detached ...>

$ sudo strace -p 8561
strace: Process 8561 attached
read(18, ^Cstrace: Process 8561 detached
 <detached ...>

مثال واحد lsof ourput لعملية 8553
celery 8553 ubuntu 10r FIFO 0,10 0t0 43059 pipe

kehpcisco picture kehpcisco في ٢٠ مارس ٢٠٢٠

https://github.com/celery/celery/issues/4185#issuecomment -545617121

auvipy picture auvipy في ٢٠ مارس ٢٠٢٠

FWIW يبدو أننا واجهنا نفس المشكلة مع استخدام خام billiard==3.6.3.0 (بصرف النظر عن الكرفس):
الأبوين:

(gdb) py-bt                                                                                
Traceback (most recent call first):             
  <built-in method acquire of _thread.lock object at remote 0x7f838895e620>
  File "/usr/lib/python3.6/threading.py", line 295, in wait                           
    waiter.acquire()
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1957, in next
    self._cond.wait(timeout)
  File "xxx.py", line 1063, in main
    for res in self.pool.imap_unordered(_partial, itertools.islice(a_generator, nb_processes)):
...

عملية طفل عالق:

0x00007f8393cc11c4 in __GI___libc_read (fd=21, buf=0x7f838892c578, nbytes=4) at ../sysdeps/unix/sysv/linux/read.c:27
Traceback (most recent call first):
  <built-in method read of module object at remote 0x7f8392d3de58>
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 422, in _recv
    chunk = read(handle, remaining)
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 456, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 243, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 395, in get_payload
    return self._reader.recv_bytes()
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
...

هذا هو نفس المكدس الموجود في https://github.com/celery/celery/issues/4185#issuecomment -554828564 py-bt stack trace or forked process

جميع عمليات الأطفال الأخرى في semlock:

Traceback (most recent call first):
  <built-in method __enter__ of _multiprocessing.SemLock object at remote 0x7f838896ff10>
  File "/usr/local/lib/python3.6/dist-packages/billiard/synchronize.py", line 116, in __enter__
    return self._semlock.__enter__()
  File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 394, in get_payload
    with self._rlock:
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
...
thomas-riccardi picture thomas-riccardi في ١٧ سبتمبر ٢٠٢٠
👍1

لقد لاحظت مشكلة العمال الذين يتقادمون في العديد من حالات OpenWISP.

celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0

أمر:

celery -A openwisp2 worker -l info --pool=gevent --concurrency=10 -Ofair

الاختلاف الوحيد الذي يمكنني رؤيته بين عامل عامل وآخر لا يعمل هو أن العامل الذي لا يقوم بمعالجة أي مهمة يظهر ذلك في الناتج strace :

select(22, [21], [], [21], {tv_sec=0, tv_usec=0}) = 0 (Timeout)

حتى إذا كان العامل يقوم بمعالجة المهام ، أرى هذين النوعين من الأسطر في إخراج strace:

epoll_ctl(4, EPOLL_CTL_ADD, 20, {EPOLLIN, {u32=20, u64=12884901908}}) = -1 EEXIST (File exists)
recvfrom(20, 0x377a460, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

لذلك لا أعتقد أن سطري السجل الأخيرين يشيران إلى مشكلة أساسية ، أو على الأقل أن هذه ليست أعراض المشكلة.

nemesisdesign picture nemesisdesign في ١٦ أكتوبر ٢٠٢٠ 
# strace -p 81
strace: Process 81 attached
read(24, ^Cstrace: Process 81 detached
 <detached ...>

# lsof -p 81 | grep 24
celery   81 root  mem       REG              252,1             535124 /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (path dev=0,112)
celery   81 root   24r     FIFO               0,13      0t0 311875006 pipe



md5-a6754624afaf3bd93564d53ec027c759



python version: 3.6
celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0
yywing picture yywing في ٤ نوفمبر ٢٠٢٠

مرحبا،

لقد كنت أتعامل مع نفس المشكلة هنا:

تقرير فحص الكرفس:

software -> celery:4.3.0 (rhubarb) kombu:4.6.4 py:3.6.6
                    billiard:3.6.1.0 py-amqp:2.5.1
        platform -> system:Linux arch:64bit
                    kernel version:3.10.0-1062.18.1.el7.x86_64 imp:CPython
        loader   -> celery.loaders.app.AppLoader
        settings -> transport:amqp results:amqp

        broker_url: 'amqp://nextgen:********@*:5672/ng_vhost'
        result_backend: 'amqp'
        task_queues: 
            (<unbound Queue default -> <unbound Exchange default(direct)> -> default>,
         <unbound Queue runForecasts -> <unbound Exchange runForecasts(direct)> -> runForecasts>,
         <unbound Queue runEnsembles -> <unbound Exchange runEnsembles(direct)> -> runEnsembles>)
        task_default_queue: 'default'
        task_default_exchange: 'default'
        task_default_routing_key: '********'
        broker_heartbeat: None
        task_acks_late: True
        worker_prefetch_multiplier: 1

CELERYD_OPTS:
--time-limit=3600 -c:worker1 3 -Q:worker1 runForecasts -c:worker2 2 -Q:worker2 runEnsembles -c:worker3 2 -Q:worker3 default

يبدو أن قائمة انتظار runForecasts فقط تتعثر ، وإليك إخراج الدعامة للعمليات الفرعية:

$ sudo strace -p 17426
strace: Process 17426 attached
read(28, ^Cstrace: Process 17426 detached
 <detached ...>
$ sudo ls -l /proc/17426/fd/28
lr-x------. 1 wind wind 64 Nov 10 22:45 /proc/17426/fd/28 -> pipe:[284399927]

لقد وجدت الأنبوب الذي تم تعليقه ، لكن لا أعرف كيفية تصحيح الأخطاء أكثر. سأكون ممتنًا إذا كان بإمكان أي شخص إلقاء بعض الضوء على هذا.

jfb10d picture jfb10d في ١٠ نوفمبر ٢٠٢٠ 
# strace -p 32
strace: Process 32 attached
epoll_wait(7, [], 1023, 787)            = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 4050)           = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [{EPOLLIN, {u32=15, u64=139972984176655}}], 1023, 275) = 1
ioctl(15, FIONBIO, [1])                 = 0
recvfrom(15, "\10\0\0\0\0\0\0", 7, 0, NULL, NULL) = 7
recvfrom(15, "\316", 1, 0, NULL, NULL)  = 1
ioctl(15, FIONBIO, [0])                 = 0
ioctl(15, FIONBIO, [1])                 = 0
recvfrom(15, 0x7f4e96911aa0, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
ioctl(15, FIONBIO, [0])                 = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 19)             = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 670)            = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000)           = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000)           = 0

lrwx------. 1 root root 64 Dec  3 08:21 7 -> 'anon_inode:[eventpoll]'

الكرفس == 4.4.7

وظائف العاملين: - المقياس التلقائي و - المهام القصوى لكل طفل

yywing picture yywing في ٣ ديسمبر ٢٠٢٠

سأختبر هذا.

أستخدم --autoscale 20,1 و --max-tasks-per-child 1

مهمة

<strong i="10">@share_task</strong>
def echo():
      print(1)

# wait for processes scaling down and run next
for i in range(100):
     echo.delay()


# once large call
for i in range(10000):
     echo.delay()
yywing picture yywing في ٣ ديسمبر ٢٠٢٠

لدي خطأ آخر ：

[2020-12-03 18:03:42,137: CRITICAL/MainProcess] Unrecoverable error: AssertionError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 303, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
    entry()
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 636, in verify_process_alive
    assert proc.outqR_fd in hub.readers
AssertionError
yywing picture yywing في ٣ ديسمبر ٢٠٢٠

من https://github.com/celery/celery/issues/4185#issuecomment -694179347
لقد راجعت billiard / pool.py line 1957 ووجدت:

    self._cond = threading.Condition(threading.Lock())
    ......
    self._lost_worker_timeout = lost_worker_timeout
    ......
    def next(self, timeout=None):
        with self._cond:
                ......
                self._cond.wait(timeout)
                ......

self._cond.wait لم يتم تعيين مهلة. هل عالق في هذا المكان؟

yywing picture yywing في ١٤ ديسمبر ٢٠٢٠

لأولئك الذين يواجهون هذا ، هل يمكنك من فضلك محاولة https://github.com/celery/billiard/pull/325؟

auvipy picture auvipy في ١٤ ديسمبر ٢٠٢٠
👍1

https://github.com/celery/billiard/pull/325 قد لا يعمل.

وأنا أستسلم. الآن ، أستخدم --pool threads

yywing picture yywing في ١٧ ديسمبر ٢٠٢٠
👀1 😄1
هل كانت هذه الصفحة مفيدة؟
0 / 5 - 0 التقييمات

القضايا ذات الصلة

jheld picture jheld  ·  3تعليقات
myisis picture myisis  ·  3تعليقات
aTylerRice picture aTylerRice  ·  3تعليقات
Xuexiang825 picture Xuexiang825  ·  3تعليقات
maxim25 picture maxim25  ·  3تعليقات