<p>les ouvriers de céleri se bloquent au hasard sur IPC (provoquant la file d'attente de tous les travaux, mais ne sont plus exécutés)</p>

J'ai le même problème ici, les travailleurs ont l'air de courir mais ils ne traitent pas les messages.

J'utilise Python 3.6 et utilise Celery 4.1.0

@codeadict êtes-vous en mesure de suivre les deux processus (travailleur et parent) pour voir s'ils sont tous les deux bloqués sur le même descripteur de fichier?

Vous pouvez vérifier qu'il s'agit du même fd avec sudo lsof | grep 53r où 53r est le numéro du descripteur de fichier sur lequel les processus sont bloqués.

@codeadict comme @chrismeyersfsu l'a mentionné, vous pouvez strace le processus de céleri maître et voir s'il est bloqué en lecture sur un tuyau:

$ sudo strace -p <pid-of-celery-master-process> -s 100
Process X attached
read(53

Si vous avez installé python-dbg avec des symboles Python, vous pouvez également utiliser gdb pour voir où le maître est bloqué:

$ gdb python -p <pid-of-master-process>
...
(gdb) py-list
 216            # header
 217
 218            while Hr < 4:
 219                try:
 220                    n = __read__(
>221                        fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,

https://github.com/celery/celery/blob/v3.1.17/celery/concurrency/asynpool.py#L219

Même problème.

céleri == 3.1.23, billard == 3.3.0.23
Les travailleurs ont commencé avec des options: -time-limit=7200 --autoscale=64,2 --maxtasksperchild=1
python 2.7.3

Après quelques heures, le céleri `` gèle ''
Je vois que le maître et l'infini de l'enfant attendent read () sur les tuyaux (btw, différents tuyaux).

Donc, c'est le processus principal:

root<strong i="13">@vm1</strong>:~# strace -p 41840
Process 41840 attached - interrupt to quit
read(50, ^C <unfinished ...>
Process 41840 detached

root<strong i="14">@vm1</strong>:~# sudo lsof | grep 50r
python     41840                 root   50r     FIFO                0,8        0t0     548796 pipe
python     46237                 root   50r      CHR                1,3        0t0       7604 /dev/null
python     46304                 root   50r     FIFO                0,8        0t0     548796 pipe


root<strong i="15">@vm1</strong>:~# ls -lh /proc/41840/fd/50
lr-x------ 1 root root 64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]

root<strong i="16">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[548796]') 2>/dev/null
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/51 -> pipe:[548796]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/51 -> pipe:[548796]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/51 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/50 -> pipe:[548796]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/51 -> pipe:[548796]

Et un des enfants:

root<strong i="5">@vm1</strong>:~# strace -p 46304
Process 46304 attached - interrupt to quit
read(16, ^C <unfinished ...>
Process 46304 detached

root<strong i="6">@vm1</strong>:~# ls -lh /proc/46304/fd/16
lr-x------ 1 root root 64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]


root<strong i="7">@vm1</strong>:~# (find /proc -type l | xargs ls -l | fgrep 'pipe:[482281]') 2>/dev/null
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/41840/task/41840/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:02 /proc/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/16 -> pipe:[482281]
l-wx------ 1 root    root    64 Oct 20 10:07 /proc/46237/task/46237/fd/17 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:02 /proc/46304/fd/16 -> pipe:[482281]
lr-x------ 1 root    root    64 Oct 20 10:07 /proc/46304/task/46304/fd/16 -> pipe:[482281]
root<strong i="8">@vm1</strong>:~# 

Et cette image `` gelée '' actuelle dans le processus maître:

Thread 0x7efc1c101700
('Frame: ', 63347776)
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
    blueprint.start(self)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 838, in start
    c.loop(*c.loop_args())
  File "/opt/waf/python/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
    next(loop)
  File "/opt/waf/python/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
    cb(*cbargs)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 279, in on_result_readable
    next(it)
  File "/opt/waf/python/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 221, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
()

@harlov @codeadict @chrismeyersfsu Avez-vous trouvé une solution ou une solution de contournement?

Non. Nous passons au céleri 4. Nous exécuterons la même série de tests qui invoquent cette erreur et verrons si nous pouvons recréer sur céleri 4.

@korycins nous retirons - autoscale et - maxtasksperchild - tout encore bien.

Je pense que les travailleurs qui réapparaissent avec autoscale ® maxtasksperchild peuvent être à l'origine de cette situation.

@chrismeyersfsu J'ai le même problème avec le céleri 4.1, donc je suppose que passer à la v4.1 ne vous aidera pas.
Auparavant, je n'utilisais que l'option --autoscale. Les ouvriers s'accrochent de temps en temps (environ une fois par semaine) mais quand j'ajoute maxtasksperchild = 20, les ouvriers s'accrochent environ une fois par heure. Je pensais que ce problème pouvait être lié à maxtasksperchild, alors je l'ai supprimé.
J'ai créé une mise à l'échelle automatique personnalisée pour augmenter / diminuer dynamiquement en fonction des ressources libres et le même problème se produit, donc je suppose que le problème est lié à la réapparition des travailleurs (comme @harlov l'a dit)

quelqu'un peut-il vérifier les problèmes sur le dernier maître? Je ne sais pas si c'est déjà corrigé avec le dernier commit.

@auvipy quel (s) commit (s) pensez-vous pouvoir résoudre ce problème? Je suis actuellement en mesure de reproduire ce problème sur céleri 4.1.0, billard 3.5.0.3 de PyPI.

Je ne suis pas sûr, alors pourriez-vous essayer la branche principale? Si, après la sortie de la version 4.2, il existe toujours, un travail serait effectué pour corriger

@auvipy nous pourrions lui donner un _try_, mais il peut être difficile de le dire avec certitude - _personnellement_, nous semblons rencontrer ce blocage une fois toutes les quelques _ semaines_, donc c'est assez difficile à réduire.

@ryanpetrello Essayez d'ajouter max_task_per_child et définissez-le sur 1. http://docs.celryproject.org/en/latest/userguide/workers.html#max -tasks-per-child-setting
Cela devrait augmenter la possibilité d'occurrence du problème

Même problème, ne peut pas se reproduire pour le moment, je pensais que c'était un problème de configuration, lors du redémarrage, il suffit de décrocher le travailleur, mais il semble que cela se produise de temps en temps.

Je viens de dire que nous nous heurtons également à ce problème. céleri 4.1.0, kombu 4.1.0, billard 3.5.0.3.
Ligne de commande: /usr/bin/python /usr/local/bin/celery worker --app myapp.celery -B --loglevel INFO --autoscale=5,1 --max-tasks-per-child=50 --max-memory-per-child=1000000 -O fair

En ce qui concerne le blocage sur les descripteurs de fichiers, nous voyons la même chose que @harlov ci-dessus: le maître est dans une impasse avec l'un de ses travailleurs.

L'envoi de SIGUSR1 au maître céleri "décolle" le processus et vide le stacktrace ci-dessous. Ce qui est suspect, c'est que dans une fonction "on_result_readable" il se bloque sur une lecture. Cela ressemble à une condition de concurrence classique: poll () renvoie lisible mais les blocs read (). Cela peut arriver, c'est pourquoi il est toujours recommandé de définir les FD en mode non-bloc pour s'assurer que rien ne va pas.

Message pertinent sur LKML: https://lkml.org/lkml/2011/6/18/105

Je ne suis pas sûr qu'il soit facile de définir les descripteurs non bloquants, car nous avons un environnement avec de nombreux maîtres de céleri et il y en a toujours quelques-uns bloqués, si c'est un simple changement, nous pourrons peut-être le tester rapidement.

MainThread                                                                                                                                                                                                                            [636/660]
=================================================                                                                                                                                                    
  File "/usr/local/bin/celery", line 11, in <module>                                                                                                                                                 
    sys.exit(main())                                                                                                                                                                                 
  File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 14, in main                                                                                                             
    _main()                                                                                                                                                                                          
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 326, in main                                                                                                              
    cmd.execute_from_commandline(argv)                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline                                                                                                            
    super(CeleryCommand, self).execute_from_commandline(argv)))                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 281, in execute_from_commandline                                                                                        
    return self.handle_argv(self.prog_name, argv[1:])                                                                                                              
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 480, in handle_argv                                                                                                       
    return self.execute(command, argv)                                                                                                                                                           
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 412, in execute                                                                                                           
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])                                                                                                                                   
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 221, in run_from_argv                                                                                                                       
    return self(*args, **options)                                                                                                                                                                    
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 244, in __call__                                                                                                            
    ret = self.run(*args, **kwargs)                                                                                                                                                                  
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 256, in run                                                                                                          
    worker.start()                                                                                                                                                                                   
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 203, in start                                                                                                     
    self.blueprint.start(self)                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start                                                                                                                                
    step.start(parent)                                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 370, in start                                                                                                              
    return self.obj.start()                                                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 320, in start                                                                                               
    blueprint.start(self)                                                                                                                                                                            
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start                                                                                                             
    step.start(parent)                                                                                                                                                                               
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 596, in start                                                                                               
    c.loop(*c.loop_args())                                                                                                                                                                           
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/loops.py", line 88, in asynloop                                                                                                         
    next(loop)                                                                                                                                                                                       
  File "/usr/local/lib/python2.7/dist-packages/kombu/async/hub.py", line 354, in create_loop                                                                        
    cb(*cbargs)                                                                                                                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 292, in on_result_readable                                                                                      
    next(it)
  File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py", line 235, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
  File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 349, in cry_handler
    safe_say(cry())
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/debug.py", line 193, in cry
    traceback.print_stack(frame, file=out)
=================================================
LOCAL VARIABLES
=================================================
{'P': <functools.partial object at 0x7f29da0f1c58>,
 'frame': <frame object at 0x37137b0>,
 'out': <vine.five.WhateverIO object at 0x7f29da14e150>,
 'sep': u'=================================================',
 'sepchr': u'=',
 'seplen': 49,
 'thread': <_MainThread(MainThread, started 139818014390016)>,
 'threading': <module 'threading' from '/usr/lib/python2.7/threading.pyc'>,
 'tid': 139818014390016,
 'tmap': {139818014390016: <_MainThread(MainThread, started 139818014390016)>}}

Juste une mise à jour. Nous avons passé du temps à tester les choses et à essayer de les reproduire.

• Définir max_tasks_per_child sur 1 aggrave la situation
• Supprimer -O fair ne fait aucune différence
• La suppression de la mise à l'échelle automatique n'aide pas non plus
• Nous ne l'avons jusqu'à présent reproduit que sur des machines avec noyau 4.9 (Stretch), pas sur des machines avec 3.16 (Jessie)

J'ai vérifié que les descripteurs de fichiers normalement utilisés par le céleri sont tous correctement marqués non bloquants et ils le sont. La question est donc de savoir comment le code se bloque si chaque descripteur est non bloquant.

La piste que je suis maintenant est que l'objet epoll () devient confus quant aux descripteurs qu'il est censé regarder. Si vous stracez du céleri, vous obtenez beaucoup de résultats comme:

14659 14:32:56.665937 epoll_ctl(4, EPOLL_CTL_ADD, 43, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=43, u64=14879199376994992171}}) = -1 EEXIST (File exists)
14659 14:32:56.667084 epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=16, u64=14879199376994992144}}) = -1 EEXIST (File exists)
14659 14:32:56.667278 epoll_ctl(4, EPOLL_CTL_ADD, 26, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=26, u64=14879199376994992154}}) = -1 EEXIST (File exists)
14659 14:32:56.667440 epoll_ctl(4, EPOLL_CTL_ADD, 23, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=23, u64=14879199376994992151}}) = -1 EEXIST (File exists)
14659 14:32:56.667607 epoll_ctl(4, EPOLL_CTL_DEL, 37, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)
14659 14:32:56.667804 epoll_ctl(4, EPOLL_CTL_DEL, 38, 0x7ffd31c17f60) = -1 ENOENT (No such file or directory)

Il est tout à fait plausible qu'à un moment donné, un descripteur de fichier qui était autrefois utilisé pour une chose soit utilisé pour autre chose, le hub soit confus et appelle la mauvaise méthode et boum!

Ok, donc j'ai trouvé le problème, et j'ai une sortie strace qui a échoué. Et c'est très subtil et à voir avec la «fonctionnalité» complètement non évidente suivante de l'interface epoll (). Depuis la page de manuel:

Q6
La fermeture d'un descripteur de fichier entraînera-t-il sa suppression automatique de tous les ensembles epoll?
A6
Oui, mais soyez conscient du point suivant. Un descripteur de fichier est une référence à une description de fichier ouvert (voir open (2)). Chaque fois qu'un descripteur de fichier est dupliqué via dup (2), dup2 (2), fcntl (2) F_DUPFD ou fork (2), un nouveau descripteur de fichier faisant référence à la même description de fichier ouvert est créé. Une description de fichier ouverte continue d'exister jusqu'à ce que tous les descripteurs de fichier s'y référant aient été fermés. Un descripteur de fichier est supprimé d'un ensemble epoll uniquement après que tous les descripteurs de fichier se référant à la description de fichier ouvert sous-jacent ont été fermés (ou avant si le descripteur de fichier est explicitement supprimé en utilisant epoll_ctl (2) EPOLL_CTL_DEL). Cela signifie que même après la fermeture d'un descripteur de fichier faisant partie d'un ensemble epoll, des événements peuvent être signalés pour ce descripteur de fichier si d'autres descripteurs de fichier se référant à la même description de fichier sous‐ jacente restent ouverts.

Ce qui se passe, c'est qu'un descripteur de fichier est utilisé par le céleri pour la communication, le descripteur de fichier fuit vers un ouvrier, le maître ferme le descripteur de fichier puis en ouvre un autre pour la sentinelle _qui n'est pas marquée comme non bloquante_. Le processus enfant meurt, ce qui fait que le descripteur de fichier est marqué comme lisible, le maître essaie de le lire et bloque.

Oui, epoll () renvoie la lisibilité pour un descripteur de fichier qui n'est plus ouvert dans ce processus.

Ici, je colle la sortie strace capturée pour seulement fd 49 comme preuve (excuses pour la sortie multiligne, strace surveillait plusieurs processus):

7417  15:04:00.001511 pipe( <unfinished ...>
7417  15:04:00.001646 <... pipe resumed> [49, 50]) = 0
7417  15:04:00.001727 fcntl(49, F_GETFL <unfinished ...>
7417  15:04:00.002091 <... fcntl resumed> ) = 0 (flags O_RDONLY)
7417  15:04:00.002165 fcntl(49, F_SETFL, O_RDONLY|O_NONBLOCK <unfinished ...>
7417  15:04:00.002222 <... fcntl resumed> ) = 0
7417  15:04:00.003052 fcntl(49, F_GETFL <unfinished ...>
7417  15:04:00.003195 <... fcntl resumed> ) = 0x800 (flags O_RDONLY|O_NONBLOCK)
...
7417  15:04:00.237131 fcntl(49, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK)
7417  15:04:00.237363 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417  15:04:00.237478 <... epoll_ctl resumed> ) = 0
...
7417  15:04:00.274482 read(49,  <unfinished ...>
7417  15:04:00.274558 <... read resumed> "\0\0\0\25", 4) = 4
7417  15:04:00.274670 read(49,  <unfinished ...>
7417  15:04:00.274734 <... read resumed> "(I15\n(I9395\ntp0\ntp1\n.", 21) = 21
7417  15:04:00.274851 epoll_ctl(4, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=49, u64=12923050543437316145}} <unfinished ...>
7417  15:04:00.274970 <... epoll_ctl resumed> ) = -1 EEXIST (File exists)
...
... from here fd 49 is used normally for a while,  have not included all the output ...
...
7417  15:04:52.871646 epoll_wait(4, [{EPOLLIN, {u32=49, u64=12923050543437316145}}, {EPOLLIN, {u32=33, u64=12923050543437316129}}, {EPOLLOUT, {u32=48, u64=12923050543437316144}}, {EPOLLOUT, {u32=8, u64=12923050543437316104}}], 1023, 155) = 4
7417  15:04:52.871888 read(49, "\0\0\0\31", 4) = 4
7417  15:04:52.872079 read(49, "(I4\n(I9400\nI155\ntp0\ntp1\n.", 25) = 25
... here it uses normal poll for a once?
7417  15:04:52.891818 poll([{fd=49, events=POLLERR}], 1, 10 <unfinished ...>
7417  15:04:52.902239 <... poll resumed> ) = 0 (Timeout)
7417  15:04:52.913780 close(49 <unfinished ...>
7417  15:04:52.914020 <... close resumed> ) = 0
... here we create a new pipe for a child process
7417  15:04:53.148195 pipe([49, 50])    = 0
7417  15:04:53.148996 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f2dee7f79d0) = 9482
... the write end got closed, now we dup it. This is the sentinal fd so is not marked non-blocking
7417  15:04:53.155409 dup(49)           = 50
7417  15:04:53.156030 epoll_ctl(4, EPOLL_CTL_ADD, 50, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=50, u64=12923050543437316146}}) = 0
... and here it breaks
7417  15:04:53.776974 epoll_wait(4,  <unfinished ...>
7417  15:04:53.785786 <... epoll_wait resumed> [{EPOLLHUP, {u32=49, u64=12923050543437316145}}], 1023, 1878) = 1
7417  15:04:53.785876 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9481, si_uid=905, si_status=155, si_utime=4, si_stime=3} ---
7417  15:04:53.786073 read(49,  <unfinished ...>
... discontinuity ...
7417  15:09:24.338109 <... read resumed> 0x23f4f10, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
7417  15:09:24.345507 epoll_ctl(4, EPOLL_CTL_DEL, 49, 0x7ffeecbf7660) = -1 ENOENT (No such file or directory)

Notez comment epoll () admet que fd 49 n'est plus dans son fdset, mais il a quand même renvoyé le nombre.

La façon de résoudre ce problème est de s'assurer que dans le worker, chaque descripteur de fichier surveillé par epoll () est fermé. Actuellement, les nœuds de calcul ferment les descripteurs de fichier dont ils ne veulent pas qu'ils correspondent à eux-mêmes, mais ils ne touchent pas aux descripteurs de fichier dont ils héritent et qui se rapportent à d'autres nœuds de calcul. Les solutions de contournement sont de marquer le sentinal fd également comme non bloquant, ou simplement d'utiliser poll () au lieu de epoll (). Je teste ce dernier localement.

Ok, il s'avère que l'utilisation de poll () ne résout pas le problème. Comme le problème réel est qu'un descripteur de fichier est en cours de fermeture alors qu'il est toujours enregistré auprès du concentrateur. Une fois que vous vous rendez compte qu'il est facile de déterminer exactement où cela se passe et que le hack suivant résout ce problème pour nous. Sans cela, nous pourrions reproduire le problème en une heure ou deux. Avec lui, nous n'avons pas encore pu le déclencher. Le hack n'est pas la bonne solution, mais il faut vraiment que quelqu'un connaissant mieux le code écrive une bonne solution.

--- /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py~  2018-03-16 01:10:04.000000000 +0000
+++ /usr/local/lib/python2.7/dist-packages/celery/concurrency/asynpool.py   2018-03-16 12:08:58.058686003 +0000
@@ -735,6 +735,7 @@
             except KeyError:
                 pass
         self.on_inqueue_close = on_inqueue_close
+        self.hub_remove = hub_remove

         def schedule_writes(ready_fds, total_write_count=[0]):
             # Schedule write operation to ready file descriptor.
@@ -1241,6 +1242,7 @@
             if queue:
                 for sock in (queue._reader, queue._writer):
                     if not sock.closed:
+                        self.hub_remove(sock)
                         try:
                             sock.close()
                         except (IOError, OSError):

Il semble que la file d'attente d'écriture fd soit correctement supprimée du concentrateur (quelques lignes plus haut), mais la file d'attente de lecture fd ne l'est pas.

Aucune mise à jour ici? :(

@auvipy des réflexions sur l' analyse de @kleptog ci - dessus à https://github.com/celery/celery/issues/4185#issuecomment -373755206?

désolé, je n'ai pas assez de temps maintenant pour enquêter sur le problème

@kleptog merci pour tant de travail à ce sujet.

Il semble que on_inqueue_close _itself_ appelle hub_remove(fd) - spécifiquement sur L744 , similaire à votre exemple:

https://github.com/celery/celery/blob/b46bea25539cc26a76f0a491b95f4899f5b32c34/celery/concurrency/asynpool.py#L735 -L745

... qui est appelé en destroy_queues() sur L1247 , mais seulement pour queues[0] .

https://github.com/celery/celery/blob/b46bea25539cc26a76f0a491b95f4899f5b32c34/celery/concurrency/asynpool.py#L1245 -L1249

Alors, cette déclaration est-elle vraiment exacte?

Le hack n'est pas la bonne solution, mais il faut vraiment que quelqu'un connaissant mieux le code écrive une bonne solution.

Je ne connais pas assez bien les composants internes du céleri pour savoir pourquoi le code ne fait actuellement que pour queues[0] . Cela vaudra peut-être au moins une demande de tirage avec votre diff.

IIRC c'est en effet l'autre file d'attente qui a été laissée de côté, alors peut-être s'agit-il de faire un inqueue_close sur l'autre file d'attente? Je n'aurai pas l'occasion de le tester avant un moment.

@auvipy Je suis désolé d'être un bourreau, et je peux sympathiser avec la tâche souvent ingrate de faire du travail gratuit sur un projet open source I - Je travaille sur plusieurs pendant mon temps libre, et un autre pour mon travail quotidien.

Savez-vous si vous ou tout autre mainteneur de céleri avez eu l'occasion de jeter un coup d'œil à celui-ci? Nous avons un nombre notable d'utilisateurs de céleri dans ce fil qui ont rencontré ce problème débilitant depuis qu'il a été signalé l'été dernier, et même quelques-uns qui ont participé et ont peut-être identifié la zone de code qui est le coupable. Existe-t-il une sorte d'ETA sur le moment où ce bogue pourrait être examiné?

@ryanpetrello @kleptog J'ai

queues dans ce contexte est un tuple inq, outq, synq retourné par AsynPool.create_process_queues() . Donc queue[0] réfère spécifiquement au inq , et il est logique qu'une fonction appelée on_inqueue_close ne soit appelée que pour cela.

Donc pour le moment, la seule chose qui est passée à hub_remove() est le inq._writer , pas son _reader , et non le outq ou synq écrivains ou lecteurs, et UNIQUEMENT lorsqu'il s'agit d'une clé dans fileno_to_inq dont la valeur est proc .

Je n'ai pas beaucoup de perspicacité à ce sujet. Mais il y a aussi des dictées fileno_to_outq et fileno_to_synq . Devrions-nous vérifier leur contenu avant d'appeler hub_remove() sur ces autres poignées? Ou devrions-nous tout hub_remove inconditionnellement? Peut-être devrions-nous supprimer l'appel hub_remove() de on_inqueue_close() et l'appeler simplement en destroy_queues() comme dans la différence ci-dessus?

@kleptog
Merci pour le travail incroyable. Nous avons eu ce problème pendant un certain temps, et cela a eu pour effet malheureux sur notre système que les tâches pré-extraites étaient abandonnées. J'ai trouvé la même cause fondamentale - les travailleurs et les esclaves accrochés aux FD, mais je n'ai pas pu trouver la condition de blocage. L'application de votre correctif a fonctionné et notre test de résistance a maintenant traité sans effort ~ 50 000 tâches sans accroc.

@AlexHill a dit:

Devrions-nous vérifier leur contenu avant d'appeler hub_remove () sur ces autres poignées? Ou devrions-nous tout supprimer de manière inconditionnelle? Peut-être devrions-nous supprimer l'appel hub_remove () de on_inqueue_close () et l'appeler simplement dans destroy_queues () comme dans le diff ci-dessus?

Très simplement, vous devez supprimer un descripteur de fichier du hub avant de le fermer, car la fermeture ne le supprimera pas du hub pour vous. Donc, votre dernière suggestion est en fait plutôt bonne, car elle est plus "évidemment correcte" et j'en suis un grand fan.

En utilisant strace, je vois ce qui suit:

epoll_ctl (5, EPOLL_CTL_DEL, 16, 0x7ffe54f25410) = -1 ENOENT (aucun fichier ou répertoire de ce type)
epoll_ctl (5, EPOLL_CTL_DEL, 21, 0x7ffe54f25410) = -1 ENOENT (aucun fichier ou répertoire de ce type)
fcntl (26, F_GETFL) = 0x2 (indicateurs O_RDWR)
fcntl (26, F_SETFL, O_RDWR | O_NONBLOCK) = 0
recvfrom (26, 0x7f42ca4409e4, 7, 0, NULL, NULL) = -1 EAGAIN (ressource temporairement indisponible)

Est-ce lié à ce problème?

Merci

peut-être que cela aide à diagnostiquer le problème avec le lien suivant: http://memos.ml/

J'ai le même problème depuis la mise à niveau vers Ubuntu 16

Je suis confronté au même problème sur 4.2.1.

essayez de configurer le céleri avec l'option de délai d'expiration du socket comme suit:

celery.config_from_object ({'broker_transport_options': {'socket_timeout': 30}})

J'ai eu le même problème. Cependant, après la publication du correctif supérieur, le problème est résolu.

J'ai aussi ce problème, y a-t-il une pull request avec le correctif de

Notre problème a été résolu après l'ajout du keepalive et du délai d'expiration du socket TCP (en utilisant redis)

BROKER_TRANSPORT_OPTIONS = {'socket_timeout': 60,
                            'socket_connect_timeout': 120,
                            'socket_keepalive': True,
                            'socket_keepalive_options': {socket.TCP_KEEPIDLE: 60,
                                 socket.TCP_KEEPINTVL: 60,
                                 socket.TCP_KEEPCNT: 3}}

@ linar-jether: quelle est cette socket dans socket.TCP_KEEPIDLE ?

@ nishith107 import socket

La modification de la configuration du courtier redis n'a presque certainement aucun rapport avec le problème de ce fil, qui consiste à rester bloqué sur une lecture bloquante sur le canal de communication maître-enfant.

Bien, il est possible que bricoler avec les paramètres du courtier puisse introduire suffisamment de gigue pour éviter le problème sous-jacent, mais le blocage IPC maître-enfant est un bogue distinct.

@auvipy cela a été signalé il y a un an aujourd'hui. Moi-même et d'autres ont laissé des informations détaillées et une analyse de l'endroit où le code semble se bloquer, y compris la sortie strace où nous l'avons réduite à quelques lignes de code. Quelqu'un a même publié un lien vers un patch (
https://github.com/celery/celery/issues/4185#issuecomment-373755206) qu'il semble que les gens utilisent et déploient (en production?) comme solution de contournement. De nombreuses personnes impliquées dans ce fil de commentaires semblent toujours rencontrer cela régulièrement dans un grand nombre de versions de céleri et de configurations de courtier.

J'apprécie que ce soit un projet open source, et toutes les personnes impliquées sont des bénévoles, donc je n'ai pas l'intention de harceler sans fin. Savez-vous si _quelqu'un_ a fait un effort pour enquêter ou diagnostiquer ce problème critique?

Il nous serait plus facile de résoudre les problèmes si quelqu'un d'entre nous y faisait face en production. Étant donné que certains d'entre vous ont utilisé une solution de contournement pour se débarrasser de ce problème, pourquoi ne pas envoyer un pr avec des cas de test appropriés? le céleri a des problèmes qui ont aussi entre 5 et 6 ans :(

croyez-moi, il est vraiment difficile de faire face à tous les problèmes qui surgissent. veuillez prendre en charge l'envoi d'un correctif car certains d'entre vous y font face en production, il vous sera donc plus facile de tester les modifications proposées dans l'environnement réel.

@ linar-jether Merci, ce correctif a fonctionné pour moi :)

pourriez-vous vérifier ce PR https://github.com/celery/celery/pull/4997 ? si cela résout votre problème, nous pourrions fusionner cela.

Je vais essayer cette branche. Il me faudra un peu de temps pour savoir si c'est réglé car je n'ai pas trouvé de moyen de le forcer à se produire.

Des nouvelles sur le test de ce correctif de bogue?

Je n'ai pas eu le problème depuis plus de 2 semaines maintenant. Je ne suis pas encore à 100% mais je pense que le PR suggéré a résolu le problème pour moi.

Nous venons de déployer la branche 4.2 avec ce patch appliqué en tant que roue personnalisée sur 3 machines clientes ... elle est en production maintenant. Ce bug bloquait les travailleurs une fois par semaine depuis des mois maintenant. Il est déployé. Nous verrons si cela empêche le problème.

Je fusionne donc le pr https://github.com/celery/celery/pull/4997, veuillez nous tenir au courant de vos problèmes de production ici, et si quelqu'un a des suggestions de tests unitaires / d'intégration, veuillez procéder à un PR.

@ryanpetrello se demandant, avez-vous déployé ce patch et testé?

@AvnerCohen à cause de cela et de quelques autres problèmes, nous avons fini par résoudre notre problème sans utiliser de céleri (nous avons pu accomplir ce dont nous avions besoin avec du vieux kombu). On dirait que d'autres ont eu de la chance avec ce patch, cependant!

Jusqu'à présent, nous n'avons plus connaissance de problèmes depuis notre déploiement en production. Je pense que cela a résolu le problème.

super merci!

@auvipy Quand faut-il s'attendre à une version officielle incluant ce patch?

mon objectif est avant Noël

Merci @kleptog pour un excellent débogage!

J'ai essayé tous les correctifs suggérés, jusqu'à exécuter Celery et ses amis depuis le maître, mais dès qu'un travailleur rencontre une ConnectionResetError, il refuse de traiter plus de tâches. Le redémarrage de RabbitMQ oblige les travailleurs bloqués à traiter à nouveau les tâches. Naturellement, ce n'est pas une bonne solution mais cela m'indique dans une direction à déboguer plus loin.

Nous utilisons ce correctif mais rencontrons toujours ce problème de temps en temps - le travailleur termine avec succès son travail et le processus principal est verrouillé en attente de résultats

J'utilise le céleri == 4.2.1; python == 3.6.7 ubuntu 16.04. et Redis comme backend
J'ai utilisé le superviseur pour démarrer le céleri avec la commande:
céleri -A project worker -l info --autoscale = 10,2

Il restait parfois bloqué.

@mikolaje ce qui fonctionne pour moi est de définir le tcp-keepalive dans redis.conf pour résoudre le blocage.

Quelqu'un a-t-il trouvé un bon moyen de surveiller la situation des travailleurs bloqués?

Nous voyons toujours ce comportement avec le code de master, je ne peux pas dire si cela est amélioré ou non, mais l'impression initiale est qu'au moins pour notre cas d'utilisation, nous sommes toujours bloqués.

La question est de savoir comment surveiller les travailleurs individuels? pour le moment, tout ce que nous pouvons faire est de surveiller le niveau de la file d'attente (messages présents et non récupérés), mais j'aimerais suivre cela beaucoup plus tôt en voyant quand un travailleur ne prend plus de travail.
Ce que j'ai remarqué, c'est que la commande "status" du céleri retournera quand même OK, aussi une fleur montrant que tout va bien.

J'ai essayé tous les correctifs suggérés, jusqu'à exécuter Celery et ses amis depuis le maître, mais dès qu'un travailleur rencontre une ConnectionResetError, il refuse de traiter plus de tâches. Le redémarrage de RabbitMQ oblige les travailleurs bloqués à traiter à nouveau les tâches. Naturellement, ce n'est pas une bonne solution mais cela m'indique dans une direction à déboguer plus loin.

@hedleyroos , vous pouvez aussi avoir ce problème # 4895? Nous exécutons Celery v4.1.x pour éviter ce problème.

@kingb oui j'ai ce problème. Je voulais éviter de passer à la version 4.1 et je suis passé à Redis en tant que courtier, mais j'ai toujours rencontré le même problème. Heureusement, Redis a une option tcp-keepalive que j'ai définie sur 60 secondes. Cela signifie que Redis mettra fin à ce qu'il considère comme des connexions obsolètes, ce qui conduira à son tour le travailleur à sortir de l'impasse et à recommencer à consommer des tâches.

J'ai gelé pour le céleri == 4.2.1, billard == 3.5.0.5, kombu == 4.2.2.post1, amqp == 2.4.0, Python 3.6.8, RabbitMQ, MongoDB v 4

@tonal J'utilise les mêmes versions que vous et parfois j'obtiens cette erreur. Ne pas réessayer des tâches ou quoi que ce soit du genre. Les tâches restent bloquées au statut RECIEVED.

cela ne se produit pas tout le temps, mais cela se produit toutes les quelques heures / tous les quelques jours et a purgé le message et redémarré le service.

4,3? 4.4rc2?

Non, celery==4.2.1, billiard==3.5.0.5, kombu==4.2.2.post1, amqp==2.4.0,

@auvipy le voit encore mourir ~ 1 heure avec 4.3. Si cela aide, il arrête de réussir / échouer les tâches et continue de les recevoir, puis échoue silencieusement, mais le processus de céleri reste en cours.

/code # celery status
celery@celery-worker: OK

1 node online.

Quelqu'un peut-il m'assurer qu'il est vraiment résolu en 4.3? Je dois prendre la décision de mise à niveau en fonction de cela.

Salut tout le monde,

Ce n'est pas corrigé dans 4.3. J'utilise cette version dans un projet et mes ouvriers restent bloqués (après avoir affiché des messages "pulsation manquante").

Quelqu'un at-il contourné ce problème avec 4.3? J'ai remarqué que le code est très différent de la solution de contournement / correctif ci-dessus, donc je suppose qu'il y a eu une tentative de résoudre le problème, mais cela ne semble pas résolu.

Merci!

Je peux confirmer que j'ai le même problème dans Celery 4.3 (Python 3.7.3).

pourriez-vous essayer le céleri == 4.4.0rc3?

@auvipy a essayé cela aussi, n'a pas aidé. juste pour info j'utilise gevent

Y a-t-il une chance que ce problème soit rouvert? On dirait que les gens le rencontrent encore sur les versions les plus récentes de céleri.

Je suis confronté au même problème. Python2.7, céleri4.1.0. J'utilise le courtier SQS. Le céleri traite les tâches pendant un certain temps et il se bloque brusquement après un certain temps.

Je suis confronté au même problème. Python2.7, céleri4.1.0. J'utilise le courtier SQS. Le céleri traite les tâches pendant un certain temps et il se bloque brusquement après un certain temps.

vous devriez essayer les nouvelles versions.

Nous n'avons pas été en mesure de reproduire ce problème, donc quelqu'un devra le déboguer pour nous et s'il le peut, fournir un correctif pour ce problème.

Nous voyons cela quotidiennement dans la production. Nous n'avons jamais réussi à recréer sur un environnement de développement ou de débogage activé.
Avez-vous des suggestions sur le type d'informations que nous pouvons fournir qui vous aideront?

Je suggère d'utiliser quelque chose comme python-hunter et / ou python-manhole .
Le problème est probablement dans le processus maître.
Vous pouvez également utiliser strace pour voir dans quel socket / tube est en cours de lecture ou d'écriture et essayez d'en déduire.
Je peux venir à vos bureaux la semaine prochaine si vous le souhaitez. Discutons de cela au téléphone.

Merci, je vais essayer ceci, juste à titre d'exemple général, voici à quoi ressemblent les processus bloqués du point de vue PS.

username 14182     1  1 03:12 ?        00:02:28 [celeryd:  workername<strong i="6">@hostname</strong>:MainProcess] -active- (moshe)
username 15487 14182  0 03:15 ?        00:00:12 [celeryd:  workername<strong i="7">@hostname</strong>:ForkPoolWorker-5]
username 15766 14182  0 03:18 ?        00:00:12 [celeryd:  workername<strong i="8">@hostname</strong>:ForkPoolWorker-14]
username 16666 14182  0 03:52 ?        00:00:10 [celeryd:  workername<strong i="9">@hostname</strong>:ForkPoolWorker-43]
username 20253 14182 65 06:45 ?        00:00:24 [celeryd:  workername<strong i="10">@hostname</strong>:ForkPoolWorker-110]

La concurrence est définie sur 4, avec "max-tasks-per-child" défini, de sorte que nous redémarrons tous les X messages.
Comme on peut le voir clairement, alors que 20253 est toujours actif (et que le compteur de travailleurs continue d'augmenter) 15487 mort prématurément et est coincé avec:

$ strace -p 15487
strace: Process 15487 attached
futex(0x2b36980, FUTEX_WAIT_PRIVATE, 0, NULL

Le maître est toujours actif et les messages sont toujours envoyés au travailleur qui est toujours actif.

Laisser ça ici au cas où ça aiderait quelqu'un d'autre. Il n'est pas clair si j'avais exactement le même problème, mais mes employés étaient également suspendus après avoir traité plusieurs centaines de milliers de tâches. Je lançais Celery en tant que service en utilisant:

celery multi start w1 w2 -A app.celery --time-limit=21600 --concurrency=4

Après avoir résolu sans succès les travailleurs bloqués avec cela, j'ai changé la commande en un simple:

celery worker -A app.celery --time-limit=21600 --concurrency=4

Et cela l'a corrigé pour moi! Lorsque j'ai exécuté Celery de cette façon manuellement, je pouvais voir les tâches arriver et s'exécuter et les tâches se coincer occasionnellement (puis le travailleur redémarré correctement). Je pense que la première méthode n'était pas de redémarrer correctement les travailleurs après l'échec d'une tâche. L'autre changement que j'ai apporté était de définir une limite de temps inférieure pour les tâches qui seraient exécutées des centaines / milliers de fois, donc lorsqu'un travailleur se bloquait inévitablement, le processus de travail principal serait en mesure de le tuer et d'en redémarrer un nouveau assez rapidement. avant que tous ne se retrouvent coincés.

Je ne connais pas suffisamment les composants internes du céleri pour déboguer efficacement ce problème, mais si quelqu'un peut donner des conseils ou des endroits à consulter, je peux l'examiner de notre côté lorsque le problème se produit.

Je ne peux pas recréer ce problème à la demande, mais cela se produit au fil du temps sur l'un de nos sites. D'après ce que j'ai pu localiser, le problème se produit lorsqu'il y a une coupure de réseau entre le travailleur et le serveur RabbitMQ alors que les travailleurs sont sous charge. L'échec empêche essentiellement le processus principal d'accepter de nouveaux travaux.

Si le système n'est pas sous charge, tout continue de fonctionner correctement. Il doit s'agir d'une combinaison de charge sur le travailleur et d'une interruption de réseau en même temps. Cela a confirmé deux façons, l'une d'un système de transfert secondaire sans charge à ce moment-là voit les échecs de mélange mais continue de fonctionner correctement par la suite. Deuxièmement, nous exécutons plusieurs travailleurs par boîte. Seuls les nœuds de calcul qui exécutent des travaux échouent. Si la charge est légère et que tous les processus principaux ne traitent pas des travaux, seuls certains des processus principaux échoueront.

celery worker -n XXX-%I.${CELERY_HOST_NAME} \
    -A XXX.app --pidfile=/run/celery/XXX-%I.pid \
    --autoscale=10,3 \
    --max-tasks-per-child 1\
            -Q YYYl \
    --loglevel=INFO --logfile=/var/log/XXX/XXX-%I${CELERY_PROCESS_INDEX_SEPERATOR}.log --soft-time-limit=7200 --time-limit

{"event": "mingle: sync with 54 nodes", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: processing reply from celery@XXXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "mingle: processing reply from celery@XXX", "logger": "celery.worker.consumer.mingle", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "mingle: sync complete", "logger": "celery.worker.consumer.mingle", "level": "info", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Gossip", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 2", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:24", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Tasks", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:25", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting Control", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "using channel_id: 3", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Channel open", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:31:26", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "^-- substep ok", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Consumer: Starting event loop", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Autoscaler...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "| Worker: Hub.register Pool...", "logger": "celery.bootsteps", "level": "debug", "timestamp": "2019-11-17 22:31:27", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "basic.qos: prefetch_count->3", "logger": "kombu.common", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', {'lang': 'py', 'task': 'XXX', 'id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ec8092ac-9b3c-4081-b11e-2d93aeac3bf1', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"xxx}\", 'origin': 'gen49433<strong i="17">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', {'lang': 'py', 'task': 'XXX', 'id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4e3acf3a-c0ec-4be3-8add-3e76d047ec09', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{'xxx}\", 'origin': 'gen49438<strong i="18">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Received task: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020]  ", "logger": "celery.worker.strategy", "level": "info", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "TaskPool: Apply <function _fast_trace_task at 0x7fb3d5399bf8> (args:('XXX', 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', {'lang': 'py', 'task': 'XXX', 'id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd5a6ada9-b9f5-435d-84cb-4dd6d7e21020', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': \"{xxx}\", 'origin': 'gen49438<strong i="19">@XXX</strong>', 'reply_to':... kwargs:{})", "logger": "celery.pool", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "celery<strong i="20">@XXX</strong> left", "logger": "celery.worker.consumer.gossip", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[4e3acf3a-c0ec-4be3-8add-3e76d047ec09] pid:111546", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[ec8092ac-9b3c-4081-b11e-2d93aeac3bf1] pid:111544", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task accepted: XXX[d5a6ada9-b9f5-435d-84cb-4dd6d7e21020] pid:111538", "logger": "celery.worker.request", "level": "debug", "timestamp": "2019-11-17 22:31:28", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "missed heartbeat from celery@XXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
...
{"event": "missed heartbeat from celery@XXXX", "logger": "celery.worker.consumer.gossip", "level": "info", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec22d68>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c5f8>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.')", "exc_info": ["<class 'billiard.exceptions.WorkerLostError'>", "WorkerLostError('Worker exited prematurely: exitcode 155.')", "<billiard.einfo.Traceback object at 0x7fb3cec1c860>"], "logger": "celery.worker.request", "level": "error", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}
{"event": "heartbeat_tick : Prev sent/recv: None/None, now - 31/490, monotonic - 4090172.991839465, last_heartbeat_sent - 4090172.991832662, heartbeat int. - 60 for connection ebf9e86b69a347aaae81a92712b65eb9", "logger": "amqp", "level": "debug", "timestamp": "2019-11-17 22:34:17", "code_version": "v1.0.0-168-ga02e6b6-dirty"}

 <built-in method read of module object at remote 0x7fb3e61f33b8>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 64, in __read__
    chunk = read(fd, size)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 241, in _recv_message
    fd, bufv[Hr:] if readcanbuf else bufv, 4 - Hr,
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 298, in on_result_readable
    next(it)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
    worker.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
    ret = self.run(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
    return self(*args, **options)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
    return self.execute(command, argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
    sys.exit(main())

#0  0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x0000000000542a85 in _Py_read (fd=fd@entry=56, buf=0x7fb3cecbad70, count=count@entry=4) at Python/fileutils.c:1407
#2  0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=56) at ./Modules/posixmodule.c:8043
#3  os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x7fb3ceb7f3b8, nargs=<optimized out>)
    at ./Modules/clinic/posixmodule.c.h:3625
#4  0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x7fb3ceb7f3b8,
    self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5  _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>,
    args=args@entry=0x7fb3ceb7f3b8, nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6  0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8  0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3ceb7f218, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 64, in __read__ (fd=56, buf=<_io.BytesIO at remote 0x7fb3ceca0f10>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>)) at Python/ceval.c:547
#9  _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf537420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x347a2b8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf3074f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='__read__',
    qualname='__read__') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#11 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#13 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cebfb138) at Objects/genobject.c:221
#14 gen_iternext (gen=0x7fb3cebfb138) at Objects/genobject.c:542
#15 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#16 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3481f10,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#17 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x3481f10,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#18 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#20 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3481d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 298, in on_result_readable (fileno=56, it=<generator at remote 0x7fb3cebfb138>))
    at Python/ceval.c:547
#21 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2ed300>,
    globals=globals@entry={'__name__': 'celery.concurrency.asynpool', '__doc__': "Version of multiprocessing.Pool using Async I/O.\n\n.. note::\n\n    This module will be moved soon, so don't use it directly.\n\nThis is a non-blocking version of :class:`multiprocessing.Pool`.\n\nThis code deals with three major challenges:\n\n#. Starting up child processes and keeping them running.\n#. Sending jobs to the processes and receiving results back.\n#. Safely shutting down this system.\n", '__package__': 'celery.concurrency', '__loader__': <SourceFileLoader(name='celery.concurrency.asynpool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py') at remote 0x7fb3cf2db828>, '__spec__': <ModuleSpec(name='celery.concurrency.asynpool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py', loader_state=None, submodule_search_locations=N...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cec173a8, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=(<cell at remote 0x7fb3ceb5f9a8>, <cell at remote 0x7fb3ceb5f9d8>, <cell at remote 0x7fb3ceb5fa08>, <cell at remote 0x7fb3ceb5fa38>, <cell at remote 0x7fb3ceb5fb58>), name='on_result_readable',
    qualname='ResultHandler._make_process_result.<locals>.on_result_readable') at Python/ceval.c:3930
#22 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec260d0>, args=args@entry=0x7fb3cec173a8,
    nargs=1, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#23 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec260d0>, args=args@entry=(56,),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#24 0x0000000000422479 in do_call_core (kwdict=0x0, callargs=(56,), func=<function at remote 0x7fb3cec260d0>) at Python/ceval.c:4645
#25 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#26 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#27 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#28 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#29 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#30 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#31 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#32 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#33 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#34 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
    globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
    closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#35 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
    nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#36 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
    args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.Celery---Type <return> to continue, or q <return> to quit---
Config', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#37 0x0000000000422479 in do_call_core (kwdict=0x0,
    callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#38 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#39 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#40 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#41 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#42 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#43 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#44 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#45 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#46 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#47 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#48 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#49 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#64 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
    globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n    See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n    Number of child processes processing the queue.  The default\n    is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n    Pool implementation:\n\n    prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n    Set custom hostname (e.g., 'w1@%%h').  Expands: %%h (hostname),\n    %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n    Also run the `celery beat` periodic task scheduler.  Please note that\n    there must only be one instance of this service.\n\n    .. note::\n\n        ``-B`` is meant to be used for development purposes. For production\n        environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n    L...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
    kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
    closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#65 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
    args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#66 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#67 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'b---Type <return> to continue, or q <return> to quit---
gl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/call.c:904
#68 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245

#69 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#70 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191

#71 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#72 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
    globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
    kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#73 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,

    args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376

#74 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98

#75 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
    obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904

#76 0x0000000000494665 in slot_tp_call (
    self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_
worker.ta...(truncated), args=(),
    kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': N---Type <return> to continue, or q <return> to quit---
one, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/typeobject.c:6379

#77 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245

#78 0x0000000000422479 in do_call_core (

    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
    func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#80 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#81 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#82 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#83 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#84 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139

#85 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#86 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
    qualname='CeleryCommand.execute') at Python/ceval.c:3930
#87 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#88 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#89 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#90 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#91 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
    qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#92 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#93 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#94 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#95 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'pr---Type <return> to continue, or q <return> to quit---
eload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#96 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#97 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#98 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#99 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#100 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#101 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
    qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#102 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#103 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#104 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#105 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#106 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#107 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#108 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#109 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
    at Objects/call.c:283
#110 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#111 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#112 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#113 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#114 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
    kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#115 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
    args=0x0,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#116 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#117 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
    at Python/pythonrun.c:1035
#118 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#119 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
    at Python/pythonrun.c:429
#120 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
    filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
    fp=0x26f05c0) at Modules/main.c:427
#121 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#122 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
---Type <return> to continue, or q <return> to quit---
#123 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#124 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#125 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#126 0x0000000000429df4 in _start ()

 Traceback (most recent call first):
  <built-in method read of module object at remote 0x7fb3e61f33b8>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 422, in _recv
    chunk = read(handle, remaining)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 456, in _recv_bytes
    buf = self._recv(4)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py", line 243, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/queues.py", line 355, in get_payload
    return self._reader.recv_bytes()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 473, in receive
    ready, req = _receive(1.0)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 351, in workloop
    req = wait_for_job()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 292, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 79, in _launch
    code = process_obj._bootstrap()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1158, in _create_worker_process
    w.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 445, in _create_worker_process
    return super(AsynPool, self)._create_worker_process(i)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1328, in _repopulate_pool
    self._create_worker_process(self._avail_index())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1343, in _maintain_pool
    self._repopulate_pool(joined)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py", line 1351, in maintain_pool
    self._maintain_pool()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 130, in _reschedules
    return fun(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
    entry()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  <built-in method next of module object at remote 0x7fb3e627ac28>
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
    worker.start()
---Type <return> to continue, or q <return> to quit---
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
    ret = self.run(*args, **kwargs)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
    return self(*args, **options)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
    return self.execute(command, argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery", line 8, in <module>
    sys.exit(main())

#0  0x00007fb3e5c2e6e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x0000000000542a85 in _Py_read (fd=fd@entry=12, buf=0x7fb3cf30b410, count=count@entry=4) at Python/fileutils.c:1407
#2  0x000000000054b1a9 in os_read_impl (module=0x0, length=4, fd=12) at ./Modules/posixmodule.c:8043
#3  os_read (module=module@entry=<module at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0, nargs=<optimized out>)
    at ./Modules/clinic/posixmodule.c.h:3625
#4  0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x3873df0,
    self=<module at remote 0x7fb3e61f33b8>, method=0x8c7ec0 <posix_methods+2848>) at Objects/call.c:651
#5  _PyCFunction_FastCallKeywords (func=<built-in method read of module object at remote 0x7fb3e61f33b8>, args=args@entry=0x3873df0,
    nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#6  0x00000000004280bd in call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#7  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#8  0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3873c38, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 422, in _recv (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, size=4, read=<built-in method read of module object at remote 0x7fb3e61f33b8>, buf=<_io.BytesIO at remote 0x7fb3cecccba0>, handle=12, remaining=4)) at Python/ceval.c:547
#9  _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53ff60>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec4f1e8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf54e488, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='_recv',
    qualname='Connection._recv') at Python/ceval.c:3930
#10 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#11 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#13 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec4f048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 456, in _recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxsize=None)) at Python/ceval.c:547
#14 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5420c0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec87a78, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf543108, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='_recv_bytes', qualname='Connection._recv_bytes') at Python/ceval.c:3930
#15 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#16 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#17 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#18 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec878e0, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/connection.py, line 243, in recv_bytes (self=<Connection(_handle=12, _readable=True, _writable=False) at remote 0x7fb3ced17fd0>, maxlength=None)) at Python/ceval.c:547
#19 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf53f300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x7fb3cec9e9e8, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf538bf8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='recv_bytes', qualname='_ConnectionBase.recv_bytes') at Python/ceval.c:3930
#20 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#21 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#22 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#23 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#24 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#25 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#26 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#27 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cec87728, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 445, in _recv (timeout=<float at remote 0x7fb3d01417c8>, loads=<function at remote 0x7fb3cf5d72f0>)) at Python/ceval.c:547
#28 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x3346fa0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3ced2c878, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3cec534f8>,), name='_recv', qualname='Worker._make_recv_method.<locals>._recv')
    at Python/ceval.c:3930
#29 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#30 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#31 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#32 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3346df8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 473, in receive (debug=<function at remote 0x7fb3d970b0d0>)) at Python/ceval.c:547
#33 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf598a50>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=0, kwnames=0x0, kwargs=kwargs@entry=0x386d2c0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cec222c8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3ceb62708>, <cell at remote 0x7fb3ceb62fd8>), name='receive',
    qualname='Worker._make_protected_receive.<locals>.receive') at Python/ceval.c:3930
#34 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#35 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#37 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x386d058, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py, line 351, in workloop (debug=<function at remote 0x7fb3d970b0d0>, now=<built-in method monotonic of module object at remote 0x7fb3de980d68>, pid=111793, put=<method at remote 0x7fb3cecf55c8>, inqW_fd=13, synqW_fd=None, maxtasks=1, max_memory_per_child=0, prepare_result=<method at remote 0x7fb3cec86548>, wait_for_job=<function at remote 0x7fb3cec7b268>, wait_for_syn=<function at remote 0x7fb3cec3ff28>, completed=0)) at Python/ceval.c:547
#38 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf5985d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x7fb3cf599060, kwargs=kwargs@entry=0x3490068, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf536f90, defcount=defcount@entry=3, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='workloop',
    qualname='Worker.workloop') at Python/ceval.c:3930
#39 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#40 0x0000000000428c25 in call_function (kwnames=('pid',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#42 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x348feb8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/si---Type <return> to continue, or q <return> to quit---
te-packages/billiard/pool.py, line 292, in __call__ (self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898...(truncated)) at Python/ceval.c:547
#43 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf598270>,
    globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea23e20, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0,
    name='__call__', qualname='Worker.__call__') at Python/ceval.c:3930
#44 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf53db70>,
    args=args@entry=0x7ffdbea23e20, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:376
#45 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea23e20,
    callable=<function at remote 0x7fb3cf53db70>) at Objects/call.c:98
#46 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf53db70>,
    obj=obj@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
    args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:904
#47 0x0000000000494665 in slot_tp_call (
    self=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated), args=(), kwds={})
    at Objects/typeobject.c:6379
#48 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated),
    args=args@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:245
#49 0x0000000000422479 in do_call_core (kwdict={}, callargs=(),
    func=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _us...(truncated)) at Python/ceval.c:4645
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#51 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#52 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#53 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#55 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#56 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#57 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#58 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#59 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#60 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#61 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#63 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf1f5b70, args=<optimized out>, args@entry=0x7ffdbea24610,
    nargs=nargs@entry=2,
    globals=globals@entry={'__name__': 'billiard.popen_fork', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.popen_fork', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py') at remote 0x7fb3cf201748>, '__spec__': <ModuleSpec(name='billiard.popen_fork', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/popen_fork.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf201780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/popen_fork.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated))
    at Objects/call.c:283
#64 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf205c80>,
    args=args@entry=0x7ffdbea24610, nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#65 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffdbea24610,
    callable=<function at remote 0x7fb3cf205c80>) at Objects/call.c:98
---Type <return> to continue, or q <return> to quit---
#66 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf205c80>,
    obj=obj@entry=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
    args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:904
#67 0x000000000048f059 in slot_tp_init (self=<Popen(returncode=None, pid=0) at remote 0x7fb3cec22da0>,
    args=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated), kwds=0x0)
    at Objects/typeobject.c:6613
#68 0x000000000048a332 in type_call (type=<optimized out>, type@entry=0x336fdb8,
    args=args@entry=(<ForkProcess(_identity=(3733,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _popen=None, _target=<Worker(initializer=<function at remote 0x7fb3cf5752f0>, initargs=(<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, ...(truncated),
    kwds=kwds@entry=0x0) at Objects/typeobject.c:949
#69 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x336fdb8>, stack=<optimized out>,
    nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#70 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#71 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#72 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#73 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#74 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#75 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#76 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#77 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#78 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#79 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#80 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#81 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#82 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#83 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#84 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3cf1f0570, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/concurrency/asynpool.py, line 445, in _create_worker_process (self=<AsynPool(sched_strategy=4, synack=False, _queues={(<_SimpleQueue(_reader=<Connection(_handle=31, _readable=True, _writable=False) at remote 0x7fb3cecdecf8>, _writer=<Connection(_handle=32, _readable=False, _writable=True) at remote 0x7fb3cecde2b0>, _poll=<method at remote 0x7fb3cf23cec8>, _rlock=None, _wlock=None) at remote 0x7fb3ced47748>, <_SimpleQueue(_reader=<Connection(_handle=33, _readable=True, _writable=False) at remote 0x7fb3cecde2e8>, _writer=<Connection(_handle=34, _readable=False, _writable=True) at remote 0x7fb3cecde828>, _poll=<method at remote 0x7fb3ceb4a188>, _rlock=None, _wlock=None) at remote 0x7fb3cecde160>, None): <ForkProcess(_identity=(3731,), _config={'authkey': <AuthenticationString at remote 0x7fb3ddedfba8>, 'semprefix': '/mp', 'daemon': True}, _parent_pid=94981, _pope...(truncated)) at Python/ceval.c:547
#85 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf2ed8a0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3ced34988, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3cf2e5eb8>,), name='_create_worker_process',
    qualname='AsynPool._create_worker_process') at Python/ceval.c:3930
#86 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#87 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#88 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#89 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#90 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#91 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#92 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#93 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#94 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#95 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#96 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#97 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf5299c0, args=<optimized out>, args@entry=0x7ffdbea25400,
    nargs=nargs@entry=1,
    globals=globals@entry={'__name__': 'billiard.pool', '__doc__': None, '__package__': 'billiard', '__loader__': <SourceFileLoader(name='billiard.pool', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py') at remote 0x7fb3cf5916d8>, '__spec__': <ModuleSpec(name='billiard.pool', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/__pycache__/pool.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf591710>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/billiard/pool.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVE...(truncated))
    at Objects/call.c:283
#98 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2d61e0>,
    args=args@entry=0x7ffdbea25400, nargs=nargs@entry=1, kwargs=kwargs@entry={}) at Objects/call.c:322
---Type <return> to continue, or q <return> to quit---
#99 0x000000000043e749 in _PyObject_FastCallDict (kwargs={}, nargs=1, args=0x7ffdbea25400,
    callable=<function at remote 0x7fb3cf2d61e0>) at Objects/call.c:98
#100 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3cf2d61e0>, obj=<optimized out>, args=(), kwargs={})
    at Objects/call.c:904
#101 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3cecd5048>, args=args@entry=(),
    kwargs=kwargs@entry={}) at Objects/call.c:245
#102 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<method at remote 0x7fb3cecd5048>) at Python/ceval.c:4645
#103 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#104 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347e4f8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py, line 130, in _reschedules (args=(), kwargs={}, last=None, now=<float at remote 0x7fb3ceb52108>, lsince=<float at remote 0x7fb3d01419f0>)) at Python/ceval.c:547
#105 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2f05d0>,
    globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3e6246060, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=(<cell at remote 0x7fb3cec65be8>, <cell at remote 0x7fb3cec30d08>, <cell at remote 0x7fb3cec30d38>, <cell at remote 0x7fb3cec30d98>, <cell at remote 0x7fb3cec30e28>), name='maintain_pool', qualname='Pool.maintain_pool') at Python/ceval.c:3930
#106 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cec6a620>, args=args@entry=0x7fb3e6246060,
    nargs=0, kwargs=kwargs@entry={}) at Objects/call.c:376
#107 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cec6a620>, args=args@entry=(),
    kwargs=kwargs@entry={}) at Objects/call.c:226
#108 0x0000000000422479 in do_call_core (kwdict={}, callargs=(), func=<function at remote 0x7fb3cec6a620>) at Python/ceval.c:4645
#109 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#110 0x00000000004200a8 in function_code_fastcall (co=co@entry=0x7fb3cf2e69c0, args=<optimized out>, args@entry=0x7ffdbea259c0,
    nargs=nargs@entry=1,
    globals=globals@entry={'__name__': 'kombu.asynchronous.timer', '__doc__': 'Timer scheduling Python callbacks.', '__package__': 'kombu.asynchronous', '__loader__': <SourceFileLoader(name='kombu.asynchronous.timer', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py') at remote 0x7fb3cf2e3c50>, '__spec__': <ModuleSpec(name='kombu.asynchronous.timer', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/timer.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/__pycache__/timer.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2e3c88>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/as...(truncated))
    at Objects/call.c:283
#111 0x000000000043b176 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3cf2ee730>,
    args=args@entry=0x7ffdbea259c0, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/call.c:322
#112 0x000000000043e749 in _PyObject_FastCallDict (kwargs=0x0, nargs=1, args=0x7ffdbea259c0,
    callable=<function at remote 0x7fb3cf2ee730>) at Objects/call.c:98
#113 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3cf2ee730>,
    obj=obj@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwargs=kwargs@entry=0x0) at Objects/call.c:904
#114 0x0000000000494665 in slot_tp_call (self=self@entry=<Entry at remote 0x7fb3cebb3768>, args=args@entry=(), kwds=kwds@entry=0x0)
    at Objects/typeobject.c:6379
#115 0x000000000043b861 in _PyObject_FastCallKeywords (callable=callable@entry=<Entry at remote 0x7fb3cebb3768>,
    stack=<optimized out>, nargs=nargs@entry=0, kwnames=kwnames@entry=0x0) at Objects/call.c:199
#116 0x0000000000424098 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4619
#117 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#118 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x3480618, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/kombu/asynchronous/hub.py, line 145, in fire_timers (self=<Hub(timer=<Timer(max_interval=<float at remote 0x7fb3d0141a08>, on_error=None, _queue=[<scheduled at remote 0x7fb3cebf0d18>, <scheduled at remote 0x7fb3cebf0a48>, <scheduled at remote 0x7fb3cecea3b8>, <scheduled at remote 0x7fb3ceceabd8>, <scheduled at remote 0x7fb3ceb60368>, <scheduled at remote 0x7fb3cec05d18>, <scheduled at remote 0x7fb3ceb606d8>, <scheduled at remote 0x7fb3cec8f7c8>, <scheduled at remote 0x7fb3cec8f688>]) at remote 0x7fb3cf2b04e0>, readers={71: (<method at remote 0x7fb3cec80b48>, (<Connection(_connection_id='ed843206bfeb424292688d00071ac41e', authentication=(<AMQPLAIN(username='aep', password='F8P7uYpFbxyWQSJF4psFc52P6') at remote 0x7fb3ceb98a20>,), client_properties={'product': 'py-amqp', 'product_version': '2.5.1', 'capabilities': {'consumer_cancel_notify': True, 'connection.blocked': True, ...(truncated)) at Python/ceval.c:547
#119 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3cf550780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=1, kwnames=0x7fb3cf2dfd80, kwargs=kwargs@entry=0x347c960, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3cf549ab0, defcount=defcount@entry=4, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='fire_timers', qualname='Hub.fire_timers') at Python/ceval.c:3930
#120 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#121 0x0000000000428c25 in call_function (kwnames=('propagate',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#122 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#123 0x00000000005c043d in gen_send_ex (closing=0, exc=0, arg=0x0, gen=0x7fb3cf275de0) at Objects/genobject.c:221
#124 gen_iternext (gen=0x7fb3cf275de0) at Objects/genobject.c:542
#125 0x00000000004e703b in builtin_next (self=self@entry=<module at remote 0x7fb3e627ac28>, args=args@entry=0x347abd0,
    nargs=<optimized out>) at Python/bltinmodule.c:1426
#126 0x000000000043bddd in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=0, args=0x347abd0,
    self=<module at remote 0x7fb3e627ac28>, method=0x8ba020 <builtin_methods+992>) at Objects/call.c:651
#127 _PyCFunction_FastCallKeywords (func=<built-in method next of module object at remote 0x7fb3e627ac28>,
    args=args@entry=0x347abd0, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:730
#128 0x00000000004280bd in call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at Python/ceval.c:4568
#129 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#130 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x347a9d8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py, line 91, in asynloop (obj=<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.l...(truncated)) at Python/ceval.c:547
#131 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3cf2c6d20>,
    globals=globals@entry={'__name__': 'celery.worker.loops', '__doc__': 'The consumers highly-optimized inner loop.', '__package__': 'celery.worker', '__loader__': <SourceFileLoader(name='celery.worker.loops', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py') at remote 0x7fb3cf2cd3c8>, '__spec__': <ModuleSpec(name='celery.worker.loops', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-em---Type <return> to continue, or q <return> to quit---
IVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/__pycache__/loops.cpython-37.pyc', _initializing=False) at remote 0x7fb3cf2cd400>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/worker/loops.py', '__cached_...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7fb3cebfba38, argcount=argcount@entry=9, kwnames=kwnames@entry=0x0,
    kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fb3cf2cd5d8, defcount=1, kwdefs=0x0,
    closure=0x0, name='asynloop', qualname='asynloop') at Python/ceval.c:3930
#132 0x000000000043b027 in _PyFunction_FastCallDict (func=<function at remote 0x7fb3cf2b1c80>, args=args@entry=0x7fb3cebfba38,
    nargs=9, kwargs=kwargs@entry=0x0) at Objects/call.c:376
#133 0x000000000043c713 in PyObject_Call (callable=callable@entry=<function at remote 0x7fb3cf2b1c80>,
    args=args@entry=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    kwargs=kwargs@entry=0x0) at Objects/call.c:226
#134 0x0000000000422479 in do_call_core (kwdict=0x0,
    callargs=(<Consumer(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker...(truncated),
    func=<function at remote 0x7fb3cf2b1c80>) at Python/ceval.c:4645
#135 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#136 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#137 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#138 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#139 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#140 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#141 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#142 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#143 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#144 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#145 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#146 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#147 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#148 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#149 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#150 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#151 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#152 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>)
    at Objects/call.c:283
#153 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#154 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#155 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#156 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#157 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#158 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#159 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#160 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x2c6b3a8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 258, in run (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at ...(truncated)) at Python/ceval.c:547
#161 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3d89706f0>,
    globals=globals@entry={'__name__': 'celery.bin.worker', '__doc__': "Program used to start a Celery worker instance.\n\nThe :program:`celery worker` command (previously known as ``celeryd``)\n\n.. program:: celery worker\n\n.. seealso::\n\n    See :ref:`preload-options`.\n\n.. cmdoption:: -c, --concurrency\n\n    Number of child processes processing the queue.  The default\n    is the number of CPUs available on your system.\n\n.. cmdoption:: -P, --pool\n\n    Pool implementation:\n\n    prefork (default), eventlet, gevent or solo.\n\n.. cmdoption:: -n, --hostname\n\n    Set custom hostname (e.g., 'w1@%%h').  Expands: %%h (hostname),\n    %%n (name) and %%d, (domain).\n\n.. cmdoption:: -B, --beat\n\n    Also run the `celery beat` periodic task scheduler.  Please note that\n    there must only be one instance of this service.\n\n    .. note::\n\n        ``-B`` is meant to be used for development purposes. For production\n        environment, you need to start :program:`celery beat` separately.\n\n.. cmdoption:: -Q, --queues\n\n    L...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea26e60, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c6b830,
    kwargs=kwargs@entry=0x2c6b838, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x7fb3d89cf8d0, defcount=9, kwdefs=0x0,
    closure=0x0, name='run', qualname='worker.run') at Python/ceval.c:3930
#162 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d8979620>,
    args=args@entry=0x7ffdbea26e60, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'schedu---Type <return> to continue, or q <return> to quit---
ler': None}) at Objects/call.c:376
#163 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea26e60, callable=<function at remote 0x7fb3d8979620>) at Objects/call.c:98
#164 _PyObject_Call_Prepend (callable=<function at remote 0x7fb3d8979620>, obj=<optimized out>, args=(),
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/call.c:904
#165 0x000000000043c638 in PyObject_Call (callable=callable@entry=<method at remote 0x7fb3e6238d08>, args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#166 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(), func=<method at remote 0x7fb3e6238d08>) at Python/ceval.c:4645
#167 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#168 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414a48, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 252, in __call__ (self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread....(truncated)) at Python/ceval.c:547
#169 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3e6172300>,
    globals=globals@entry={'__name__': 'celery.bin.base', '__doc__': 'Base command-line interface.', '__package__': 'celery.bin', '__loader__': <SourceFileLoader(name='celery.bin.base', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py') at remote 0x7fb3e6154748>, '__spec__': <ModuleSpec(name='celery.bin.base', loader=<...>, origin='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/__pycache__/base.cpython-37.pyc', _initializing=False) at remote 0x7fb3e6154780>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py', '__cached__': '/app/aep/.local/share/virtualenvs/a02e6b...(truncated),
    locals=locals@entry=0x0, args=args@entry=0x7ffdbea271d0, argcount=argcount@entry=1, kwnames=kwnames@entry=0x2c69670,
    kwargs=kwargs@entry=0x2c69678, kwcount=78, kwcount@entry=39, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0, name='__call__', qualname='Command.__call__') at Python/ceval.c:3930
#170 0x000000000043b027 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7fb3d89a1a60>,
    args=args@entry=0x7ffdbea271d0, nargs=nargs@entry=1,
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:376
#171 0x000000000043e749 in _PyObject_FastCallDict (
    kwargs={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, nargs=1, args=0x7ffdbea271d0, callable=<function at remote 0x7fb3d89a1a60>) at Objects/call.c:98
#172 _PyObject_Call_Prepend (callable=callable@entry=<function at remote 0x7fb3d89a1a60>,
    obj=obj@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:904
---Type <return> to continue, or q <return> to quit---
#173 0x0000000000494665 in slot_tp_call (
    self=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated), args=(),
    kwds={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None})
    at Objects/typeobject.c:6379
#174 0x000000000043c638 in PyObject_Call (
    callable=callable@entry=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated),
    args=args@entry=(),
    kwargs=kwargs@entry={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}) at Objects/call.c:245
#175 0x0000000000422479 in do_call_core (
    kwdict={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child': 1, 'max_memory_per_child': None, 'purge': False, 'queues': 'bgl', 'exclude_queues': [], 'include': [], 'without_gossip': False, 'without_mingle': False, 'without_heartbeat': False, 'heartbeat_interval': None, 'autoscale': '10,3', 'logfile': '/var/log/ae-worker/ae-worker-3%I.log', 'pidfile': '/run/celery/ae-worker-3.pid', 'uid': None, 'gid': None, 'umask': None, 'executable': None, 'beat': False, 'schedule_filename': 'celerybeat-schedule', 'scheduler': None}, callargs=(),
    func=<worker(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_thread.lock at remote 0x7fb3d85114e0>, _pending=<collections.deque at remote 0x7fb3d85766c8>, _tasks=<TaskRegistry at remote 0x7fb3d855ab48>, _using_v1_reduce=None, _preconf={'include': ['ae_worker.ta...(truncated)) at Python/ceval.c:4645
#176 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3191
#177 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d7a13808, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/worker.py, line 223, in run_from_argv (prog_name='celery', argv=['-n', 'ae-worker-3.%h', '-A', 'ae_worker.app', '--pidfile=/run/celery/ae-worker-3.pid', '--autoscale=10,3', '--max-tasks-per-child', '1', '-Q', 'bgl', '--loglevel=INFO', '--logfile=/var/log/ae-worker/ae-worker-3%I.log', '--soft-time-limit=7200', '--time-limit=7300', '-Ofair', '--max-tasks-per-child', '1', '-E'], command='worker', options={'app': 'ae_worker.app', 'broker': None, 'result_backend': None, 'loader': None, 'config': None, 'workdir': None, 'no_color': None, 'quiet': False, 'hostname': 'ae-worker-3.%h', 'detach': False, 'statedb': None, 'loglevel': 'INFO', 'optimization': 'fair', 'prefetch_multiplier': 1, 'concurrency': 0, 'pool': 'prefork', 'task_events': True, 'time_limit': <float at remote 0x7fb3d7a32078>, 'soft_time_limit': <float at remote 0x7fb3d9031d38>, 'max_tasks_per_child':...(truncated)) at Python/ceval.c:547
#178 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8970300>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x7fb3d8a12f78, kwargs=kwargs@entry=0x7fb3d54143f0, kwcount=1, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d8973660, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='run_from_argv', qualname='worker.run_from_argv') at Python/ceval.c:3930
#179 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#180 0x0000000000428c25 in call_function (kwnames=('command',), oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at Python/ceval.c:4616
#181 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3139
#182 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d5414248, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 420, in execute (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<...(truncated)) at Python/ceval.c:547
#183 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d89f7540>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x2957570, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a54f8, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='execute',
    qualname='CeleryCommand.execute') at Python/ceval.c:3930
#184 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#185 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#186 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#187 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x29573b8, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 488, in handle_argv (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_per---Type <return> to continue, or q <return> to quit---
iodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _finalize_mutex=<_...(truncated)) at Python/ceval.c:547
#188 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a045d0>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=3, kwnames=0x0, kwargs=kwargs@entry=0x7fb3d897c1f0, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name='handle_argv',
    qualname='CeleryCommand.handle_argv') at Python/ceval.c:3930
#189 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#190 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#191 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#192 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3d897c048, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/base.py, line 298, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, _f...(truncated)) at Python/ceval.c:547
#193 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3e6172420>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x7fb3e60f0f18, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3e6174bc0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name='execute_from_commandline', qualname='Command.execute_from_commandline') at Python/ceval.c:3930
#194 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#195 0x0000000000428b80 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#196 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3093
#197 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb3e60f0d68, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/lib/python3.7/site-packages/celery/bin/celery.py, line 496, in execute_from_commandline (self=<CeleryCommand(app=<Celery(clock=<LamportClock(value=33276810, mutex=<DummyLock at remote 0x7fb3cf2b0358>) at remote 0x7fb3d85155c0>, main='ae_worker', amqp_cls='celery.app.amqp:AMQP', events_cls='celery.app.events:Events', loader_cls='celery.loaders.app:AppLoader', log_cls='celery.app.log:Logging', control_cls='celery.app.control:Control', task_cls='celery.app.task:Task', set_as_current=True, registry_cls=<type at remote 0x2918d18>, user_options={'preload': set(), 'worker': set()}, steps={'worker': set(), 'consumer': set()}, autofinalize=True, namespace=None, strict_typing=True, configured=True, _config_source='ae_worker.config.production.CeleryConfig', _pending_defaults=<collections.deque at remote 0x7fb3d898a730>, _pending_periodic_tasks=<collections.deque at remote 0x7fb3d898a3f0>, finalized=True, ...(truncated)) at Python/ceval.c:547
#198 _PyEval_EvalCodeWithName (_co=<code at remote 0x7fb3d8a06780>, globals=<optimized out>, locals=locals@entry=0x0,
    args=<optimized out>, argcount=2, kwnames=0x0, kwargs=kwargs@entry=0x274a150, kwcount=0, kwstep=kwstep@entry=1,
    defs=defs@entry=0x7fb3d89a55a0, defcount=defcount@entry=1, kwdefs=kwdefs@entry=0x0,
    closure=closure@entry=(<cell at remote 0x7fb3d895e5e8>,), name='execute_from_commandline',
    qualname='CeleryCommand.execute_from_commandline') at Python/ceval.c:3930
#199 0x000000000043b238 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:433
#200 0x00000000004280f2 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#201 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3110
#202 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>)
    at Objects/call.c:283
#203 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#204 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#205 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#206 0x00000000004200a8 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>)
    at Objects/call.c:283
#207 0x000000000043b306 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:415
#208 0x0000000000427d48 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4616
#209 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3124
#210 0x00000000004eba97 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x26c8708, for file /app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery, line 8, in <module> ()) at Python/ceval.c:547
#211 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0,
    kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0,
    name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:3930
#212 0x00000000004ebbc0 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0,
    args=0x0,
    locals=locals@entry='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery',
    globals=globals@entry=<unknown at remote 0x7fb3de9e4648>, _co=_co@entry=<code at remote 0x7fb3de9fb300>) at Python/ceval.c:3959
#213 PyEval_EvalCode (co=co@entry=<code at remote 0x7fb3de9fb300>,
    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}) at Python/ceval.c:524
#214 0x00000000005265f8 in run_mod (arena=0x7fb3e6282078, flags=0x7ffdbea285f0,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    filename='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', mod=0x26c1cc8)
    at Python/pythonrun.c:1035
#215 PyRun_FileExFlags (fp=0x26f05c0, filename_str=<optimized out>, start=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': ---Type <return> to continue, or q <return> to quit---
None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery') at remote 0x7fb3de9f2710>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fb3e627ac28>, '__file__': '/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery', '__cached__': None, 're': <module at remote 0x7fb3de9ac728>, 'sys': <module at remote 0x7fb3e6274f98>, 'main': <function at remote 0x7fb3de9b9ea0>}, closeit=1, flags=0x7ffdbea285f0) at Python/pythonrun.c:988
#216 0x00000000005267dd in PyRun_SimpleFileExFlags (fp=0x26f05c0, filename=<optimized out>, closeit=1, flags=0x7ffdbea285f0)
    at Python/pythonrun.c:429
#217 0x000000000042fdfd in pymain_run_file (p_cf=0x7ffdbea285f0,
    filename=0x2655d10 L"/app/aep/.local/share/virtualenvs/a02e6b6588a005e3a7627e805171154cf77d15b0-n-emIVEOn1/bin/celery",
    fp=0x26f05c0) at Modules/main.c:427
#218 pymain_run_filename (cf=0x7ffdbea285f0, pymain=0x7ffdbea286d0) at Modules/main.c:1627
#219 pymain_run_python (pymain=0x7ffdbea286d0) at Modules/main.c:2877
#220 pymain_main (pymain=pymain@entry=0x7ffdbea286d0) at Modules/main.c:3038
#221 0x00000000004300cd in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3073
#222 0x00007fb3e516c495 in __libc_start_main () from /lib64/libc.so.6
#223 0x0000000000429df4 in _start ()

Dépendances:

celery==4.3.0
kombu==4.6.5
billard==3.6.1.0
amqp==2.5.1
python==3.7.3

pouvez-vous essayer le céleri == 4.4.0rc4?

Il me faudra une semaine pour l'examiner en phase de mise en scène, mais je vais essayer de faire fonctionner une partie de la charge en cours de production.

@ wakemaster39 :

Aussi curieux de savoir si cela semble avoir résolu le problème @ wakemaster39

céleri == 4.4.0rc5?

J'ai été obligé de résoudre différents problèmes ces dernières semaines et le site avec lequel nous avons des problèmes est en train de se déconnecter pendant les vacances, je ne veux donc pas y apporter de modifications. Je le vérifie en préparation pour la semaine prochaine dans une demie dans l'espoir de se déployer pour produire la nouvelle année pour obtenir une charge contre elle.

Savons-nous si ce problème est un problème avec les ouvrières du céleri en particulier et pas avec d'autres parties du céleri comme beat ? Ergo si je passe à quelque chose comme go-celery et continue à utiliser celery.beat est-ce que cela résout théoriquement mon problème? Ou les gens ont-ils eu des problèmes avec beat stagnant pour quelque raison que ce soit?

Dans mon cas, ce n'est même pas le battement qui meurt, ses fils de céleri dans l'impasse. Je l'ai corrigé pour moi en ajoutant un délai d'attente, mais de temps en temps, le fil s'arrête. Cela s'est produit beaucoup plus lorsque j'ai eu des transactions de base de données qui n'étaient pas atomiques, mais la correction de cela a corrigé certains de mes maux de tête.

tldr: ce n'est pas un battement, mais un ouvrier qui "se bloque aléatoirement sur IPC"

Vu des choses similaires en cours d'exécution céleri = "== 4.4.0". Beat fonctionne bien mais le céleri reste coincé pour une raison quelconque. Peut-être une impasse mais très difficile à dire

J'ai eu une expérience similaire à @ vincent31337 . Je pense que cela est évité en n'utilisant pas la commande celery multi start .

Je suis confronté au même problème. Python2.7, céleri4.1.0. J'utilise le courtier SQS. Le céleri traite les tâches pendant un certain temps et il se bloque brusquement après un certain temps.

vous devriez essayer les nouvelles versions.

Il est corrigé après la mise à niveau.

Je voudrais savoir si c'est toujours le cas avec le céleri => 4.4.x

Je suis confronté à ce même problème:

python 3.8
Django 3.0.3
céleri 4.4.0
redis 3.4.1 (pour le courtier et les résultats)

Je dirige cinq ouvriers. Je peux exécuter cinq tâches qui se terminent, puis le travailleur raccroche sur read([different pipe value for each] . Une fois que les tâches sont acceptées et apparaissent dans Inspect(app=app).active() , cependant, leur worker_pid n'est pas l'un des pids des travailleurs. Souvent, les pids sont des valeurs faibles comme 50. En utilisant ps -p [low pid] je ne vois pas ces pids en cours d'exécution. Je n'ai pas encore vu cela mentionné dans ce fil.

même avec du céleri == 4.4.2 non?

Nous sommes confrontés exactement au même problème - avec le céleri == 4.3 et en essayant le dernier céleri = 4.4.2
nous utilisons rabbitmq comme backend (courtier et résultats)

kombu == 4.6.8
céleri == 4.4.2
amqp == 2.5.2
billard == 3.6.3.0
librabbitmq == 2.0.0
Python 3.7.4

le céleri est bloqué lors de la lecture et ne peut effectuer aucune autre tâche.

$ sudo strace -p 8553
strace: Process 8553 attached
read(10, ^Cstrace: Process 8553 detached
 <detached ...>

$ sudo strace -p 8557
strace: Process 8557 attached
read(14, ^Cstrace: Process 8557 detached
 <detached ...>

$ sudo strace -p 8561
strace: Process 8561 attached
read(18, ^Cstrace: Process 8561 detached
 <detached ...>

Un exemple de lsof ourput pour le processus 8553
celery 8553 ubuntu 10r FIFO 0,10 0t0 43059 pipe

https://github.com/celery/celery/issues/4185#issuecomment -545617121

FWIW, nous semblons avoir rencontré le même problème avec une utilisation brute de billiard==3.6.3.0 (indépendamment du céleri):
parent:

(gdb) py-bt                                                                                
Traceback (most recent call first):             
  <built-in method acquire of _thread.lock object at remote 0x7f838895e620>
  File "/usr/lib/python3.6/threading.py", line 295, in wait                           
    waiter.acquire()
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1957, in next
    self._cond.wait(timeout)
  File "xxx.py", line 1063, in main
    for res in self.pool.imap_unordered(_partial, itertools.islice(a_generator, nb_processes)):
...

processus enfant bloqué:

0x00007f8393cc11c4 in __GI___libc_read (fd=21, buf=0x7f838892c578, nbytes=4) at ../sysdeps/unix/sysv/linux/read.c:27
Traceback (most recent call first):
  <built-in method read of module object at remote 0x7f8392d3de58>
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 422, in _recv
    chunk = read(handle, remaining)
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 456, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.6/dist-packages/billiard/connection.py", line 243, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 395, in get_payload
    return self._reader.recv_bytes()
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
...

C'est la même pile que dans https://github.com/celery/celery/issues/4185#issuecomment -554828564 py-bt stack trace or forked process

tous les autres processus enfants sont en semlock:

Traceback (most recent call first):
  <built-in method __enter__ of _multiprocessing.SemLock object at remote 0x7f838896ff10>
  File "/usr/local/lib/python3.6/dist-packages/billiard/synchronize.py", line 116, in __enter__
    return self._semlock.__enter__()
  File "/usr/local/lib/python3.6/dist-packages/billiard/queues.py", line 394, in get_payload
    with self._rlock:
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
...

J'ai observé le problème des travailleurs obsolètes sur plusieurs instances d'OpenWISP.

celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0

Commander:

celery -A openwisp2 worker -l info --pool=gevent --concurrency=10 -Ofair

La seule différence que je peux voir entre un travailleur qui travaille et un qui ne travaille pas est que le travailleur qui ne traite aucune tâche le montre dans la sortie strace :

select(22, [21], [], [21], {tv_sec=0, tv_usec=0}) = 0 (Timeout)

Même si le travailleur traite des tâches, je vois ces deux types de lignes dans la sortie strace:

epoll_ctl(4, EPOLL_CTL_ADD, 20, {EPOLLIN, {u32=20, u64=12884901908}}) = -1 EEXIST (File exists)
recvfrom(20, 0x377a460, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

Par conséquent, je ne pense pas que ces deux dernières lignes de journal indiquent un problème sous-jacent, ou du moins que ce ne sont pas des symptômes du problème.

# strace -p 81
strace: Process 81 attached
read(24, ^Cstrace: Process 81 detached
 <detached ...>

# lsof -p 81 | grep 24
celery   81 root  mem       REG              252,1             535124 /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (path dev=0,112)
celery   81 root   24r     FIFO               0,13      0t0 311875006 pipe



md5-a6754624afaf3bd93564d53ec027c759



python version: 3.6
celery==4.4.7
kombu==4.6.11
billiard==3.6.3.0

Bonjour,

J'ai été confronté au même problème ici:

rapport d'inspection du céleri:

software -> celery:4.3.0 (rhubarb) kombu:4.6.4 py:3.6.6
                    billiard:3.6.1.0 py-amqp:2.5.1
        platform -> system:Linux arch:64bit
                    kernel version:3.10.0-1062.18.1.el7.x86_64 imp:CPython
        loader   -> celery.loaders.app.AppLoader
        settings -> transport:amqp results:amqp

        broker_url: 'amqp://nextgen:********@*:5672/ng_vhost'
        result_backend: 'amqp'
        task_queues: 
            (<unbound Queue default -> <unbound Exchange default(direct)> -> default>,
         <unbound Queue runForecasts -> <unbound Exchange runForecasts(direct)> -> runForecasts>,
         <unbound Queue runEnsembles -> <unbound Exchange runEnsembles(direct)> -> runEnsembles>)
        task_default_queue: 'default'
        task_default_exchange: 'default'
        task_default_routing_key: '********'
        broker_heartbeat: None
        task_acks_late: True
        worker_prefetch_multiplier: 1

CELERYD_OPTS:
--time-limit=3600 -c:worker1 3 -Q:worker1 runForecasts -c:worker2 2 -Q:worker2 runEnsembles -c:worker3 2 -Q:worker3 default

Seule la file d'attente runForecasts semble rester bloquée, et voici la sortie strace pour les processus enfants:

$ sudo strace -p 17426
strace: Process 17426 attached
read(28, ^Cstrace: Process 17426 detached
 <detached ...>
$ sudo ls -l /proc/17426/fd/28
lr-x------. 1 wind wind 64 Nov 10 22:45 /proc/17426/fd/28 -> pipe:[284399927]

J'ai trouvé le tuyau sur lequel il est collé, mais je ne sais pas comment déboguer davantage. J'apprécierais que quelqu'un puisse faire la lumière à ce sujet.

# strace -p 32
strace: Process 32 attached
epoll_wait(7, [], 1023, 787)            = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 4050)           = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [{EPOLLIN, {u32=15, u64=139972984176655}}], 1023, 275) = 1
ioctl(15, FIONBIO, [1])                 = 0
recvfrom(15, "\10\0\0\0\0\0\0", 7, 0, NULL, NULL) = 7
recvfrom(15, "\316", 1, 0, NULL, NULL)  = 1
ioctl(15, FIONBIO, [0])                 = 0
ioctl(15, FIONBIO, [1])                 = 0
recvfrom(15, 0x7f4e96911aa0, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
ioctl(15, FIONBIO, [0])                 = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 19)             = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
epoll_wait(7, [], 1023, 670)            = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000)           = 0
epoll_ctl(7, EPOLL_CTL_DEL, 43, 0x7ffe2bd34534) = -1 ENOENT (No such file or directory)
wait4(631, 0x7ffe2bd33c3c, WNOHANG, NULL) = 0
epoll_wait(7, [], 1023, 5000)           = 0

lrwx------. 1 root root 64 Dec  3 08:21 7 -> 'anon_inode:[eventpoll]'

céleri == 4.4.7

Arguments de travail: --autoscale et --max-tasks-per-child

Je vais tester cela.

J'utilise --autoscale 20,1 et --max-tasks-per-child 1

tâche

<strong i="10">@share_task</strong>
def echo():
      print(1)

# wait for processes scaling down and run next
for i in range(100):
     echo.delay()


# once large call
for i in range(10000):
     echo.delay()

J'ai une autre erreur ：

[2020-12-03 18:03:42,137: CRITICAL/MainProcess] Unrecoverable error: AssertionError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 303, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 145, in fire_timers
    entry()
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 636, in verify_process_alive
    assert proc.outqR_fd in hub.readers
AssertionError

DE https://github.com/celery/celery/issues/4185#issuecomment -694179347
J'ai examiné la ligne de billard / pool.py 1957, et j'ai trouvé:

    self._cond = threading.Condition(threading.Lock())
    ......
    self._lost_worker_timeout = lost_worker_timeout
    ......
    def next(self, timeout=None):
        with self._cond:
                ......
                self._cond.wait(timeout)
                ......

self._cond.wait pas de délai d'expiration. Est-il coincé à cet endroit?

pour ceux qui font face à cela, pouvez-vous essayer https://github.com/celery/billiard/pull/325?

https://github.com/celery/billiard/pull/325 peut ne pas fonctionner.

Et j'abandonne. Maintenant, j'utilise --pool threads