celery -A proj report
en la edición.software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.4.3
billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
master
de Apio.Actualice a Apio 4.0.2.
No hay error.
Recibiendo este error de mis trabajadores:
[2017-01-19 04:07:57,411: CRITICAL/MainProcess] Couldn't ack 461, reason:BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/channel.py", line 1408, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/abstract_channel.py", line 64, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/method_framing.py", line 174, in write_frame
write(view[:offset])
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/transport.py", line 269, in write
self._write(s)
BrokenPipeError: [Errno 32] Broken pipe
Esto hace que el trabajador restablezca su conexión a RabbitMQ cada pocos segundos y, a veces, lleva a alcanzar el límite máximo de conexión permitido en el servidor RabbitMQ.
Posiblemente relacionado con # 3328 y # 3377.
No estoy seguro de qué está causando esto, pero comienza a suceder uno o dos días. Reiniciar los trabajadores lo soluciona por un tiempo.
También veo este problema, aunque en una versión un poco más antigua de apio.
software -> celery:3.1.23 (Cipater) kombu:3.0.35 py:3.5.3
billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:djcelery.backends.database:DatabaseBackend
+1
+1 viendo este problema últimamente en apio 3.2.1
+1 Viendo este problema últimamente el 3.1.18
+1 al ver el problema en 4.0.2
+1 al ver el problema en 4.0.2
+1 al ver el problema en 4.0.2
+1 al ver el problema en 4.0.2
+1 viendo este problema en 4.0.2
Para su información, este y otros problemas de Celery 4.x se solucionaron cuando bajé la calificación a 3.x. La solución a largo plazo es cambiar a https://github.com/Bogdanp/dramatiq , que tiene la misma sintaxis que Celery .delay()
por lo que es muy fácil de cambiar.
+1 viendo en 4.0.2
+1 viendo en 4.0.2
+1 viendo en 4.1.0
+1 viendo en 4.1.0
+1 viendo en apio (4.1.0), gevent
+1 apio 4.1.0
+1
Viendo esto en 4.1.0 con el broker rabbitmq
+1
Ocurre cuando una tarea de revocación con terminate
argumento: app.control.revoke(t.id, terminate=True)
Lo mismo cuando termina la tarea de Flower.
Python 3.6.2rc2
Celery 4.1.0 (latentcall)
broker: rabbitmq:3.6.12
backend: redis:3.2.10
[2017-10-11 09:31:49,471: ERROR/MainProcess] Control command error: BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/celery/worker/pidbox.py", line 42, in on_message
self.node.handle_message(body, message)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 129, in handle_message
return self.dispatch(**body)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 112, in dispatch
ticket=ticket)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 135, in reply
serializer=self.mailbox.serializer)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 265, in _publish_reply
**opts
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/channel.py", line 1734, in _basic_publish
(0, exchange, routing_key, mandatory, immediate), msg
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/transport.py", line 258, in write
self._write(s)
BrokenPipeError: [Errno 32] Broken pipe
+1 apio 4.0.0
ver la última explicación de @mihajenko https://github.com/celery/celery/issues/3377
+1 apio (4.0.2)
Traceback (most recent call last):
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/channel.py", line 1408, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame
write(view[:offset])
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/transport.py", line 269, in write
self._write(s)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 397, in sendall
tail = self.send(data, flags)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 391, in send
return self._send_loop(self.fd.send, data, flags)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 378, in _send_loop
return send_method(data, *args)
error: [Errno 32] Broken pipe
@auvipy , vuelva a abrir. Esto es un error ... un truco no califica como una solución.
@auvipy ¿Podría dar una solución rápida? No estoy familiarizado con el apio. ¡Gracias!
+1 4,4
+1 @ _ @
+1
en lugar de poner +1, tómese su tiempo para investigar los problemas relacionados para que pueda encontrar una posible solución. ¿Sería genial si alguien pudiera informar que el problema persiste en 4.2rc2?
@auvipy Hemos estado investigando el problema, tenemos una infra bastante grande de apio + conejomq. Hoy hemos cambiado de la versión 3.1.25 a la versión 4.2. Empezamos a ver un% 4 de caída de tareas. Este error coincidió con la caída / pérdida de tareas. ¿Alguna vez ha enfrentado / escuchado un problema similar en la transición?
¿Podría probar primero https://github.com/celery/celery/releases/tag/v3.1.26 localmente y luego actualizar a 4.2rc2? obtuvimos un informe de zapier, luego se estaban moviendo de 3.1 a 4.1, otra cosa es que aunque el apio es 4.2rc2 en pypi, las dependencias aún no están en pypi. Si pudiera tomarse su tiempo para actualizar las dependencias de la rama maestra con 4.2rc2 y ver qué sucede y háganoslo saber, sería realmente genial. después de eso, podríamos marcarlo como de alta prioridad / bloqueador para la versión 4.2 final.
también solicito plantear el problema en la lista de correo
Lo siento @auvipy , solo para aclarar: ¿le gustaría que probemos el código @ 4.2rc2 de pypi, pero con las dependencias actualizadas para que coincidan con la lista de dependencias en la rama maestra?
no todos, pero si hay alguno, apio 4.2rc2 de pypi y otras dependencias de apio de github masters brnach en servidores locales / de ensayo
Trabajo con @burakbostancioglu , así que sí, podemos probar esto =]
gracias :)
¿Alguna actualización chicos? usando 4.2rc4?
Veo que esto ocurre con bastante regularidad en Celery 4.2.0 cuando está bajo carga (probando la migración 3.1.25 -> 4.2.0, pero esto lo está bloqueando). No soy un mono parcheando greenlets. RMQ como broker, redis como backend de resultados.
Estamos atrapados demasiado para una migración si podemos evitarlo, pero agradezco el consejo :-)
Por favor, avíseme si hay algo en lo que pueda profundizar con respecto a este error.
Apliqué la solución mencionada en
https://github.com/celery/celery/issues/4226 y me resolvió el problema.
Entonces, ¿este problema está resuelto o en la forma en que se resolverá? Todo mi flujo de trabajo depende del apio y realmente disfruto de esta herramienta.
Si el error está en camino de resolverse, ¿hay un quickfick antes de la resolución real?
@xirdneh si puede investigar el problema relacionado siempre que pueda.
Configuré estático --concurrency
lugar de --autoscale
y esto me resolvió el problema.
(Tuve el error con la concurrencia 1 y sin escala automática)
Sigo teniendo el mismo problema con el apio 4.2.1. Espero que un informe más sea útil.
software -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:2.7.15rc1
billiard:3.5.0.4 redis:2.10.6
platform -> system:Linux arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379/0
broker_url: u'redis://127.0.0.1:6379/0'
result_backend: u'redis://127.0.0.1:6379/0'
task_routes: {
u'celery.crawler.fbk.download_alt_text': { u'queue': u'fbk_alt_text'}}
¿Alguien quiere probar con el maestro del apio?
esto todavía existe.
default celery-67c7fff78b-g48vx celery Couldn't ack 23, reason:BrokenPipeError(32, 'Broken pipe')
default celery-67c7fff78b-g48vx celery Traceback (most recent call last):
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 130, in ack_log_error
default celery-67c7fff78b-g48vx celery self.ack(multiple=multiple)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 125, in ack
default celery-67c7fff78b-g48vx celery self.channel.basic_ack(self.delivery_tag, multiple=multiple)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 1399, in basic_ack
default celery-67c7fff78b-g48vx celery spec.Basic.Ack, argsig, (delivery_tag, multiple),
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 51, in send_method
default celery-67c7fff78b-g48vx celery conn.frame_writer(1, self.channel_id, sig, args, content)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/method_framing.py", line 172, in write_frame
default celery-67c7fff78b-g48vx celery write(view[:offset])
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 288, in write
default celery-67c7fff78b-g48vx celery self._write(s)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 458, in sendall
default celery-67c7fff78b-g48vx celery return _socketcommon._sendall(self, data_memory, flags)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 374, in _sendall
default celery-67c7fff78b-g48vx celery timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk
default celery-67c7fff78b-g48vx celery data_sent += socket.send(chunk, flags)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 439, in send
default celery-67c7fff78b-g48vx celery return _socket.socket.send(self._sock, data, flags)
default celery-67c7fff78b-g48vx celery BrokenPipeError: [Errno 32] Broken pipe
parece ser un tema gevent? ile "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", línea 439, en enviar
predeterminado celery-67c7fff78b-g48vx apio return _socket.socket.send (self._sock, datos, banderas)
predeterminado celery-67c7fff78b-g48vx apio BrokenPipeError: [Errno 32] Tubo roto
apio 4.4.0rc1?
probado en la rama maestra con el parche propuesto en https://github.com/celery/celery/issues/3377
todavía se atasca después de una hora de funcionamiento
+1 apio 4.2.1
@auvipy He visto este problema con Celery 4.3.0 + gevent + RabbitMQ. Usando strace
encontré que el problema se relaciona con recvfrom
y epoll_wait
syscall. Parece que el socket no puede recibir msg de RabbitMQ y cae en un bucle infinito, el proceso de Apio se atascó allí.
26738
es ID de proceso de apio
sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
Más información sobre el inodo 5, 15, 21:
$ sudo ls -la /proc/26738/fd/5
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/5 -> socket:[99157475]
$ sudo ls -la /proc/26738/fd/21
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/21 -> socket:[99144296]
$ sudo ls -la /proc/26738/fd/15
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/15 -> anon_inode:[eventpoll]
$ sudo lsof -p 26738 | grep 99157475
celery 26738 xxx 5u IPv4 99157475 0t0 TCP xxx-1084:50954->rabbit.xxx-1084:amqp (ESTABLISHED)
$ sudo lsof -p 26738 | grep 99144296
celery 26738 xxx 21u IPv4 99144296 0t0 TCP xxx-1084:38194->rabbit.xxx-1084:amqp (ESTABLISHED)
$ sudo head -n1 /proc/26738/net/tcp; grep -a 99157475 /proc/26738/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
10: 8A01010A:C70A 5E00010A:1628 01 00000000:00000000 02:00000351 00000000 1005 0 99157475 2 0000000000000000 20 4 30 10 -1
Espero eso ayude.
+1
+1
@duydo Me encuentro con la misma situación contigo. Y rabbitMQ no pudo confirmar nada.
Traceback (most recent call last):
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/channel.py", line 1399, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/abstract_channel.py", line 51, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/method_framing.py", line 172, in write_frame
write(view[:offset])
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/transport.py", line 288, in write
self._write(s)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 458, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 374, in _sendall
timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk
data_sent += socket.send(chunk, flags)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 439, in send
return _socket.socket.send(self._sock, data, flags)
BrokenPipeError: [Errno 32] Broken pipe
Uso strace para rastrear PID
$ sudo strace -p 27977 -f
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108490973}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108664276}) = 0
[pid 27977] epoll_wait(4, [], 64, 1999) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=108549437}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109251966}) = 0
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109598283}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109777281}) = 0
[pid 27977] epoll_wait(4, [], 64, 1529) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=640754845}) = 0
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641809883}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641940654}) = 0
[pid 27977] epoll_wait(4, [], 64, 467) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=109901002}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=110880127}) = 0
[pid 27977] getpid() = 27977
$ sudo lsof -p 27977|grep 1651852686
python 27977 ymserver 10u IPv4 1651852686 0t0 TCP vuljp-ag-proxy-01:33006->10.55.4.70:amqp (ESTABLISHED)
https://github.com/celery/celery/pull/5581/files prueba esto
+1 apio v4.4.2
+1 apio 4.3.0
+1 apio 4.3.0
en lugar de +1 spam, compruebe las relaciones públicas relacionadas y mejórelas con pruebas unitarias.
+1 para apio 4.4.6., No importa con python2 o python3 env.
software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:2.7.10
billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Darwin arch:64bit
kernel version:17.7.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:redis://10.28.218.6/
El registro está aquí:
worker: Warm shutdown (MainProcess)
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Hub...
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Pool...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Closing Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Connection...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Events...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Heart...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Control...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping Control...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Closed channel #3
[2020-07-17 22:27:52,153: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:27:53,279: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2020-07-17 22:27:53,280: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] Closed channel #2
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Events...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2020-07-17 22:27:53,843: DEBUG/MainProcess] | Worker: Stopping Pool...
^@^@[2020-07-17 22:31:07,678: INFO/ForkPoolWorker-1] Task tasks.handle_compressed[5e493207-a3c5-422e-ab82-1f382342273c] succeeded in 200.757257249s: {'name': 'ray'}
[2020-07-17 22:31:09,188: DEBUG/MainProcess] | Worker: Stopping Hub...
[2020-07-17 22:31:09,189: CRITICAL/MainProcess] Couldn't ack 1, reason:error(32, 'Broken pipe')
Traceback (most recent call last):
File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 131, in ack_log_error
self.ack(multiple=multiple)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 126, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py", line 1394, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py", line 59, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py", line 172, in write_frame
write(view[:offset])
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py", line 305, in write
self._write(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2020-07-17 22:31:09,195: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Closing consumer channel...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2020-07-17 22:31:09,431: DEBUG/MainProcess] Closed channel #1
[2020-07-17 22:31:09,663: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2020-07-17 22:31:09,665: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
FYI. deshabilitar broker_heartbeat (broker_heartbeat = 0) puede solucionar mi problema. No estoy seguro de que sea una buena práctica.
Espero que esta información de clasificación ayude al análisis del caso raíz.
> ps -ef | grep worker
502 92665 75622 0 10:27PM ttys008 0:00.78 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
502 92672 92665 0 10:27PM ttys008 0:00.01 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
> sudo dtruss -p 75622 -f
......
92665/0xf94bb7: fcntl(0x5, 0x4, 0x4) = 0 0
92665/0xf94bb7: wait4(0x16A00, 0x7FFEEB4B0C10, 0x1) = 0 0
92665/0xf94bb7: wait4(0x16A00, 0x7FFEEB4B13E0, 0x0) = 92672 0
92665/0xf94bb7: close(0xB) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x15) = -1 Err#32
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF680, 0x0) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: mmap(0x0, 0x40000, 0x3, 0x1002, 0xFFFFFFFFFFFFFFFF, 0x0) = 0x1069D1000 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xE, 0x1068F0000, 0x13) = 19 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970E4, 0x7) = 7 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970B4, 0x4) = 4 0
92665/0xf94bb7: recvfrom(0xE, 0x10676348C, 0x1) = 1 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: madvise(0x1068CF000, 0x21000, 0x9) = 0 0
92665/0xf94bb7: sendto(0xE, 0x1068F0000, 0x13) = 19 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970E4, 0x7) = 7 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970B4, 0x4) = 4 0
92665/0xf94bb7: recvfrom(0xE, 0x10676348C, 0x1) = 1 0
92665/0xf94bb7: shutdown(0xE, 0x2, 0x0) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x13) = -1 Err#32
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x13) = -1 Err#32
92665/0xf94bb7: shutdown(0xD, 0x2, 0x0) = -1 Err#57
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: select(0x4, 0x7FFEEB4A6E58, 0x7FFEEB4A6ED8, 0x7FFEEB4A6F58, 0x7FFEEB4A6FD8) = 0 0
92665/0xf94bb7: fcntl(0x4, 0x3, 0x0) = 1 0
92665/0xf94bb7: fcntl(0x4, 0x4, 0x1) = 0 0
dtrace: error on enabled probe ID 2172 (ID 161: syscall::write:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: lstat64("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x7FFEEB4B34F0, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x1100004, 0x1213200) = 11 0
92665/0xf94bb7: fstatfs64(0xB, 0x7FFEEB4B2DB8, 0x0) = 0 0
92665/0xf94bb7: getdirentries64(0xB, 0x7FBE640A0A00, 0x1000) = 64 0
92665/0xf94bb7: getdirentries64(0xB, 0x7FBE640A0A00, 0x1000) = 0 0
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: rmdir(0x7FBE61640B10, 0x0, 0x0) = 0 0
92665/0xf94bb7: sigaction(0x2, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0x3, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0xF, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0x1E, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: madvise(0x105752000, 0x3C000, 0x9) = -1 Err#22
92665/0xf94bb7: close(0xD) = 0 0
92665/0xf94bb7: madvise(0x1068AF000, 0x20000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x106A11000, 0x23000, 0x9) = -1 Err#22
92665/0xf94bb7: close(0xE) = 0 0
92665/0xf94bb7: madvise(0x1068F0000, 0x20000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x10486D000, 0x30000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C25000, 0x5000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C81000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C83000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C85000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C87000, 0x2000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C8A000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C8F000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C92000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C99000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C9E000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA1000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA3000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA6000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CAB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CAE000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CB2000, 0x2000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CB6000, 0x4000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CBB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CBE000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CEA000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CFB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: close(0x6) = 0 0
dtrace: error on enabled probe ID 2366 (ID 897: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
^@^@^@^@^@^C
ranc-a01:~ ranc$ lsof -p 92665
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 92665 ranc cwd DIR 1,4 2176 12342436 /Users/ranc/vsan_vsancertification/tools/graphrunner
python 92665 ranc txt REG 1,4 51744 12719059 /Users/ranc/p2dev/bin/python
python 92665 ranc txt REG 1,4 52768 378470 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
python 92665 ranc txt REG 1,4 84240 378488 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
python 92665 ranc txt REG 1,4 83232 378468 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
python 92665 ranc txt REG 1,4 60192 378500 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
python 92665 ranc txt REG 1,4 71392 378444 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
python 92665 ranc txt REG 1,4 85136 378540 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
python 92665 ranc txt REG 1,4 108192 378532 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
python 92665 ranc txt REG 1,4 59984 378462 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
python 92665 ranc txt REG 1,4 48256 378458 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
python 92665 ranc txt REG 1,4 253840 378466 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
python 92665 ranc txt REG 1,4 73456 378552 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
python 92665 ranc txt REG 1,4 200880 378448 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
python 92665 ranc txt REG 1,4 65952 378558 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
python 92665 ranc txt REG 1,4 88816 378494 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
python 92665 ranc txt REG 1,4 78896 378534 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
python 92665 ranc txt REG 1,4 58240 378460 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
python 92665 ranc txt REG 1,4 52656 378478 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
python 92665 ranc txt REG 1,4 57040 378508 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
python 92665 ranc txt REG 1,4 74592 378446 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_csv.so
python 92665 ranc txt REG 1,4 43536 378526 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
python 92665 ranc txt REG 1,4 65184 378562 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
python 92665 ranc txt REG 1,4 144736 378482 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
python 92665 ranc txt REG 1,4 172352 378486 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
python 92665 ranc txt REG 1,4 44752 378480 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
python 92665 ranc txt REG 1,4 43708 12722902 /Users/ranc/p2dev/lib/python2.7/site-packages/_scandir.so
python 92665 ranc txt REG 1,4 152272 378514 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
python 92665 ranc txt REG 1,4 65920 378550 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
python 92665 ranc txt REG 1,4 52048 378520 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
python 92665 ranc txt REG 1,4 67920 378476 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
python 92665 ranc txt REG 1,4 146112 378506 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
python 92665 ranc txt REG 1,4 52048 378548 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/resource.so
python 92665 ranc txt REG 1,4 84000 378504 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/bz2.so
python 92665 ranc txt REG 1,4 43104 378430 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
python 92665 ranc txt REG 1,4 104256 378544 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so
python 92665 ranc txt REG 1,4 25512 15160994 /Users/ranc/p2dev/lib/python2.7/site-packages/tornado/speedups.so
python 92665 ranc txt REG 1,4 138464 378452 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
python 92665 ranc txt REG 1,4 1409984 378560 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/unicodedata.so
python 92665 ranc txt REG 1,4 27108 12722492 /Users/ranc/p2dev/lib/python2.7/site-packages/markupsafe/_speedups.so
python 92665 ranc txt REG 1,4 43588 14636845 /Users/ranc/p2dev/lib/python2.7/site-packages/_billiard.so
python 92665 ranc txt REG 1,4 65776 378536 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/mmap.so
python 92665 ranc txt REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc txt REG 1,4 841456 2716638 /usr/lib/dyld
python 92665 ranc txt REG 1,4 1172373504 13558190 /private/var/db/dyld/dyld_shared_cache_x86_64h
python 92665 ranc 0u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 1u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 2u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 3 PIPE 0x75a36a27dcf37cad 16384 ->0x75a36a27dcf355ad
python 92665 ranc 4 PIPE 0x75a36a27dcf355ad 16384 ->0x75a36a27dcf37cad
python 92665 ranc 5 PIPE 0x75a36a27dcf3746d 16384 ->0x75a36a27dcf36dad
python 92665 ranc 6r CHR 14,1 0t15192 584 /dev/urandom
python 92665 ranc 7 PIPE 0x75a36a27dcf36dad 16384 ->0x75a36a27dcf3746d
python 92665 ranc 8u REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc 9u REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc 10r PSXSEM 0t0 /mp-r6SMtW
python 92665 ranc 11 PIPE 0x75a36a27dcf351ed 16384 ->0x75a36a27dcf36e6d
python 92665 ranc 12u systm 0x75a36a27dd0002cd 0t0 [ctl com.apple.netsrc id 8 unit 47]
python 92665 ranc 13u IPv4 0x75a36a280ae00f75 0t0 TCP 10.117.234.174:62477->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 14u IPv4 0x75a36a27fae89f75 0t0 TCP 10.117.234.174:62480->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 15u IPv4 0x75a36a27db0e88d5 0t0 TCP 10.117.234.174:62485->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 16 PIPE 0x75a36a27dcf351ed 16384 ->0x75a36a27dcf36e6d
¿Alguien descubrió una versión sin este error con gevent
?
Veo este problema (o posiblemente uno similar) al usar eventlet
.
¡Hola a todos!
+1 para apio 4.4.6.
Tuve este problema hoy) gracias a los comentarios anteriores y a una cuidadosa investigación en mis servidores, encontré el problema)
el problema en RabbitMQ y Redis)
Para solucionarlo, necesitamos instalar HAProxy para RabbitMQ y Redis) lamentablemente sin él, rompen la conexión y nos sale este error CRÍTICO) también hay información adicional en la documentación)
https://www.rabbitmq.com/networking.html#proxy -effects
CELERY_BROKER_URL - RabbitMQ
CELERY_RESULT_BACKEND - Redis
También configuré el reintento de apio y otros parámetros para que la tarea se ejecute exactamente porque para mí estos datos son importantes)
funciona con y sin gevent) espero que esto ayude mucho)
¡el apio no es el problema!
¡el apio no es el problema!
Sí, el apio es el problema; de lo contrario, la degradación a Apio 3 no solucionaría el problema.
@alanhamlett Hola)
Ha estado trabajando con HAProxy durante más de 5 días y todo está bien)
Recibí este error todo el tiempo y mis trabajadores fallaron en unos minutos)
Entiendo que esto se puede arreglar en el apio, pero la solución puede llevar mucho tiempo) es difícil hacer que la lógica funcione como HAProxy)
y no es necesario) HAProxy resuelve este problema perfectamente)
+1 Mismos síntomas que @duydo
Apio 4.4.7
No estoy seguro de cómo esto aún no se ha solucionado.
39315 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 534, in sendall
39316 return _socketcommon._sendall(self, data_memory, flags)
39317 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 392, in _sendall
39318 timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
39319 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 321, in __send_chunk
39320 data_sent += socket.send(chunk, flags)
39321 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 515, in send
39322 return self._sock.send(data, flags)
39323 BrokenPipeError: [Errno 32] Broken pipe
getpid() = 6
epoll_wait(3, [{EPOLLIN, {u32=125, u64=266287972477}}], 262, 767) = 1
recvfrom(125, "\1\0\1\0\0\0#", 7, 0, NULL, NULL) = 7
recvfrom(125, "\0<\0<\6None12\0\0\0\0\0\0\4\220\0\rcelery.pidb"..., 35, 0, NULL, NULL) = 35
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
recvfrom(125, "\2\0\1\0\0\0G", 7, 0, NULL, NULL) = 7
recvfrom(125, "\0<\0\0\0\0\0\0\0\0\0c\370\0\20application/json\5"..., 71, 0, NULL, NULL) = 71
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
recvfrom(125, "\3\0\1\0\0\0c", 7, 0, NULL, NULL) = 7
recvfrom(125, "{\"method\": \"enable_events\", \"arg"..., 99, 0, NULL, NULL) = 99
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
getpid() = 6
write(2, "[2020-10-18 01:21:47,952: DEBUG/"..., 112) = 112
recvfrom(125, 0x7f8fbffe1078, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, [], 262, 108) = 0
getpid() = 6
epoll_wait(3, [], 262, 805) = 0
recvfrom(6, 0x7f8fbffe1de8, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, [], 262, 85) = 0
recvfrom(125, 0x7f8fbffe1d98, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, ^Cstrace: Process 4844 detached
@yshaban ¿Has probado HAProxy?
@yshaban ¿Has probado HAProxy?
Atasqué mis cosas detrás de haproxy hoy, te lo haré saber.
@yshaban ¿Has probado HAProxy?
Como sugirió Rostislav, crear un clúster rabbitmq y pegarlo detrás de haproxy resolvió el problema por mí mismo.
@yshaban agradable!
Esto solo se corrige cuando se usa haproxy, pero el error aún existe cuando se usa Celery directamente con RabbitMQ.
+1 apio v5.0.1
apio 4.3, haproxy soluciona el problema, deberían intentarlo.
Haga el uso requerido de Haproxy con Apio o vuelva a abrir este problema.
Un problema requiere elementos procesables.
Como no sabemos cómo solucionar esto y carecemos del entorno para reproducir este problema, diría que debería permanecer cerrado hasta que alguien que tenga acceso a un entorno donde esto se reproduzca pueda proporcionar una idea de lo que debería solucionarse o al menos qué está mal.
Estoy de acuerdo con los de otros, obviamente, hay alguna causa raíz dentro del apio que está causando que esto suceda y pegarlo detrás de haproxy, mientras resuelve el problema, no aborda la causa subyacente.
Estoy de acuerdo con los de otros, obviamente, hay alguna causa raíz dentro del apio que está causando que esto suceda y pegarlo detrás de haproxy, mientras resuelve el problema, no aborda la causa subyacente.
mi sospecha es que, como el transporte amqp no es asíncrono, es posible que la conexión no dure mucho.
las personas que se enfrentan a esto pueden verificar y probar
Comentario más útil
@auvipy , vuelva a abrir. Esto es un error ... un truco no califica como una solución.