Celery: OSError: Socket ditutup pada pekerja seledri dengan eventlet

Perhatikan bahwa saat menggunakan prefork alih-alih eventlet , server rabbitmq masih melaporkan peringatan detak jantung yang terlewat, tetapi tidak ada peringatan yang dicetak di pekerja:

2018-06-14 20:36:04.807 [warning] <0.698.0> closing AMQP connection <0.698.0> (172.19.0.3:46040 -> 172.19.0.2:5672):
missed heartbeats from client, timeout: 60s

Dan gevent gagal dengan pesan kesalahan yang mirip dengan eventlet :

[2018-06-14 20:43:29,908: WARNING/MainProcess] Traceback (most recent call last):
[2018-06-14 20:43:29,908: WARNING/MainProcess] File "src/gevent/_waiter.py", line 119, in gevent.__waiter.Waiter.switch
[2018-06-14 20:43:29,908: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 120, in loop
    connection.drain_events(timeout=1.0)
[2018-06-14 20:43:29,909: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
[2018-06-14 20:43:29,909: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
[2018-06-14 20:43:29,909: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 491, in drain_events
    while not self.blocking_read(timeout):
[2018-06-14 20:43:29,910: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 496, in blocking_read
    frame = self.transport.read_frame()
[2018-06-14 20:43:29,910: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/transport.py", line 243, in read_frame
    frame_header = read(7, True)
[2018-06-14 20:43:29,911: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/transport.py", line 418, in _read
    s = recv(n - len(rbuf))
[2018-06-14 20:43:29,911: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 380, in recv
    return _socket.socket.recv(self._sock, *args)
[2018-06-14 20:43:29,912: WARNING/MainProcess] ConnectionResetError: [Errno 104] Connection reset by peer
[2018-06-14 20:43:29,912: WARNING/MainProcess] 2018-06-14T20:43:29Z
[2018-06-14 20:43:29,913: WARNING/MainProcess] <built-in method switch of gevent.__greenlet_primitives.TrackedRawGreenlet object at 0x7f791830ce88> failed with ConnectionResetError

Juga, saya dapat mengonfirmasi bahwa saya melihat perilaku yang sama pada versi pengembangan terbaru dari semua pustaka, menggunakan Dockerfile ini:

FROM alpine

RUN apk add --no-cache build-base python3 python3-dev
RUN pip3 install https://github.com/celery/celery/zipball/master#egg=celery https://github.com/celery/billiard/zipball/master#egg=billiard https://github.com/celery/py-amqp/zipball/master#egg=amqp https://github.com/celery/kombu/zipball/master#egg=kombu https://github.com/celery/vine/zipball/master#egg=vine eventlet gevent

CMD celery -b amqp://rabbitmq worker -P eventlet --loglevel=DEBUG

Ini cocok dengan apa yang kita alami. Dalam satu antrian, kami memiliki banyak pekerja task_acks_late = True dan worker_prefetch_multiplier = 1 dan kami memiliki campuran tugas jangka panjang dan tugas pendek.

Beralih kembali ke Celery 3.x membuat masalah ini "hilang"

FYI

Pada awalnya, percobaan ulang tidak berfungsi saat menggunakan seledri 4.1.0 dan kombu 4.1.0. Setelah memperbarui masing-masing ke 4.2.0 dan 4.2.1, percobaan ulang mulai berfungsi tetapi pesan waktu tunggu yang sama mulai muncul dan tugas tampaknya dikirim dengan benar tetapi tidak pernah diproses oleh pekerja. Kami menggunakan prefork

-amqp==2.2.2
+amqp==1.4.9
-billiard==3.5.0.3
+billiard==3.3.0.23
-celery==4.2.0
+celery==3.1.23
-kombu==4.2.1
+kombu==3.0.34
-pyramid-celery==3.0.0
+pyramid-celery==2.0.0

Koneksi pialang menggunakan pengaturan detak jantung dari file konfigurasi aplikasi secara default (sejak versi 4.2.0 , # 4148).

Konfigurasi dan default - Dokumentasi

broker_heartbeat

transportasi yang didukung:

• pyamqp

Default: 120.0 (dinegosiasikan oleh server).

Anda dapat mencoba menyetel broker_heartbeat=0 . Semoga ini membantu.

@ y0ngdi Saya menetapkan broker_heartbeat=0 , ada pengurangan kesalahan, tetapi masih ada.

amqp==2.3.2
celery==4.2.0
gevent==1.3.4
greenlet==0.4.13
kombu==4.2.0

pekerja melaporkan beberapa kesalahan dengan Traceback:

[2018-06-22 09:41:21,467: INFO/MainProcess] sync with celery<strong i="11">@device_producer_tasks</strong>
[2018-06-22 09:41:21,468: ERROR/MainProcess] Control command error: error(32, 'Broken pipe')
Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/celery/worker/pidbox.py", line 46, in on_message
    self.node.handle_message(body, message)
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/amqp/channel.py", line 1732, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/home/ubuntu/.local/share/virtualenvs/backend-uRCQ3Clv/local/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
    self._write(s)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe

Saya mengalami masalah ini dengan seledri 4.2 dan eventlet tetapi tidak ada solusi sempurna yang ditemukan

Saya melakukan sedikit investigasi. Seledri memang membuat koneksi baru tetapi tidak memanggil heartbeat_check untuk itu. Ini menyebabkan server RabbitMQ menutup koneksi. Saya mengalami kesalahan ini dengan koneksi perintah (mis. stats , ping , ..)

Contoh minimal:

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery(
    'myapp',
    broker='amqp://guest@localhost//',
)
app.conf.broker_heartbeat = 5


if __name__ == '__main__':
    app.start()

Jalankan contohnya

$ python3 example.py worker -l DEBUG

RabbitMQ log

2018-06-27 19:49:41.109 [info] <0.471.1> accepting AMQP connection <0.471.1> (127.0.0.1:42744 -> 127.0.0.1:5672)
2018-06-27 19:49:41.111 [info] <0.471.1> connection <0.471.1> (127.0.0.1:42744 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2018-06-27 19:49:41.116 [info] <0.479.1> accepting AMQP connection <0.479.1> (127.0.0.1:42746 -> 127.0.0.1:5672)
2018-06-27 19:49:41.118 [info] <0.479.1> connection <0.479.1> (127.0.0.1:42746 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2018-06-27 19:49:41.131 [info] <0.500.1> accepting AMQP connection <0.500.1> (127.0.0.1:42748 -> 127.0.0.1:5672)
2018-06-27 19:49:41.133 [info] <0.500.1> connection <0.500.1> (127.0.0.1:42748 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2018-06-27 19:49:56.135 [warning] <0.500.1> closing AMQP connection <0.500.1> (127.0.0.1:42748 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 5s

Mencoba mengirim perintah setelah koneksi ditutup oleh RabbitMQ:

$ celery inspect ping

Log pekerja seledri mencetak ini ketika menerima perintah:

[2018-06-27 19:51:25,638: DEBUG/MainProcess] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '5a7fe1f1-be67-397f-879c-d939ea3c076e'} ticket:d80183f3-a236-4057-841b-6b8cd2926917]
[2018-06-27 19:51:25,639: ERROR/MainProcess] Control command error: ConnectionResetError(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/celery/worker/pidbox.py", line 46, in on_message
    self.node.handle_message(body, message)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/messaging.py", line 194, in _publish
    [maybe_declare(entity) for entity in declare]
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/messaging.py", line 194, in <listcomp>
    [maybe_declare(entity) for entity in declare]
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/messaging.py", line 102, in maybe_declare
    return maybe_declare(entity, self.channel, retry, **retry_policy)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/common.py", line 129, in maybe_declare
    return _maybe_declare(entity, declared, ident, channel, orig)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/common.py", line 135, in _maybe_declare
    entity.declare(channel=channel)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/kombu/entity.py", line 185, in declare
    nowait=nowait, passive=passive,
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/amqp/channel.py", line 614, in exchange_declare
    wait=None if nowait else spec.Exchange.DeclareOk,
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/home/bar/Desktop/foo/virtualenv/lib/python3.6/site-packages/amqp/transport.py", line 275, in write
    self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer
[2018-06-27 19:51:25,647: DEBUG/MainProcess] Closed channel #2
[2018-06-27 19:51:25,647: DEBUG/MainProcess] using channel_id: 2
[2018-06-27 19:51:25,648: DEBUG/MainProcess] Channel open

Kemungkinan penyebab kesalahan

Saya pikir ada koneksi yang diakhiri setelah dari sisi server karena kombu.connection.Connection.heartbeat_check() tidak dipanggil berulang kali pada instance.

Saya melihat masalah yang sama dengan gevent.

Solusi dengan broker_heartbeat = 0 berhasil untuk saya.

Versi yang diinstal:

amqp==2.2.2
celery==4.2.0
eventlet==0.23.0
kombu==4.2.1

Pekerja:
celery -A celery_app worker --loglevel=info -P eventlet

PS Diuji pada Windows 10, dengan Python 2 & 3.

@auvipy : Mengapa ditutup? Masalahnya belum diperbaiki ...

apakah solusi dari @ stojan-jovic berhasil?

Sepertinya ya, tetapi solusinya bukanlah perbaikan bug.
Dan itu masih menyebabkan banyak waktu terbuang percuma untuk mencari solusi oleh lebih banyak orang setiap hari.

jika Anda menemukan kemungkinan perbaikan maka itu akan dibuka kembali.

Solusi dengan broker_heartbeat = 0 tampaknya berhasil untuk saya untuk seledri 4.2.0 dan gevent 1.2.2 (python 2) / RabbitMQ 3.7.8. Setidaknya itu tidak menutup para pekerja pada batas waktu default 60-an untuk saya.

Django 1.8.4, Python 2.7.12
Masalah muncul setelah peningkatan dari 3.1.25 ke 4.2.1
CELERY_BROKER_HEARTBEAT = 0 di settings.py menyelesaikan ini.

@auvipy kenapa kamu menutup ini? Masalahnya masih ada.

@djlambert Dia dikenal menutup masalah / prs tanpa alasan.

Ada pembaruan tentang ini? Solusi bukanlah perbaikan bug.

jangan ragu untuk menyelidiki masalah @gvdmarck saat Anda menghadapi ini, akan lebih mudah bagi Anda untuk mengetahui akar masalahnya.

Django 1.8.4, Python 2.7.12
Masalah muncul setelah peningkatan dari 3.1.25 ke 4.2.1
CELERY_BROKER_HEARTBEAT = 0 di settings.py menyelesaikan ini.

seharusnya BROKER_HEARTBEAT bukan CELERY_BROKER_HEARTBEAT?

amqp == 2.2.2

Saya mencoba menjalankan pekerja seledri dengan konfigurasi ini, tetapi pekerja seledri tidak memulai.
Perintah yang saya gunakan adalah sebagai berikut
pipenv menjalankan pekerja seledri -A src.celery_app -l debug -P eventlet
Tetapi saya tidak melihat log startup seledri, atau apa pun yang langsung kembali ke prompt windows cmd.

@ stojan-jovic, dapatkah Anda memberi tahu saya cara mengatur lingkungan untuk seledri.