Celery: Tidak dapat menjawab, alasan: BrokenPipeError (32, 'Pipa rusak')

Dibuat pada 19 Jan 2017 · 83Komentar · Sumber: celery/celery

Daftar Periksa

[X] Saya telah memasukkan keluaran dari celery -A proj report dalam terbitan.
(jika Anda tidak dapat melakukan ini, setidaknya tentukan Celery
versi terpengaruh).

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

[X] Saya telah memverifikasi bahwa masalah tersebut ada pada cabang Celery master .

Langkah-langkah untuk mereproduksi

Tingkatkan ke Celery 4.0.2.

Perilaku yang diharapkan

Tidak ada kesalahan.

Perilaku sebenarnya

Mendapatkan kesalahan ini dari pekerja saya:

[2017-01-19 04:07:57,411: CRITICAL/MainProcess] Couldn't ack 461, reason:BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
  File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 130, in ack_log_error
    self.ack(multiple=multiple)
  File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 125, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/channel.py", line 1408, in basic_ack
    spec.Basic.Ack, argsig, (delivery_tag, multiple),
  File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/abstract_channel.py", line 64, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/method_framing.py", line 174, in write_frame
    write(view[:offset])
  File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/transport.py", line 269, in write
    self._write(s)
BrokenPipeError: [Errno 32] Broken pipe

Hal ini menyebabkan pekerja menyetel ulang koneksinya ke RabbitMQ setiap beberapa detik, dan terkadang menyebabkan mencapai batas koneksi maksimal yang diizinkan di server RabbitMQ.

RabbitMQ Broker HAProxy Bug Report Critical

Sumber

alanhamlett

👍38

Komentar yang paling membantu

@auvipy tolong buka kembali. Ini adalah bug ... peretasan tidak memenuhi syarat sebagai perbaikan.

alanhamlett pada 22 Des 2017

👍13

Semua 83 komentar

Mungkin terkait dengan # 3328 dan # 3377.

alanhamlett pada 19 Jan 2017

Tidak yakin apa yang menyebabkan ini, tetapi itu mulai terjadi setiap satu atau dua hari. Memulai ulang pekerja akan memperbaikinya untuk sementara waktu.

alanhamlett pada 25 Jan 2017

👍7

Saya juga melihat masalah ini, meskipun pada versi seledri yang sedikit lebih tua.

software -> celery:3.1.23 (Cipater) kombu:3.0.35 py:3.5.3
            billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:djcelery.backends.database:DatabaseBackend

chrisseto pada 14 Feb 2017

alfredn pada 1 Mar 2017

+1 melihat masalah ini akhir-akhir ini di seledri 3.2.1

azhard4int pada 17 Mar 2017

+1 Melihat masalah ini belakangan ini di 3.1.18

noorul pada 23 Mar 2017

+1 Melihat masalah di 4.0.2

hmphu pada 16 Mei 2017

+1 Melihat masalah di 4.0.2

Sebubu pada 29 Mei 2017

+1 Melihat masalah di 4.0.2

MarcoGlauser pada 1 Jun 2017

+1 Melihat masalah di 4.0.2

lpig pada 2 Jun 2017

+1 melihat masalah ini di 4.0.2

sjitendra pada 12 Jun 2017

FYI, masalah ini dan Celery 4.x lainnya telah diperbaiki ketika saya diturunkan ke 3.x. Solusi jangka panjang adalah beralih ke https://github.com/Bogdanp/dramatiq , yang memiliki sintaks yang sama dengan Celery .delay() sehingga sangat mudah untuk beralih.

alanhamlett pada 15 Jun 2017

👍1

+1 lihat di 4.0.2

gordania pada 27 Jun 2017

+1 lihat di 4.0.2

markkohdev pada 28 Jun 2017

+1 lihat di 4.1.0

hqtoan94 pada 3 Agu 2017

👍2

+1 lihat di 4.1.0

donhoff pada 10 Agu 2017

👍2

+1 melihat di seledri (4.1.0), gevent

leobuskin pada 22 Agu 2017

+1 seledri 4.1.0

elephanter pada 25 Agu 2017

nrg1984 pada 8 Sep 2017

Melihat ini di 4.1.0 dengan broker kelincimq

vesterbaek pada 9 Okt 2017

+1
Terjadi saat pencabutan tugas dengan terminate argumen: app.control.revoke(t.id, terminate=True)
Sama saat menghentikan tugas dari Flower.

Python 3.6.2rc2
Celery 4.1.0 (latentcall)
  broker: rabbitmq:3.6.12
  backend: redis:3.2.10

[2017-10-11 09:31:49,471: ERROR/MainProcess] Control command error: BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/celery/worker/pidbox.py", line 42, in on_message
    self.node.handle_message(body, message)
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/channel.py", line 1734, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/transport.py", line 258, in write
    self._write(s)
BrokenPipeError: [Errno 32] Broken pipe

gustavorps pada 11 Okt 2017

+1 seledri 4.0.0

fanhaipeng0403 pada 8 Nov 2017

lihat penjelasan terakhir dari @mihajenko https://github.com/celery/celery/issues/3377

auvipy pada 20 Des 2017

👎11

+1 seledri (4.0.2)

Traceback (most recent call last):
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 130, in ack_log_error
    self.ack(multiple=multiple)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 125, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/channel.py", line 1408, in basic_ack
    spec.Basic.Ack, argsig, (delivery_tag, multiple),
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame
    write(view[:offset])
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/transport.py", line 269, in write
    self._write(s)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 397, in sendall
    tail = self.send(data, flags)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 391, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 378, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe

nixawk pada 22 Des 2017

@auvipy tolong buka kembali. Ini adalah bug ... peretasan tidak memenuhi syarat sebagai perbaikan.

alanhamlett pada 22 Des 2017

👍13

@auvipy Bisakah Anda memberikan perbaikan cepat? Saya tidak begitu paham dengan seledri. Terima kasih!

Neal-Wang pada 26 Jan 2018

+1 4.4

dpanic pada 27 Jan 2018

+1 @ _ @

hpulfc pada 29 Mar 2018

burakbostancioglu pada 6 Apr 2018

daripada memberi +1, harap luangkan waktu Anda untuk menyelidiki masalah terkait sehingga Anda dapat menemukan solusi yang memungkinkan. Akan lebih bagus lagi jika ada yang bisa melaporkan masalah tersebut masih bertahan di 4.2rc2?

auvipy pada 7 Apr 2018

@auvipy Kami telah menyelidiki masalah ini, kami memiliki seledri yang agak besar + infra kelincimq. Kami hari ini beralih dari 3.1.25 ke patch 4.2. Kami mulai melihat% 4 dari penurunan tugas. Kesalahan ini bertepatan dengan hilangnya / kehilangan tugas. Pernahkah Anda menghadapi / mendengar masalah serupa di masa transisi?

burakbostancioglu pada 7 Apr 2018

dapatkah Anda terlebih dahulu mencoba https://github.com/celery/celery/releases/tag/v3.1.26 secara lokal kemudian meningkatkan ke 4.2rc2? kami mendapat laporan dari zapier kemudian mereka pindah dari 3.1 ke 4.1, hal lain adalah meskipun seledri 4.2rc2 di pypi, dependensinya belum ada di pypi. Jika Anda bisa meluangkan waktu untuk memperbarui dependensi dari cabang master dengan 4.2rc2 dan melihat apa yang terjadi dan memberi tahu kami, itu akan sangat bagus. setelah itu kami dapat menandainya sebagai prioritas tinggi / pemblokir untuk rilis final 4.2.

auvipy pada 7 Apr 2018

juga saya meminta untuk mengangkat masalah ini di milis

auvipy pada 7 Apr 2018

Maaf @auvipy , hanya untuk menjelaskan: Anda ingin kami mencoba kode @ 4.2rc2 dari pypi, tetapi dengan dependensi yang diperbarui agar sesuai dengan daftar dependensi di cabang master?

rickhutcheson pada 7 Apr 2018

tidak semua kecuali jika ada, seledri 4.2rc2 dari pypi, dan seledri ketergantungan lainnya dari github master brnach di server lokal / pementasan

auvipy pada 7 Apr 2018

Saya bekerja dengan @burakbostancioglu , jadi ya, kita bisa coba ini =]

rickhutcheson pada 7 Apr 2018

👍1

Terima kasih :)

auvipy pada 7 Apr 2018

ada update guys? menggunakan 4.2rc4?

auvipy pada 27 Mei 2018

Saya melihat ini terjadi cukup teratur pada Celery 4.2.0 ketika sedang dimuat (menguji migrasi 3.1.25 -> 4.2.0, tetapi ini memblokirnya). Saya bukan monyet menambal greenlets. RMQ sebagai broker, redis sebagai hasil backend.

Shookit pada 21 Jun 2018

Kami menggali terlalu keras untuk migrasi jika kami dapat menghindarinya, tetapi saya menghargai tipnya :-)

Beri tahu saya jika ada sesuatu yang dapat saya gali terkait kesalahan ini.

Shookit pada 22 Jun 2018

Saya menerapkan perbaikan yang disebutkan dalam
https://github.com/celery/celery/issues/4226 dan itu memecahkan masalah bagi saya.

ewjoachim pada 20 Jul 2018

🎉1

Jadi apakah masalah ini teratasi atau dengan cara diselesaikan? Semua alur kerja saya bergantung pada seledri dan saya sangat menikmati alat ini.

Jika bug sedang diatasi, apakah ada quickfick sebelum resolusi sebenarnya?

lauevrar77 pada 29 Jul 2018

👍2

@xirdneh jika Anda dapat menyelidiki masalah terkait kapan pun Anda bisa.

auvipy pada 10 Agu 2018

Saya menetapkan --concurrency statis daripada --autoscale dan ini memecahkan masalah untuk saya.

oneor0 pada 16 Agu 2018

(Saya memiliki bug dengan konkurensi 1 dan tidak ada skala otomatis)

ewjoachim pada 16 Agu 2018

👍4

Masih mengalami masalah yang sama dengan seledri 4.2.1. Semoga satu laporan lagi bisa membantu.

software -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:2.7.15rc1
            billiard:3.5.0.4 redis:2.10.6
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379/0

broker_url: u'redis://127.0.0.1:6379/0'
result_backend: u'redis://127.0.0.1:6379/0'
task_routes: {
    u'celery.crawler.fbk.download_alt_text': {   u'queue': u'fbk_alt_text'}}

sp1thas pada 25 Okt 2018

Adakah yang mau menguji dengan master seledri?

auvipy pada 6 Mei 2019

ini masih ada.
default celery-67c7fff78b-g48vx celery Couldn't ack 23, reason:BrokenPipeError(32, 'Broken pipe') default celery-67c7fff78b-g48vx celery Traceback (most recent call last): default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 130, in ack_log_error default celery-67c7fff78b-g48vx celery self.ack(multiple=multiple) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 125, in ack default celery-67c7fff78b-g48vx celery self.channel.basic_ack(self.delivery_tag, multiple=multiple) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 1399, in basic_ack default celery-67c7fff78b-g48vx celery spec.Basic.Ack, argsig, (delivery_tag, multiple), default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 51, in send_method default celery-67c7fff78b-g48vx celery conn.frame_writer(1, self.channel_id, sig, args, content) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/method_framing.py", line 172, in write_frame default celery-67c7fff78b-g48vx celery write(view[:offset]) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 288, in write default celery-67c7fff78b-g48vx celery self._write(s) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 458, in sendall default celery-67c7fff78b-g48vx celery return _socketcommon._sendall(self, data_memory, flags) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 374, in _sendall default celery-67c7fff78b-g48vx celery timeleft = __send_chunk(socket, chunk, flags, timeleft, end) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk default celery-67c7fff78b-g48vx celery data_sent += socket.send(chunk, flags) default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 439, in send default celery-67c7fff78b-g48vx celery return _socket.socket.send(self._sock, data, flags) default celery-67c7fff78b-g48vx celery BrokenPipeError: [Errno 32] Broken pipe

jagguli pada 5 Jun 2019

😕1 👍1

sepertinya itu masalah gevent? ile "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", baris 439, sedang dikirim
default celery-67c7fff78b-g48vx celery return _socket.socket.send (self._sock, data, flags)
default celery-67c7fff78b-g48vx seledri BrokenPipeError: [Errno 32] Pipa rusak

auvipy pada 7 Jun 2019

seledri 4.4.0rc1?

auvipy pada 7 Jun 2019

dicoba di cabang master dengan tambalan yang diusulkan di https://github.com/celery/celery/issues/3377

masih macet setelah satu jam operasi

jagguli pada 12 Jun 2019

+1 seledri 4.2.1

aeciovc pada 2 Agu 2019

@auvipy Saya telah melihat masalah ini dengan Celery 4.3.0 + gevent + RabbitMQ. Menggunakan strace Saya menemukan bahwa masalah terkait dengan recvfrom dan epoll_wait syscall. Tampaknya soket tidak dapat menerima pesan dari RabbitMQ dan jatuh ke loop tak terbatas, proses Celery macet di sana.

26738 adalah ID proses Celery

 sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1)   = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1)   = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

Informasi selengkapnya tentang inode 5, 15, 21:

$ sudo ls -la /proc/26738/fd/5
lrwx------ 1 xxx xxx 64 Aug  2 06:05 /proc/26738/fd/5 -> socket:[99157475]
$ sudo ls -la /proc/26738/fd/21
lrwx------ 1 xxx xxx 64 Aug  2 06:05 /proc/26738/fd/21 -> socket:[99144296]
$ sudo ls -la /proc/26738/fd/15
lrwx------ 1 xxx xxx 64 Aug  2 06:05 /proc/26738/fd/15 -> anon_inode:[eventpoll]

$ sudo lsof -p 26738 | grep 99157475
celery  26738 xxx    5u     IPv4 99157475      0t0      TCP xxx-1084:50954->rabbit.xxx-1084:amqp (ESTABLISHED)

$ sudo lsof -p 26738 | grep 99144296
celery  26738 xxx   21u     IPv4 99144296      0t0      TCP xxx-1084:38194->rabbit.xxx-1084:amqp (ESTABLISHED)

$ sudo head -n1 /proc/26738/net/tcp; grep -a 99157475 /proc/26738/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
  10: 8A01010A:C70A 5E00010A:1628 01 00000000:00000000 02:00000351 00000000  1005        0 99157475 2 0000000000000000 20 4 30 10 -1

Semoga membantu.

duydo pada 14 Agu 2019

eugene-s pada 25 Mar 2020

coweos pada 30 Mar 2020

@duydo Saya menemui situasi yang sama dengan Anda. Dan rabbitMQ tidak bisa menjawab apa pun.

Traceback (most recent call last):
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 130, in ack_log_error
    self.ack(multiple=multiple)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 125, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/channel.py", line 1399, in basic_ack
    spec.Basic.Ack, argsig, (delivery_tag, multiple),
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/abstract_channel.py", line 51, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/transport.py", line 288, in write
    self._write(s)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 458, in sendall
    return _socketcommon._sendall(self, data_memory, flags)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 374, in _sendall
    timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk
    data_sent += socket.send(chunk, flags)
  File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 439, in send
    return _socket.socket.send(self._sock, data, flags)
BrokenPipeError: [Errno 32] Broken pipe

Saya menggunakan strace untuk melacak PID
$ sudo strace -p 27977 -f

[pid 27977] getpid()                    = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108490973}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108664276}) = 0
[pid 27977] epoll_wait(4, [], 64, 1999) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=108549437}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109251966}) = 0
[pid 27977] getpid()                    = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109598283}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109777281}) = 0
[pid 27977] epoll_wait(4, [], 64, 1529) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=640754845}) = 0
[pid 27977] getpid()                    = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641809883}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641940654}) = 0
[pid 27977] epoll_wait(4, [], 64, 467)  = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=109901002}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=110880127}) = 0
[pid 27977] getpid()                    = 27977

$ sudo lsof -p 27977|grep 1651852686

python  27977 ymserver   10u     IPv4 1651852686      0t0        TCP vuljp-ag-proxy-01:33006->10.55.4.70:amqp (ESTABLISHED)

esingtse pada 16 Apr 2020

👍1

https://github.com/celery/celery/pull/5581/files coba ini

auvipy pada 16 Apr 2020

+1 seledri v4.4.2

eltix pada 30 Apr 2020

+1 seledri 4.3.0

grayguest pada 8 Mei 2020

+1 seledri 4.3.0

malikfaiq pada 9 Jun 2020

daripada +1 spam silakan periksa PR terkait & tingkatkan dengan tes unit.

auvipy pada 9 Jun 2020

1 untuk seledri 4.4.6., Tidak masalah dengan python2 atau python3 env.

software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:2.7.10
            billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Darwin arch:64bit
            kernel version:17.7.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:redis://10.28.218.6/

Log ada di sini:

worker: Warm shutdown (MainProcess)
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Hub...
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Pool...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Closing Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Connection...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Events...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Heart...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Control...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping Control...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Closed channel #3
[2020-07-17 22:27:52,153: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:27:53,279: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2020-07-17 22:27:53,280: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] Closed channel #2
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Events...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2020-07-17 22:27:53,843: DEBUG/MainProcess] | Worker: Stopping Pool...
^@^@[2020-07-17 22:31:07,678: INFO/ForkPoolWorker-1] Task tasks.handle_compressed[5e493207-a3c5-422e-ab82-1f382342273c] succeeded in 200.757257249s: {'name': 'ray'}
[2020-07-17 22:31:09,188: DEBUG/MainProcess] | Worker: Stopping Hub...
[2020-07-17 22:31:09,189: CRITICAL/MainProcess] Couldn't ack 1, reason:error(32, 'Broken pipe')
Traceback (most recent call last):
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 131, in ack_log_error
    self.ack(multiple=multiple)
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 126, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py", line 1394, in basic_ack
    spec.Basic.Ack, argsig, (delivery_tag, multiple),
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py", line 59, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py", line 305, in write
    self._write(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2020-07-17 22:31:09,195: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Closing consumer channel...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2020-07-17 22:31:09,431: DEBUG/MainProcess] Closed channel #1
[2020-07-17 22:31:09,663: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2020-07-17 22:31:09,665: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

crook pada 17 Jul 2020

FYI. nonaktifkan broker_heartbeat (broker_heartbeat = 0) dapat memperbaiki masalah saya. Tidak yakin ini praktik yang baik.

crook pada 17 Jul 2020

Semoga info triase ini membantu analisis kasus akar.

  > ps -ef | grep worker
  502 92665 75622   0 10:27PM ttys008    0:00.78 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
  502 92672 92665   0 10:27PM ttys008    0:00.01 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
  > sudo dtruss -p 75622 -f

......
92665/0xf94bb7:  fcntl(0x5, 0x4, 0x4)        = 0 0
92665/0xf94bb7:  wait4(0x16A00, 0x7FFEEB4B0C10, 0x1)         = 0 0
92665/0xf94bb7:  wait4(0x16A00, 0x7FFEEB4B13E0, 0x0)         = 92672 0
92665/0xf94bb7:  close(0xB)      = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  sendto(0xD, 0x1068AF000, 0x15)      = -1 Err#32
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF280, 0x0)         = 0 0
92665/0xf94bb7:  open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x0, 0x1B6)       = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF680, 0x0)         = 0 0
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x7FFEEB4AF280, 0x0)      = 0 0
92665/0xf94bb7:  open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x0, 0x1B6)        = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x7FFEEB4AF280, 0x0)         = 0 0
92665/0xf94bb7:  open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x0, 0x1B6)       = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x7FFEEB4AF280, 0x0)       = 0 0
92665/0xf94bb7:  open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x0, 0x1B6)         = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x7FFEEB4AF280, 0x0)        = 0 0
92665/0xf94bb7:  open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x0, 0x1B6)      = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  mmap(0x0, 0x40000, 0x3, 0x1002, 0xFFFFFFFFFFFFFFFF, 0x0)        = 0x1069D1000 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  stat64("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x7FFEEB4AF280, 0x0)       = 0 0
92665/0xf94bb7:  open_nocancel("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x0, 0x1B6)         = 11 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AF0F8, 0x0)       = 0 0
92665/0xf94bb7:  fstat64(0xB, 0x7FFEEB4AD098, 0x0)       = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  sendto(0xE, 0x1068F0000, 0x13)      = 19 0
92665/0xf94bb7:  recvfrom(0xE, 0x1069970E4, 0x7)         = 7 0
92665/0xf94bb7:  recvfrom(0xE, 0x1069970B4, 0x4)         = 4 0
92665/0xf94bb7:  recvfrom(0xE, 0x10676348C, 0x1)         = 1 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  madvise(0x1068CF000, 0x21000, 0x9)      = 0 0
92665/0xf94bb7:  sendto(0xE, 0x1068F0000, 0x13)      = 19 0
92665/0xf94bb7:  recvfrom(0xE, 0x1069970E4, 0x7)         = 7 0
92665/0xf94bb7:  recvfrom(0xE, 0x1069970B4, 0x4)         = 4 0
92665/0xf94bb7:  recvfrom(0xE, 0x10676348C, 0x1)         = 1 0
92665/0xf94bb7:  shutdown(0xE, 0x2, 0x0)         = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  sendto(0xD, 0x1068AF000, 0x13)      = -1 Err#32
92665/0xf94bb7:  sendto(0xD, 0x1068AF000, 0x13)      = -1 Err#32
92665/0xf94bb7:  shutdown(0xD, 0x2, 0x0)         = -1 Err#57
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  select(0x4, 0x7FFEEB4A6E58, 0x7FFEEB4A6ED8, 0x7FFEEB4A6F58, 0x7FFEEB4A6FD8)         = 0 0
92665/0xf94bb7:  fcntl(0x4, 0x3, 0x0)        = 1 0
92665/0xf94bb7:  fcntl(0x4, 0x4, 0x1)        = 0 0
dtrace: error on enabled probe ID 2172 (ID 161: syscall::write:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7:  lstat64("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x7FFEEB4B34F0, 0x0)      = 0 0
92665/0xf94bb7:  open_nocancel("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x1100004, 0x1213200)       = 11 0
92665/0xf94bb7:  fstatfs64(0xB, 0x7FFEEB4B2DB8, 0x0)         = 0 0
92665/0xf94bb7:  getdirentries64(0xB, 0x7FBE640A0A00, 0x1000)        = 64 0
92665/0xf94bb7:  getdirentries64(0xB, 0x7FBE640A0A00, 0x1000)        = 0 0
92665/0xf94bb7:  close_nocancel(0xB)         = 0 0
92665/0xf94bb7:  rmdir(0x7FBE61640B10, 0x0, 0x0)         = 0 0
92665/0xf94bb7:  sigaction(0x2, 0x7FFEEB4B4548, 0x7FFEEB4B4588)      = 0 0
92665/0xf94bb7:  sigaction(0x3, 0x7FFEEB4B4548, 0x7FFEEB4B4588)      = 0 0
92665/0xf94bb7:  sigaction(0xF, 0x7FFEEB4B4548, 0x7FFEEB4B4588)      = 0 0
92665/0xf94bb7:  sigaction(0x1E, 0x7FFEEB4B4548, 0x7FFEEB4B4588)         = 0 0
92665/0xf94bb7:  madvise(0x105752000, 0x3C000, 0x9)      = -1 Err#22
92665/0xf94bb7:  close(0xD)      = 0 0
92665/0xf94bb7:  madvise(0x1068AF000, 0x20000, 0x9)      = 0 0
92665/0xf94bb7:  madvise(0x106A11000, 0x23000, 0x9)      = -1 Err#22
92665/0xf94bb7:  close(0xE)      = 0 0
92665/0xf94bb7:  madvise(0x1068F0000, 0x20000, 0x9)      = 0 0
92665/0xf94bb7:  madvise(0x10486D000, 0x30000, 0x9)      = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C25000, 0x5000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C81000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C83000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C85000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C87000, 0x2000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C8A000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C8F000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C92000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C99000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62C9E000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CA1000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CA3000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CA6000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CAB000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CAE000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CB2000, 0x2000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CB6000, 0x4000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CBB000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CBE000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CEA000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  madvise(0x7FBE62CFB000, 0x1000, 0x7)        = 0 0
92665/0xf94bb7:  close(0x6)      = 0 0
dtrace: error on enabled probe ID 2366 (ID 897: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
^@^@^@^@^@^C





  ranc-a01:~ ranc$ lsof -p 92665
COMMAND   PID USER   FD     TYPE             DEVICE   SIZE/OFF     NODE NAME
python  92665 ranc  cwd      DIR                1,4       2176 12342436 /Users/ranc/vsan_vsancertification/tools/graphrunner
python  92665 ranc  txt      REG                1,4      51744 12719059 /Users/ranc/p2dev/bin/python
python  92665 ranc  txt      REG                1,4      52768   378470 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
python  92665 ranc  txt      REG                1,4      84240   378488 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
python  92665 ranc  txt      REG                1,4      83232   378468 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
python  92665 ranc  txt      REG                1,4      60192   378500 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
python  92665 ranc  txt      REG                1,4      71392   378444 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
python  92665 ranc  txt      REG                1,4      85136   378540 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
python  92665 ranc  txt      REG                1,4     108192   378532 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
python  92665 ranc  txt      REG                1,4      59984   378462 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
python  92665 ranc  txt      REG                1,4      48256   378458 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
python  92665 ranc  txt      REG                1,4     253840   378466 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
python  92665 ranc  txt      REG                1,4      73456   378552 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
python  92665 ranc  txt      REG                1,4     200880   378448 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
python  92665 ranc  txt      REG                1,4      65952   378558 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
python  92665 ranc  txt      REG                1,4      88816   378494 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
python  92665 ranc  txt      REG                1,4      78896   378534 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
python  92665 ranc  txt      REG                1,4      58240   378460 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
python  92665 ranc  txt      REG                1,4      52656   378478 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
python  92665 ranc  txt      REG                1,4      57040   378508 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
python  92665 ranc  txt      REG                1,4      74592   378446 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_csv.so
python  92665 ranc  txt      REG                1,4      43536   378526 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
python  92665 ranc  txt      REG                1,4      65184   378562 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
python  92665 ranc  txt      REG                1,4     144736   378482 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
python  92665 ranc  txt      REG                1,4     172352   378486 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
python  92665 ranc  txt      REG                1,4      44752   378480 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
python  92665 ranc  txt      REG                1,4      43708 12722902 /Users/ranc/p2dev/lib/python2.7/site-packages/_scandir.so
python  92665 ranc  txt      REG                1,4     152272   378514 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
python  92665 ranc  txt      REG                1,4      65920   378550 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
python  92665 ranc  txt      REG                1,4      52048   378520 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
python  92665 ranc  txt      REG                1,4      67920   378476 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
python  92665 ranc  txt      REG                1,4     146112   378506 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
python  92665 ranc  txt      REG                1,4      52048   378548 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/resource.so
python  92665 ranc  txt      REG                1,4      84000   378504 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/bz2.so
python  92665 ranc  txt      REG                1,4      43104   378430 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
python  92665 ranc  txt      REG                1,4     104256   378544 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so
python  92665 ranc  txt      REG                1,4      25512 15160994 /Users/ranc/p2dev/lib/python2.7/site-packages/tornado/speedups.so
python  92665 ranc  txt      REG                1,4     138464   378452 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
python  92665 ranc  txt      REG                1,4    1409984   378560 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/unicodedata.so
python  92665 ranc  txt      REG                1,4      27108 12722492 /Users/ranc/p2dev/lib/python2.7/site-packages/markupsafe/_speedups.so
python  92665 ranc  txt      REG                1,4      43588 14636845 /Users/ranc/p2dev/lib/python2.7/site-packages/_billiard.so
python  92665 ranc  txt      REG                1,4      65776   378536 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/mmap.so
python  92665 ranc  txt      REG                1,4       4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python  92665 ranc  txt      REG                1,4     841456  2716638 /usr/lib/dyld
python  92665 ranc  txt      REG                1,4 1172373504 13558190 /private/var/db/dyld/dyld_shared_cache_x86_64h
python  92665 ranc    0u     CHR               16,8  0t1188168     1405 /dev/ttys008
python  92665 ranc    1u     CHR               16,8  0t1188168     1405 /dev/ttys008
python  92665 ranc    2u     CHR               16,8  0t1188168     1405 /dev/ttys008
python  92665 ranc    3     PIPE 0x75a36a27dcf37cad      16384          ->0x75a36a27dcf355ad
python  92665 ranc    4     PIPE 0x75a36a27dcf355ad      16384          ->0x75a36a27dcf37cad
python  92665 ranc    5     PIPE 0x75a36a27dcf3746d      16384          ->0x75a36a27dcf36dad
python  92665 ranc    6r     CHR               14,1    0t15192      584 /dev/urandom
python  92665 ranc    7     PIPE 0x75a36a27dcf36dad      16384          ->0x75a36a27dcf3746d
python  92665 ranc    8u     REG                1,4       4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python  92665 ranc    9u     REG                1,4       4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python  92665 ranc   10r  PSXSEM                           0t0          /mp-r6SMtW
python  92665 ranc   11     PIPE 0x75a36a27dcf351ed      16384          ->0x75a36a27dcf36e6d
python  92665 ranc   12u   systm 0x75a36a27dd0002cd        0t0          [ctl com.apple.netsrc id 8 unit 47]
python  92665 ranc   13u    IPv4 0x75a36a280ae00f75        0t0      TCP 10.117.234.174:62477->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python  92665 ranc   14u    IPv4 0x75a36a27fae89f75        0t0      TCP 10.117.234.174:62480->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python  92665 ranc   15u    IPv4 0x75a36a27db0e88d5        0t0      TCP 10.117.234.174:62485->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python  92665 ranc   16     PIPE 0x75a36a27dcf351ed      16384          ->0x75a36a27dcf36e6d

crook pada 17 Jul 2020

ada yang menemukan versi tanpa bug ini w / gevent ?

json2d pada 23 Agu 2020

Saya melihat masalah ini (atau mungkin serupa) menggunakan eventlet .

kylemiller pada 26 Agu 2020

Halo semuanya!
1 untuk seledri 4.4.6.
Saya mengalami masalah ini hari ini) berkat komentar di atas dan penelitian yang cermat di server saya, saya menemukan masalah tersebut)
masalah di RabbitMQ dan Redis)
Untuk mengatasinya, kita perlu menginstal HAProxy untuk RabbitMQ dan Redis) sayangnya tanpa itu, mereka memutus koneksi dan kita mendapatkan kesalahan KRITIS ini) ada juga informasi tambahan di dokumentasi)
https://www.rabbitmq.com/networking.html#proxy -effects

CELERY_BROKER_URL - RabbitMQ
CELERY_RESULT_BACKEND - Redis

Saya juga mengkonfigurasi coba ulang seledri dan parameter lainnya sehingga tugas tersebut benar-benar dijalankan karena bagi saya data ini penting)
bekerja dengan dan tanpa gevent) Saya harap ini banyak membantu)

seledri bukanlah masalahnya!

RostislavBiloshapka pada 7 Okt 2020

👍4

seledri bukanlah masalahnya!

Ya, Celery adalah masalahnya, jika tidak, menurunkan ke Celery 3 tidak akan memperbaiki masalah.

alanhamlett pada 12 Okt 2020

@jamur_kejang
Telah bekerja dengan HAProxy selama lebih dari 5 hari dan semuanya baik-baik saja)
Saya mendapatkan kesalahan ini sepanjang waktu dan pekerja saya mogok dalam beberapa menit)

Saya mengerti bahwa ini bisa diperbaiki di seledri, tetapi perbaikannya bisa memakan waktu sangat lama) sulit membuat logika berfungsi seperti HAProxy)

RostislavBiloshapka pada 12 Okt 2020

dan itu tidak perlu) HAProxy menyelesaikan masalah ini dengan sempurna)

RostislavBiloshapka pada 12 Okt 2020

👍1

+1 Gejala yang sama seperti @duydo
Seledri 4.4.7

Tidak yakin bagaimana ini masih belum diperbaiki.

39315 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 534, in sendall 39316 return _socketcommon._sendall(self, data_memory, flags) 39317 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 392, in _sendall 39318 timeleft = __send_chunk(socket, chunk, flags, timeleft, end) 39319 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 321, in __send_chunk 39320 data_sent += socket.send(chunk, flags) 39321 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 515, in send 39322 return self._sock.send(data, flags) 39323 BrokenPipeError: [Errno 32] Broken pipe

getpid() = 6 epoll_wait(3, [{EPOLLIN, {u32=125, u64=266287972477}}], 262, 767) = 1 recvfrom(125, "\1\0\1\0\0\0#", 7, 0, NULL, NULL) = 7 recvfrom(125, "\0<\0<\6None12\0\0\0\0\0\0\4\220\0\rcelery.pidb"..., 35, 0, NULL, NULL) = 35 recvfrom(125, "\316", 1, 0, NULL, NULL) = 1 recvfrom(125, "\2\0\1\0\0\0G", 7, 0, NULL, NULL) = 7 recvfrom(125, "\0<\0\0\0\0\0\0\0\0\0c\370\0\20application/json\5"..., 71, 0, NULL, NULL) = 71 recvfrom(125, "\316", 1, 0, NULL, NULL) = 1 recvfrom(125, "\3\0\1\0\0\0c", 7, 0, NULL, NULL) = 7 recvfrom(125, "{\"method\": \"enable_events\", \"arg"..., 99, 0, NULL, NULL) = 99 recvfrom(125, "\316", 1, 0, NULL, NULL) = 1 getpid() = 6 write(2, "[2020-10-18 01:21:47,952: DEBUG/"..., 112) = 112 recvfrom(125, 0x7f8fbffe1078, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) getpid() = 6 epoll_wait(3, [], 262, 108) = 0 getpid() = 6 epoll_wait(3, [], 262, 805) = 0 recvfrom(6, 0x7f8fbffe1de8, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) getpid() = 6 epoll_wait(3, [], 262, 85) = 0 recvfrom(125, 0x7f8fbffe1d98, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) getpid() = 6 epoll_wait(3, ^Cstrace: Process 4844 detached

yshaban pada 18 Okt 2020

@yshaban Sudahkah Anda mencoba HAProxy?

RostislavBiloshapka pada 18 Okt 2020

@yshaban Sudahkah Anda mencoba HAProxy?

Terjebak barang saya di belakang haproxy hari ini, saya akan memberi tahu Anda.

yshaban pada 20 Okt 2020

👍1

@yshaban Sudahkah Anda mencoba HAProxy?

Seperti yang disarankan Rostislav, membuat cluster rabbitmq dan menyimpannya di belakang haproxy memecahkan masalah untuk saya sendiri.

yshaban pada 21 Okt 2020

👍1

@yshaban bagus!

RostislavBiloshapka pada 21 Okt 2020

Ini hanya diperbaiki saat menggunakan haproxy, tetapi bug masih ada saat menggunakan Celery secara langsung dengan RabbitMQ.

alanhamlett pada 31 Okt 2020

+1 seledri v5.0.1

lswjkllc pada 11 Nov 2020

seledri 4.3, haproxy mengatasi masalahnya, kalian harus mencobanya.

madpudding pada 13 Nov 2020

Gunakan Haproxy yang diperlukan dengan Celery atau buka kembali masalah ini.

alanhamlett pada 14 Nov 2020

👍1

Suatu masalah membutuhkan item yang dapat ditindaklanjuti.
Karena kami tidak tahu cara memperbaikinya dan tidak memiliki lingkungan untuk mereproduksi masalah ini, saya akan mengatakan bahwa ini harus tetap ditutup sampai seseorang yang memiliki akses ke lingkungan tempat reproduksi ini dapat memberikan beberapa wawasan tentang apa yang harus diperbaiki atau setidaknya apa yang salah.

thedrow pada 15 Nov 2020

Saya setuju dengan orang lain jelas ada beberapa akar penyebab dalam seledri yang menyebabkan hal ini terjadi dan menempelkannya di belakang haproxy, saat menyelesaikan masalah, tidak mengatasi penyebab yang mendasarinya.

yshaban pada 16 Nov 2020

Saya setuju dengan orang lain jelas ada beberapa akar penyebab dalam seledri yang menyebabkan hal ini terjadi dan menempelkannya di belakang haproxy, saat menyelesaikan masalah, tidak mengatasi penyebab yang mendasarinya.

kecurigaan saya adalah, karena transportasi amqp tidak asinkron, koneksi mungkin tidak berumur panjang.

auvipy pada 19 Nov 2020

orang yang menghadapi ini dapatkah Anda memeriksa & mencoba # 6528

auvipy pada 8 Des 2020

Apakah halaman ini membantu?

0 / 5 - 0 peringkat

Masalah terkait

Mencabut/Membatalkan tugas saat pekerja dimatikan

asmodehn · 3Komentar

Mendapatkan kesalahan segmentasi -sigserv saat menjalankan tugas seledri yang menghasilkan pdf menggunakan weasyprint

kgritesh · 3Komentar

celery beat tidak berfungsi, tetapi pekerja seledri --beat berfungsi, saat menggunakan basis data

xiaods · 3Komentar

Kesalahan dalam dokumen: CELERY_ROUTES -> CELERY_TASK_ROUTES

maxim25 · 3Komentar

Systemd: Seledri yang dimuat ulang akan menutupnya dengan bersih, tetapi tidak mau menyala kembali

jaddison · 3Komentar