celery -A proj report
μ κ²°κ³Όλ₯Ό ν¬ν¨νμ΅λλ€.software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.4.3
billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
master
μ§μ μ λ¬Έμ κ° μμμ νμΈνμ΅λλ€.Celery 4.0.2λ‘ μ κ·Έλ μ΄λν©λλ€.
μ€λ₯κ° μμ΅λλ€.
λ΄ μμ μλ‘λΆν°μ΄ μ€λ₯κ° λ°μν©λλ€.
[2017-01-19 04:07:57,411: CRITICAL/MainProcess] Couldn't ack 461, reason:BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/opt/app/current/venv/lib/python3.4/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/channel.py", line 1408, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/abstract_channel.py", line 64, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/method_framing.py", line 174, in write_frame
write(view[:offset])
File "/opt/app/current/venv/lib/python3.4/site-packages/amqp/transport.py", line 269, in write
self._write(s)
BrokenPipeError: [Errno 32] Broken pipe
μ΄λ‘ μΈν΄ μμ μλ λͺ μ΄λ§λ€ RabbitMQμ λν μ°κ²°μ μ¬μ€μ νκ³ λλλ‘ RabbitMQ μλ²μμ νμ©λλ μ΅λ μ°κ²° μ νμ λλ¬νκ²λ©λλ€.
# 3328 λ° # 3377κ³Ό κ΄λ ¨μ΄μμ μ μμ΅λλ€.
μμΈμ νμ€νμ§ μμ§λ§ ν루λ μ΄νλ§λ€ μΌμ΄λκΈ° μμν©λλ€. μμ μλ₯Ό λ€μ μμνλ©΄ μ μ λ¬Έμ κ° ν΄κ²°λ©λλ€.
μ½κ° μ€λλ λ²μ μ μ λ¬λ¦¬μμλμ΄ λ¬Έμ κ° λ°μν©λλ€.
software -> celery:3.1.23 (Cipater) kombu:3.0.35 py:3.5.3
billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:djcelery.backends.database:DatabaseBackend
+1
+1 μ΅κ·Ό μ λ¬λ¦¬ 3.2.1μμμ΄ λ¬Έμ νμΈ
+1 μ΅κ·Ό 3.1.18μμμ΄ λ¬Έμ νμΈ
+1 4.0.2μμ λ¬Έμ 보기
+1 4.0.2μμ λ¬Έμ 보기
+1 4.0.2μμ λ¬Έμ 보기
+1 4.0.2μμ λ¬Έμ 보기
+1 4.0.2μμμ΄ λ¬Έμ νμΈ
μ°Έκ³ λ‘,μ΄ λ¬Έμ μ κΈ°ν Celery 4.x λ¬Έμ λ 3.xλ‘ λ€μ΄ κ·Έλ μ΄λνμ λ μμ λμμ΅λλ€. μ₯κΈ°μ μΈ ν΄κ²°μ±
μ https://github.com/Bogdanp/dramatiq λ‘ μ ννλ κ²μ
λλ€. Celery .delay()
μ λμΌν ꡬ문μ μ¬μ©νλ―λ‘ μ νμ΄ λ§€μ° μ½μ΅λλ€.
4.0.2μμ 보λ +1
4.0.2μμ 보λ +1
4.1.0μμ 보λ +1
4.1.0μμ 보λ +1
+1 μ λ¬λ¦¬ (4.1.0), gevent보기
+1 μ λ¬λ¦¬ 4.1.0
+1
rabbitmq λΈλ‘컀λ₯Ό μ¬μ©νμ¬ 4.1.0μμ νμΈ
+1
terminate
μΈμλ₯Ό μ¬μ©νμ¬ μμ
μ μ·¨μ ν λ λ°μ : app.control.revoke(t.id, terminate=True)
Flowerμμ μμ
μ μ’
λ£ ν λλ λμΌν©λλ€.
Python 3.6.2rc2
Celery 4.1.0 (latentcall)
broker: rabbitmq:3.6.12
backend: redis:3.2.10
[2017-10-11 09:31:49,471: ERROR/MainProcess] Control command error: BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/celery/worker/pidbox.py", line 42, in on_message
self.node.handle_message(body, message)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 129, in handle_message
return self.dispatch(**body)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 112, in dispatch
ticket=ticket)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 135, in reply
serializer=self.mailbox.serializer)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/pidbox.py", line 265, in _publish_reply
**opts
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/channel.py", line 1734, in _basic_publish
(0, exchange, routing_key, mandatory, immediate), msg
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/home/gustavorps/.pyenv/versions/3.6.2rc2/lib/python3.6/site-packages/amqp/transport.py", line 258, in write
self._write(s)
BrokenPipeError: [Errno 32] Broken pipe
+1 μ λ¬λ¦¬ 4.0.0
@mihajenko μ λ§μ§λ§ μ€λͺ μ μ°Έμ‘°νμμμ€ https://github.com/celery/celery/issues/3377
μ λ¬λ¦¬ +1 (4.0.2)
Traceback (most recent call last):
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/channel.py", line 1408, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame
write(view[:offset])
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/amqp/transport.py", line 269, in write
self._write(s)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 397, in sendall
tail = self.send(data, flags)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 391, in send
return self._send_loop(self.fd.send, data, flags)
File "/Users/celery/.virtualenvs/opb/lib/python2.7/site-packages/eventlet/greenio/base.py", line 378, in _send_loop
return send_method(data, *args)
error: [Errno 32] Broken pipe
@auvipy λ€μ μ΄μ΄μ£ΌμΈμ. μ΄κ²μ λ²κ·Έμ λλ€. ν΄νΉμ μμ μ¬νμ΄ μλλλ€.
@auvipy λΉ λ₯Έ μμ μ
+1 4.4
+1 @ _ @
+1
+1μ μ λ ₯νλ λμ μκ°μλ΄μ΄ κ΄λ ¨ λ¬Έμ λ₯Ό μ‘°μ¬νμ¬ κ°λ₯ν ν΄κ²°μ± μ μ°ΎμΌμμμ€. λꡬλ μ§ 4.2rc2μμ μ¬μ ν λ¬Έμ λ₯Όλ³΄κ³ ν μ μλ€λ©΄ μ’μκΉμ?
@auvipy μ°λ¦¬λ λ¬Έμ λ₯Ό μ‘°μ¬ν΄ μκ³ , μ°λ¦¬λ λ€μ ν° μ λ¬λ¦¬ + rabbitmq μΈνλΌλ₯Ό κ°μ§κ³ μμ΅λλ€. μ°λ¦¬λ μ€λ 3.1.25μμ 4.2 ν¨μΉλ‘ μ ννμ΅λλ€. μμ μ΄ % 4 κ°μνλ κ²μ보기 μμνμ΅λλ€. μ΄ μ€λ₯λ μμ μ€λ¨ / μ€ν¨μ μΌμΉνμ΅λλ€. μ ν κ³Όμ μμ μ μ¬ν λ¬Έμ μ μ§λ©΄νκ±°λ λ€μ΄ λ³Έ μ μ΄ μμ΅λκΉ?
λ¨Όμ λ‘컬μμ https://github.com/celery/celery/releases/tag/v3.1.26 μ μλν λ€μ 4.2rc2λ‘ μ κ·Έλ μ΄λ ν μ μμ΅λκΉ? zapierμμ λ³΄κ³ μλ₯Ό λ°μλλ° 3.1μμ 4.1λ‘ μ΄λνκ³ μμλλ°, λ λ€λ₯Έ μ μ pypiμμ celeryκ° 4.2rc2μ΄μ§λ§ μ’ μμ±μ΄ μμ§ pypiμ μλ€λ κ²μ λλ€. 4.2rc2λ‘ λ§μ€ν° λΈλμΉμ μ’ μμ±μ μ λ°μ΄νΈνκ³ μ΄λ€ μΌμ΄ λ°μνλμ§λ³΄κ³ μλ €μ£Όλ λ° μκ°μ ν μ ν μ μλ€λ©΄ μ λ§ μ’μ κ²μ λλ€. κ·Έ ν 4.2 μ΅μ’ 릴리μ€μ λν΄ λμ μ°μ μμ / μ°¨λ¨κΈ°λ‘ νμ ν μ μμ΅λλ€.
λν λ©μΌ λ§λ¦¬μ€νΈμ λν λ¬Έμ μ κΈ°λ₯Ό μμ²ν©λλ€
μ£μ‘ν©λλ€ @auvipy , λͺ νννκΈ° μν΄ : pypiμμ @ 4.2rc2 μ½λλ₯Ό μλν΄λ³΄κ³ μΆμ§λ§ μ’ μ νλͺ©μ΄ λ§μ€ν° λΈλμΉμ μ’ μμ± λͺ©λ‘κ³Ό μΌμΉνλλ‘ μ λ°μ΄νΈ λμμ΅λκΉ?
λͺ¨λλ μλμ§λ§ λκ΅°κ°κ° μλ€λ©΄ pypiμ celery 4.2rc2 λ° github λ§μ€ν°μ λ€λ₯Έ μλ¬λ¦¬ μ’ μ νλͺ©μ λ‘컬 / μ€ν μ΄μ§ μλ²μ brnachμ λλ€.
λλ @burakbostancioglu μ ν¨κ»
κ°μ¬ν©λλ€ :)
μ΄λ€ μ λ°μ΄νΈ λ μ? 4.2rc4λ₯Ό μ¬μ©νκ³ κ³μλκΉ?
λ‘λ μ€μΌ λ Celery 4.2.0μμ μλΉν μ κΈ°μ μΌλ‘ λ°μνλ κ²μΌλ‘ λνλ¬μ΅λλ€ (3.1.25-> 4.2.0 λ§μ΄κ·Έλ μ΄μ ν μ€νΈ μ€μ΄μ§λ§ μ°¨λ¨ μ€μ λλ€). λλ μμμ΄ ν¨μΉ κ·Έλ¦° λ¦Ώμ΄ μλλλ€. λΈλ‘컀λ‘μμ RMQ, κ²°κ³Ό λ°±μλλ‘ redis.
μ°λ¦¬κ° νΌν μ μλ€λ©΄ λ§μ΄κ·Έλ μ΄μ νκΈ°μλ λ무 μ΄μ¬ν νκ³ λ€μ§λ§ νμ κ°μ¬λ립λλ€ :-)
μ΄ μ€λ₯μ κ΄λ ¨νμ¬ μ‘°μ¬ ν μμλ μ¬νμ΄ μμΌλ©΄ μλ €μ£ΌμΈμ.
μΈκΈ λ μμ μ¬νμ μ μ©νμ΅λλ€.
https://github.com/celery/celery/issues/4226 κ·Έλ¦¬κ³ κ·Έκ²μ λλ₯Ό μν΄ λ¬Έμ λ₯Ό ν΄κ²°νμ΅λλ€.
κ·Έλ λ€λ©΄μ΄ λ¬Έμ κ° ν΄κ²° λμμ΅λκΉ? μλλ©΄ ν΄κ²° λ°©λ²μ΄ μμ΅λκΉ? λ΄ λͺ¨λ μμ νλ¦μ μ λ¬λ¦¬μ μμ‘΄νλ©°μ΄ λꡬλ₯Ό μ λ§ μ’μν©λλ€.
λ²κ·Έλ₯Ό ν΄κ²°νλ λ° λ°©ν΄κ°λλ κ²½μ° μ€μ ν΄κ²° μ μ ν΅ν½μ΄ μμ΅λκΉ?
@xirdneh κ°λ₯νλ©΄ κ΄λ ¨
λλ μ μ μ€μ --concurrency
λμ --autoscale
μ΄ λλ₯Ό μν΄ λ¬Έμ λ₯Ό ν΄κ²°νλ€.
(λμμ± 1μ λ²κ·Έκ° μκ³ μλ ν¬κΈ° μ‘°μ μ΄ μμ)
μ λ¬λ¦¬ 4.2.1μμ μ¬μ ν λμΌν λ¬Έμ κ° μμ΅λλ€. ν κ°μ§ λ λ§μ λ³΄κ³ μκ° λμμ΄λκΈ°λ₯Ό λ°λλλ€.
software -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:2.7.15rc1
billiard:3.5.0.4 redis:2.10.6
platform -> system:Linux arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379/0
broker_url: u'redis://127.0.0.1:6379/0'
result_backend: u'redis://127.0.0.1:6379/0'
task_routes: {
u'celery.crawler.fbk.download_alt_text': { u'queue': u'fbk_alt_text'}}
μ λ¬λ¦¬ λ§μ€ν°λ‘ ν μ€νΈ ν μ¬λ μλμ?
μ΄κ²μ μ¬μ ν ββμ‘΄μ¬ν©λλ€.
default celery-67c7fff78b-g48vx celery Couldn't ack 23, reason:BrokenPipeError(32, 'Broken pipe')
default celery-67c7fff78b-g48vx celery Traceback (most recent call last):
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 130, in ack_log_error
default celery-67c7fff78b-g48vx celery self.ack(multiple=multiple)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/kombu/message.py", line 125, in ack
default celery-67c7fff78b-g48vx celery self.channel.basic_ack(self.delivery_tag, multiple=multiple)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 1399, in basic_ack
default celery-67c7fff78b-g48vx celery spec.Basic.Ack, argsig, (delivery_tag, multiple),
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 51, in send_method
default celery-67c7fff78b-g48vx celery conn.frame_writer(1, self.channel_id, sig, args, content)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/method_framing.py", line 172, in write_frame
default celery-67c7fff78b-g48vx celery write(view[:offset])
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 288, in write
default celery-67c7fff78b-g48vx celery self._write(s)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 458, in sendall
default celery-67c7fff78b-g48vx celery return _socketcommon._sendall(self, data_memory, flags)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 374, in _sendall
default celery-67c7fff78b-g48vx celery timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk
default celery-67c7fff78b-g48vx celery data_sent += socket.send(chunk, flags)
default celery-67c7fff78b-g48vx celery File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 439, in send
default celery-67c7fff78b-g48vx celery return _socket.socket.send(self._sock, data, flags)
default celery-67c7fff78b-g48vx celery BrokenPipeError: [Errno 32] Broken pipe
gevent λ¬Έμ μΈ κ² κ°μ΅λκΉ? ile "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", 439 ν, μ μ‘ μ€
κΈ°λ³Έ celery-67c7fff78b-g48vx celery return _socket.socket.send (self._sock, data, flags)
default celery-67c7fff78b-g48vx celery BrokenPipeError : [Errno 32] κΉ¨μ§ νμ΄ν
μ λ¬λ¦¬ 4.4.0rc1?
https://github.com/celery/celery/issues/3377 μμ μ μ λ ν¨μΉλ‘ λ§μ€ν° λΈλμΉμμ μλνμ΅λλ€.
1 μκ° μλ νμλ μ¬μ ν λ©μΆ€
+1 μ λ¬λ¦¬ 4.2.1
@auvipy Celery 4.3.0 + gevent + RabbitMQμμμ΄ λ¬Έμ λ₯Ό 보μμ΅λλ€. strace
λ₯Ό μ¬μ©νμ¬ λ¬Έμ κ° recvfrom
λ° epoll_wait
syscallκ³Ό κ΄λ ¨λμ΄ μμμ λ°κ²¬νμ΅λλ€. μμΌμ΄ RabbitMQμμ λ©μμ§λ₯Ό μμ νμ§ λͺ»νκ³ λ¬΄ν 루νμ λΉ μ§λ κ² κ°μ΅λλ€. Celery νλ‘μΈμ€κ° κ±°κΈ°μ λΆμ΄ μμ΅λλ€.
26738
λ Celery νλ‘μΈμ€ IDμ
λλ€.
sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) sudo strace -p 26738 -f
...
[pid 26738] recvfrom(5, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441000388}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 441208787}) = 0
[pid 26738] epoll_wait(15, [], 64, 502) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944133989}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944706657}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399616, 944870229}) = 0
[pid 26738] epoll_wait(15, [], 64, 999) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944471272}) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 944612143}) = 0
[pid 26738] epoll_wait(15, [], 64, 1) = 0
[pid 26738] clock_gettime(CLOCK_MONOTONIC, {3399617, 945931383}) = 0
[pid 26738] recvfrom(21, 0x7fbead4995c4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
inode 5, 15, 21μ λν μΆκ° μ 보 :
$ sudo ls -la /proc/26738/fd/5
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/5 -> socket:[99157475]
$ sudo ls -la /proc/26738/fd/21
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/21 -> socket:[99144296]
$ sudo ls -la /proc/26738/fd/15
lrwx------ 1 xxx xxx 64 Aug 2 06:05 /proc/26738/fd/15 -> anon_inode:[eventpoll]
$ sudo lsof -p 26738 | grep 99157475
celery 26738 xxx 5u IPv4 99157475 0t0 TCP xxx-1084:50954->rabbit.xxx-1084:amqp (ESTABLISHED)
$ sudo lsof -p 26738 | grep 99144296
celery 26738 xxx 21u IPv4 99144296 0t0 TCP xxx-1084:38194->rabbit.xxx-1084:amqp (ESTABLISHED)
$ sudo head -n1 /proc/26738/net/tcp; grep -a 99157475 /proc/26738/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
10: 8A01010A:C70A 5E00010A:1628 01 00000000:00000000 02:00000351 00000000 1005 0 99157475 2 0000000000000000 20 4 30 10 -1
λμμ΄ λμκΈ°λ₯Ό λ°λλλ€.
+1
+1
@duydo λλ λΉμ κ³Ό κ°μ μν©μ λ§λκ³ μμ΅λλ€. κ·Έλ¦¬κ³ rabbitMQλ μ무κ²λ νμΈνμ§ λͺ»νμ΅λλ€.
Traceback (most recent call last):
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/channel.py", line 1399, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/abstract_channel.py", line 51, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/method_framing.py", line 172, in write_frame
write(view[:offset])
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/amqp/transport.py", line 288, in write
self._write(s)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 458, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 374, in _sendall
timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socketcommon.py", line 303, in __send_chunk
data_sent += socket.send(chunk, flags)
File "/home/ymserver/.pyenv/versions/adspider/lib/python3.6/site-packages/gevent/_socket3.py", line 439, in send
return _socket.socket.send(self._sock, data, flags)
BrokenPipeError: [Errno 32] Broken pipe
straceλ₯Ό μ¬μ©νμ¬ PIDλ₯Ό μΆμ ν©λλ€.
$ sudo strace -p 27977 -f
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108490973}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372759, tv_nsec=108664276}) = 0
[pid 27977] epoll_wait(4, [], 64, 1999) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=108549437}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109251966}) = 0
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109598283}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372761, tv_nsec=109777281}) = 0
[pid 27977] epoll_wait(4, [], 64, 1529) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=640754845}) = 0
[pid 27977] getpid() = 27977
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641809883}) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372762, tv_nsec=641940654}) = 0
[pid 27977] epoll_wait(4, [], 64, 467) = 0
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=109901002}) = 0
[pid 27977] recvfrom(10, 0x7fc8c406a758, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27977] clock_gettime(CLOCK_MONOTONIC, {tv_sec=10372763, tv_nsec=110880127}) = 0
[pid 27977] getpid() = 27977
$ sudo lsof -p 27977|grep 1651852686
python 27977 ymserver 10u IPv4 1651852686 0t0 TCP vuljp-ag-proxy-01:33006->10.55.4.70:amqp (ESTABLISHED)
https://github.com/celery/celery/pull/5581/files μλν΄λ³΄μμμ€
+1 μ λ¬λ¦¬ v4.4.2
+1 μ λ¬λ¦¬ 4.3.0
+1 μ λ¬λ¦¬ 4.3.0
μ€νΈ +1보λ€λ κ΄λ ¨ PRμ νμΈνκ³ λ¨μ ν μ€νΈλ₯Ό ν΅ν΄ κ°μ νμμμ€.
python2 λλ python3 envμ μκ΄μμ΄ μ λ¬λ¦¬ 4.4.6μ κ²½μ° +1.
software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:2.7.10
billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Darwin arch:64bit
kernel version:17.7.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:redis://10.28.218.6/
λ‘κ·Έλ λ€μκ³Ό κ°μ΅λλ€.
worker: Warm shutdown (MainProcess)
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Hub...
[2020-07-17 22:27:51,844: DEBUG/MainProcess] | Worker: Closing Pool...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Closing Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Connection...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Events...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Heart...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing Control...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Closing event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2020-07-17 22:27:51,845: DEBUG/MainProcess] | Consumer: Stopping Control...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Closed channel #3
[2020-07-17 22:27:52,153: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2020-07-17 22:27:52,153: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:27:53,279: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2020-07-17 22:27:53,280: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] Closed channel #2
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Events...
[2020-07-17 22:27:53,842: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2020-07-17 22:27:53,843: DEBUG/MainProcess] | Worker: Stopping Pool...
^@^@[2020-07-17 22:31:07,678: INFO/ForkPoolWorker-1] Task tasks.handle_compressed[5e493207-a3c5-422e-ab82-1f382342273c] succeeded in 200.757257249s: {'name': 'ray'}
[2020-07-17 22:31:09,188: DEBUG/MainProcess] | Worker: Stopping Hub...
[2020-07-17 22:31:09,189: CRITICAL/MainProcess] Couldn't ack 1, reason:error(32, 'Broken pipe')
Traceback (most recent call last):
File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 131, in ack_log_error
self.ack(multiple=multiple)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py", line 126, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py", line 1394, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py", line 59, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py", line 172, in write_frame
write(view[:offset])
File "/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py", line 305, in write
self._write(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2020-07-17 22:31:09,195: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Canceling task consumer...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] Closing consumer channel...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2020-07-17 22:31:09,196: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2020-07-17 22:31:09,431: DEBUG/MainProcess] Closed channel #1
[2020-07-17 22:31:09,663: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2020-07-17 22:31:09,665: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
μ°Έκ³ λ‘. disable broker_heartbeat (broker_heartbeat = 0)λ λ΄ λ¬Έμ λ₯Ό ν΄κ²°ν μ μμ΅λλ€. μ’μ μ€νμΈμ§ νμ€νμ§ μμ΅λλ€.
μ΄ λΆλ₯ μ λ³΄κ° κ·Όλ³Έ μ¬λ‘ λΆμμ λμμ΄λκΈ°λ₯Ό λ°λλλ€.
> ps -ef | grep worker
502 92665 75622 0 10:27PM ttys008 0:00.78 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
502 92672 92665 0 10:27PM ttys008 0:00.01 /Users/ranc/p2dev/bin/python /Users/ranc/p2dev/bin/celery -A tasks worker -l debug -c 1
> sudo dtruss -p 75622 -f
......
92665/0xf94bb7: fcntl(0x5, 0x4, 0x4) = 0 0
92665/0xf94bb7: wait4(0x16A00, 0x7FFEEB4B0C10, 0x1) = 0 0
92665/0xf94bb7: wait4(0x16A00, 0x7FFEEB4B13E0, 0x0) = 92672 0
92665/0xf94bb7: close(0xB) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x15) = -1 Err#32
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/kombu/message.py\0", 0x7FFEEB4AF680, 0x0) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/channel.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/abstract_channel.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/method_framing.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/Users/ranc/p2dev/lib/python2.7/site-packages/amqp/transport.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: mmap(0x0, 0x40000, 0x3, 0x1002, 0xFFFFFFFFFFFFFFFF, 0x0) = 0x1069D1000 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: stat64("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x7FFEEB4AF280, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py\0", 0x0, 0x1B6) = 11 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AF0F8, 0x0) = 0 0
92665/0xf94bb7: fstat64(0xB, 0x7FFEEB4AD098, 0x0) = 0 0
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2175 (ID 945: syscall::read_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: close_nocancel(0xB) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xE, 0x1068F0000, 0x13) = 19 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970E4, 0x7) = 7 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970B4, 0x4) = 4 0
92665/0xf94bb7: recvfrom(0xE, 0x10676348C, 0x1) = 1 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: madvise(0x1068CF000, 0x21000, 0x9) = 0 0
92665/0xf94bb7: sendto(0xE, 0x1068F0000, 0x13) = 19 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970E4, 0x7) = 7 0
92665/0xf94bb7: recvfrom(0xE, 0x1069970B4, 0x4) = 4 0
92665/0xf94bb7: recvfrom(0xE, 0x10676348C, 0x1) = 1 0
92665/0xf94bb7: shutdown(0xE, 0x2, 0x0) = 0 0
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x13) = -1 Err#32
92665/0xf94bb7: sendto(0xD, 0x1068AF000, 0x13) = -1 Err#32
92665/0xf94bb7: shutdown(0xD, 0x2, 0x0) = -1 Err#57
dtrace: error on enabled probe ID 2173 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: select(0x4, 0x7FFEEB4A6E58, 0x7FFEEB4A6ED8, 0x7FFEEB4A6F58, 0x7FFEEB4A6FD8) = 0 0
92665/0xf94bb7: fcntl(0x4, 0x3, 0x0) = 1 0
92665/0xf94bb7: fcntl(0x4, 0x4, 0x1) = 0 0
dtrace: error on enabled probe ID 2172 (ID 161: syscall::write:return): invalid kernel access in action #13 at DIF offset 68
92665/0xf94bb7: lstat64("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x7FFEEB4B34F0, 0x0) = 0 0
92665/0xf94bb7: open_nocancel("/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs\0", 0x1100004, 0x1213200) = 11 0
92665/0xf94bb7: fstatfs64(0xB, 0x7FFEEB4B2DB8, 0x0) = 0 0
92665/0xf94bb7: getdirentries64(0xB, 0x7FBE640A0A00, 0x1000) = 64 0
92665/0xf94bb7: getdirentries64(0xB, 0x7FBE640A0A00, 0x1000) = 0 0
92665/0xf94bb7: close_nocancel(0xB) = 0 0
92665/0xf94bb7: rmdir(0x7FBE61640B10, 0x0, 0x0) = 0 0
92665/0xf94bb7: sigaction(0x2, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0x3, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0xF, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: sigaction(0x1E, 0x7FFEEB4B4548, 0x7FFEEB4B4588) = 0 0
92665/0xf94bb7: madvise(0x105752000, 0x3C000, 0x9) = -1 Err#22
92665/0xf94bb7: close(0xD) = 0 0
92665/0xf94bb7: madvise(0x1068AF000, 0x20000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x106A11000, 0x23000, 0x9) = -1 Err#22
92665/0xf94bb7: close(0xE) = 0 0
92665/0xf94bb7: madvise(0x1068F0000, 0x20000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x10486D000, 0x30000, 0x9) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C25000, 0x5000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C81000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C83000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C85000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C87000, 0x2000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C8A000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C8F000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C92000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C99000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62C9E000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA1000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA3000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CA6000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CAB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CAE000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CB2000, 0x2000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CB6000, 0x4000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CBB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CBE000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CEA000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: madvise(0x7FBE62CFB000, 0x1000, 0x7) = 0 0
92665/0xf94bb7: close(0x6) = 0 0
dtrace: error on enabled probe ID 2366 (ID 897: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
^@^@^@^@^@^C
ranc-a01:~ ranc$ lsof -p 92665
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 92665 ranc cwd DIR 1,4 2176 12342436 /Users/ranc/vsan_vsancertification/tools/graphrunner
python 92665 ranc txt REG 1,4 51744 12719059 /Users/ranc/p2dev/bin/python
python 92665 ranc txt REG 1,4 52768 378470 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
python 92665 ranc txt REG 1,4 84240 378488 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
python 92665 ranc txt REG 1,4 83232 378468 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
python 92665 ranc txt REG 1,4 60192 378500 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
python 92665 ranc txt REG 1,4 71392 378444 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
python 92665 ranc txt REG 1,4 85136 378540 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
python 92665 ranc txt REG 1,4 108192 378532 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
python 92665 ranc txt REG 1,4 59984 378462 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
python 92665 ranc txt REG 1,4 48256 378458 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
python 92665 ranc txt REG 1,4 253840 378466 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
python 92665 ranc txt REG 1,4 73456 378552 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
python 92665 ranc txt REG 1,4 200880 378448 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
python 92665 ranc txt REG 1,4 65952 378558 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
python 92665 ranc txt REG 1,4 88816 378494 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
python 92665 ranc txt REG 1,4 78896 378534 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
python 92665 ranc txt REG 1,4 58240 378460 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
python 92665 ranc txt REG 1,4 52656 378478 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
python 92665 ranc txt REG 1,4 57040 378508 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
python 92665 ranc txt REG 1,4 74592 378446 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_csv.so
python 92665 ranc txt REG 1,4 43536 378526 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
python 92665 ranc txt REG 1,4 65184 378562 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
python 92665 ranc txt REG 1,4 144736 378482 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
python 92665 ranc txt REG 1,4 172352 378486 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
python 92665 ranc txt REG 1,4 44752 378480 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
python 92665 ranc txt REG 1,4 43708 12722902 /Users/ranc/p2dev/lib/python2.7/site-packages/_scandir.so
python 92665 ranc txt REG 1,4 152272 378514 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
python 92665 ranc txt REG 1,4 65920 378550 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
python 92665 ranc txt REG 1,4 52048 378520 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
python 92665 ranc txt REG 1,4 67920 378476 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
python 92665 ranc txt REG 1,4 146112 378506 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
python 92665 ranc txt REG 1,4 52048 378548 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/resource.so
python 92665 ranc txt REG 1,4 84000 378504 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/bz2.so
python 92665 ranc txt REG 1,4 43104 378430 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
python 92665 ranc txt REG 1,4 104256 378544 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so
python 92665 ranc txt REG 1,4 25512 15160994 /Users/ranc/p2dev/lib/python2.7/site-packages/tornado/speedups.so
python 92665 ranc txt REG 1,4 138464 378452 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
python 92665 ranc txt REG 1,4 1409984 378560 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/unicodedata.so
python 92665 ranc txt REG 1,4 27108 12722492 /Users/ranc/p2dev/lib/python2.7/site-packages/markupsafe/_speedups.so
python 92665 ranc txt REG 1,4 43588 14636845 /Users/ranc/p2dev/lib/python2.7/site-packages/_billiard.so
python 92665 ranc txt REG 1,4 65776 378536 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/mmap.so
python 92665 ranc txt REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc txt REG 1,4 841456 2716638 /usr/lib/dyld
python 92665 ranc txt REG 1,4 1172373504 13558190 /private/var/db/dyld/dyld_shared_cache_x86_64h
python 92665 ranc 0u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 1u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 2u CHR 16,8 0t1188168 1405 /dev/ttys008
python 92665 ranc 3 PIPE 0x75a36a27dcf37cad 16384 ->0x75a36a27dcf355ad
python 92665 ranc 4 PIPE 0x75a36a27dcf355ad 16384 ->0x75a36a27dcf37cad
python 92665 ranc 5 PIPE 0x75a36a27dcf3746d 16384 ->0x75a36a27dcf36dad
python 92665 ranc 6r CHR 14,1 0t15192 584 /dev/urandom
python 92665 ranc 7 PIPE 0x75a36a27dcf36dad 16384 ->0x75a36a27dcf3746d
python 92665 ranc 8u REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc 9u REG 1,4 4096 20515699 /private/var/folders/d9/xww6x1gj50j9769f0x47nw_h0000gp/T/pymp-kJBMDs/pym-92665-St6myq
python 92665 ranc 10r PSXSEM 0t0 /mp-r6SMtW
python 92665 ranc 11 PIPE 0x75a36a27dcf351ed 16384 ->0x75a36a27dcf36e6d
python 92665 ranc 12u systm 0x75a36a27dd0002cd 0t0 [ctl com.apple.netsrc id 8 unit 47]
python 92665 ranc 13u IPv4 0x75a36a280ae00f75 0t0 TCP 10.117.234.174:62477->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 14u IPv4 0x75a36a27fae89f75 0t0 TCP 10.117.234.174:62480->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 15u IPv4 0x75a36a27db0e88d5 0t0 TCP 10.117.234.174:62485->prme-vsan-hol-vm-dhcp-218-6.eng.vmware.com:amqp (ESTABLISHED)
python 92665 ranc 16 PIPE 0x75a36a27dcf351ed 16384 ->0x75a36a27dcf36e6d
λκ΅°κ° gevent
ν¨κ»μ΄ λ²κ·Έκ°μλ λ²μ μ λ°κ²¬ νμ΅λκΉ?
eventlet
) μ¬μ©νμ¬μ΄ λ¬Έμ (λλ μ μ¬ν λ¬Έμ )κ° νμλ©λλ€.
μλ
νμΈμ μ¬λ¬λΆ!
μ
λ¬λ¦¬μ κ²½μ° +1 4.4.6.
λλ μ€λμ΄ λ¬Έμ κ° μμλ€) μμ μ견과 λ΄ μλ²μ λν μ μ€ν μ‘°μ¬ λλΆμ λ¬Έμ λ₯Ό λ°κ²¬νμ΅λλ€)
RabbitMQ λ° Redisμ λ¬Έμ )
μ΄ λ¬Έμ λ₯Ό ν΄κ²°νλ €λ©΄ RabbitMQ λ° Redis μ© HAProxyλ₯Ό μ€μΉν΄μΌν©λλ€) μνκΉκ²λ μ°κ²°μ΄ λμ΄μ§κ³ μ΄ μ¬κ°ν μ€λ₯κ° λ°μν©λλ€. λ¬Έμμ μΆκ° μ λ³΄κ° μμ΅λλ€.)
https://www.rabbitmq.com/networking.html#proxy -effects
CELERY_BROKER_URL-RabbitMQ
CELERY_RESULT_BACKEND-Redis
μ΄ λ°μ΄ν°κ° μ€μνκΈ° λλ¬Έμ μμ
μ΄ μ ννκ² μ€νλλλ‘ μ
λ¬λ¦¬ μ¬μλ λ° κΈ°ν λ§€κ° λ³μλ ꡬμ±νμ΅λλ€.)
gevent μ 무μ κ΄κ³μμ΄ μλ) μ΄κ²μ΄ λ§μ λμμ΄λκΈ°λ₯Ό λ°λλλ€)
μ λ¬λ¦¬λ λ¬Έμ κ° μλλλ€!
μ λ¬λ¦¬λ λ¬Έμ κ° μλλλ€!
μ, Celeryκ° λ¬Έμ μ λλ€. κ·Έλ μ§ μμΌλ©΄ Celery 3λ‘ λ€μ΄ κ·Έλ μ΄λν΄λ λ¬Έμ κ° ν΄κ²°λμ§ μμ΅λλ€.
@alanhamlett μλ
νμΈμ)
HAProxyλ‘ 5 μΌ μ΄μ μμ
νμΌλ©° λͺ¨λ κ²μ΄ μ μμ
λλ€)
λλ νμμ΄ μ€λ₯κ° λ°μνκ³ λ΄ μμ
μλ λͺ λΆ λ§μ μΆ©λνμ΅λλ€)
λλ μ΄κ²μ΄ μ λ¬λ¦¬μμ κ³ μΉ μ μλ€λ κ²μ μ΄ν΄νμ§λ§ μμ μ λ§€μ° μ€λ 걸릴 μ μμ΅λλ€) λ Όλ¦¬λ₯Ό HAProxyμ²λΌ μλνκ² λ§λλ κ²μ μ΄λ ΅μ΅λλ€)
νμνμ§ μμ΅λλ€) HAProxyλμ΄ λ¬Έμ λ₯Ό μλ²½νκ² ν΄κ²°ν©λλ€)
+1 @duydoμ λμΌν μ¦μ
μ
λ¬λ¦¬ 4.4.7
μ΄κ²μ΄ μ΄λ»κ² ν΄κ²°λμ§ μμλμ§ νμ€νμ§ μμ΅λλ€.
39315 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 534, in sendall
39316 return _socketcommon._sendall(self, data_memory, flags)
39317 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 392, in _sendall
39318 timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
39319 File "/usr/local/lib64/python3.6/site-packages/gevent/_socketcommon.py", line 321, in __send_chunk
39320 data_sent += socket.send(chunk, flags)
39321 File "/usr/local/lib64/python3.6/site-packages/gevent/_socket3.py", line 515, in send
39322 return self._sock.send(data, flags)
39323 BrokenPipeError: [Errno 32] Broken pipe
getpid() = 6
epoll_wait(3, [{EPOLLIN, {u32=125, u64=266287972477}}], 262, 767) = 1
recvfrom(125, "\1\0\1\0\0\0#", 7, 0, NULL, NULL) = 7
recvfrom(125, "\0<\0<\6None12\0\0\0\0\0\0\4\220\0\rcelery.pidb"..., 35, 0, NULL, NULL) = 35
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
recvfrom(125, "\2\0\1\0\0\0G", 7, 0, NULL, NULL) = 7
recvfrom(125, "\0<\0\0\0\0\0\0\0\0\0c\370\0\20application/json\5"..., 71, 0, NULL, NULL) = 71
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
recvfrom(125, "\3\0\1\0\0\0c", 7, 0, NULL, NULL) = 7
recvfrom(125, "{\"method\": \"enable_events\", \"arg"..., 99, 0, NULL, NULL) = 99
recvfrom(125, "\316", 1, 0, NULL, NULL) = 1
getpid() = 6
write(2, "[2020-10-18 01:21:47,952: DEBUG/"..., 112) = 112
recvfrom(125, 0x7f8fbffe1078, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, [], 262, 108) = 0
getpid() = 6
epoll_wait(3, [], 262, 805) = 0
recvfrom(6, 0x7f8fbffe1de8, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, [], 262, 85) = 0
recvfrom(125, 0x7f8fbffe1d98, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
getpid() = 6
epoll_wait(3, ^Cstrace: Process 4844 detached
@yshaban HAProxy λ₯Ό μ¬μ©ν΄ λ³΄μ ¨μ΅λκΉ?
@yshaban HAProxy λ₯Ό μ¬μ©ν΄ λ³΄μ ¨μ΅λκΉ?
μ€λ haproxy λ€μ λ΄ λ¬Όκ±΄μ΄ λΆμ΄ μμΌλ©΄ μλ € λλ¦¬κ² μ΅λλ€.
@yshaban HAProxy λ₯Ό μ¬μ©ν΄ λ³΄μ ¨μ΅λκΉ?
Rostislavκ° μ μνλ―μ΄ rabbitmq ν΄λ¬μ€ν°λ₯Ό λ§λ€κ³ haproxy λ€μ λΆμ΄λ©΄ λ¬Έμ κ° ν΄κ²°λμμ΅λλ€.
νΈμ λ΄μ κ°κΈ°
μ΄κ²μ haproxyλ₯Ό μ¬μ©ν λλ§ μμ λμ§λ§, RabbitMQμ ν¨κ» Celeryλ₯Ό μ§μ μ¬μ©ν λ λ²κ·Έκ° μ¬μ ν μ‘΄μ¬ν©λλ€.
+1 μ λ¬λ¦¬ v5.0.1
μ λ¬λ¦¬ 4.3, haproxyλ λ¬Έμ λ₯Ό ν΄κ²°ν©λλ€. μ¬λ¬λΆμ μλν΄μΌν©λλ€.
Celeryμμ Haproxyλ₯Ό νμ μ¬μ©μΌλ‘ μ€μ νκ±°λμ΄ λ¬Έμ λ₯Ό λ€μ μ½λ λ€.
λ¬Έμ μλ μ€ν κ°λ₯ν νλͺ©μ΄ νμν©λλ€.
μ΄ λ¬Έμ λ₯Ό ν΄κ²°νλ λ°©λ²μ λͺ¨λ₯΄κ³ μ΄ λ¬Έμ λ₯Ό μ¬ν ν μμλ νκ²½μ΄ μκΈ° λλ¬Έμμ΄ λ¬Έμ λ₯Ό μ¬ννλ νκ²½μ μ‘μΈμ€ ν μμλ μ¬λμ΄ μμ ν΄μΌ ν μ¬νμ λν ν΅μ°°λ ₯μ μ 곡 ν μμμ λκΉμ§ νμ μνλ₯Ό μ μ§ν΄μΌνλ€κ³ λ§νκ³ μΆμ΅λλ€. λκ° μλͺ» λμ λ.
λλ λ€λ₯Έ μ¬λλ€μ΄ μ λ¬λ¦¬ λ΄μ λΆλͺ ν μ΄κ²μ΄ λ°μνλ κ·Όλ³Έ μμΈμ΄ μκ³ λ¬Έμ λ₯Ό ν΄κ²°νλ λμ haproxy λ€μ κ³ μ°©νλ κ²μ΄ κ·Όλ³Έμ μΈ μμΈμ λ€λ£¨μ§ μλλ€λ λ° λμν©λλ€.
λλ λ€λ₯Έ μ¬λλ€μ΄ μ λ¬λ¦¬ λ΄μ λΆλͺ ν μ΄κ²μ΄ λ°μνλ κ·Όλ³Έ μμΈμ΄ μκ³ λ¬Έμ λ₯Ό ν΄κ²°νλ λμ haproxy λ€μ κ³ μ°©νλ κ²μ΄ κ·Όλ³Έμ μΈ μμΈμ λ€λ£¨μ§ μλλ€λ λ° λμν©λλ€.
λ΄ μμ¬μ amqp μ μ‘μ΄ λΉλκΈ°κ° μλκΈ° λλ¬Έμ μ°κ²°μ΄ μ€λ μ§μλμ§ μμ μ μλ€λ κ²μ λλ€.
μ΄ λ¬Έμ μ μ§λ©΄ ν μ¬λλ€μ # 6528μ νμΈνκ³ μλν΄ λ³Ό μ μμ΅λλ€.
κ°μ₯ μ μ©ν λκΈ
@auvipy λ€μ μ΄μ΄μ£ΌμΈμ. μ΄κ²μ λ²κ·Έμ λλ€. ν΄νΉμ μμ μ¬νμ΄ μλλλ€.