Celery: Multiple bindings to the same queue: Only the last binding is being created in RabbitMQ

Created on 31 Oct 2016 · 3Comments · Source: celery/celery

Checklist

[X] I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
[x] I have verified that the issue exists against the master branch of Celery.

software -> celery:3.1.0 (Cipater) kombu:3.0.37 py:2.7.10
            billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:redis://localhost:6379/0

CELERY_QUEUES:
    (<unbound Queue Test_Queue -> <unbound Exchange test_exchange(topic)> -> myapp.tasks.some_tasks.#>,
 <unbound Queue Test_Queue -> <unbound Exchange test_exchange(topic)> -> myapp.tasks.more_tasks.#>)
RABBITMQ_MANAGEMENT_URL: 'localhost:15672'
CELERY_IMPORTS:
    ('myapp.tasks.some_tasks', 'myapp.tasks.more_tasks')
CELERY_ROUTES:
    (<myapp.routers.DefaultRouter object at 0x1077952d0>,)
BROKER_URL: 'amqp://guest:********@localhost:5672//'
CELERY_RESULT_BACKEND: 'redis://localhost:6379/0'

Steps to reproduce

Create 2 bindings on a topic exchange with different routing keys but point them to the same queue in CELERY_QUEUES.

CELERY_QUEUES = (
    Queue("Test_Queue", default_exchange, routing_key="myapp.tasks.some_tasks.#"),
    Queue("Test_Queue", default_exchange, routing_key="myapp.tasks.more_tasks.#"),
)

Instantiate a celery worker
Navigate to the RabbitMQ admin console http://localhost:15672/
Click onto the Queues tab, then the queue you created in your settings
See that only 1 binding exists

Expected behavior

There should be a second binding with the routing key myapp.tasks.some_tasks.#
This can be replicated in the celery amqp console:

celery amqp
queue.bind Test_Queue test_exchange Key1
queue.bind Test_Queue test_exchange Key2

Actual behavior

Only the last binding of a given queue name is created in rabbitmq

Source

sklarsa

Most helpful comment

CELERY_QUEUES is a mapping from queue_name to queue, so since you are defining two queues
with the same name, only one of them will be used.

To define a queue with multiple bindings in Kombu you should use:

from kombu import Exchange, Queue, binding
default_exchange = Exchange('default', type='direct')

CELERY_QUEUES = (
    Queue('Test_Queue', [
         binding(default_exchange, routing_key='myapp.tasks.some_tasks.#'),
         binding(default_exchange, routing_key='myapp.tasks.more_tasks.#'),
    ]),
)

ask on 31 Oct 2016

👍8

All 3 comments

CELERY_QUEUES is a mapping from queue_name to queue, so since you are defining two queues
with the same name, only one of them will be used.

To define a queue with multiple bindings in Kombu you should use:

from kombu import Exchange, Queue, binding
default_exchange = Exchange('default', type='direct')

CELERY_QUEUES = (
    Queue('Test_Queue', [
         binding(default_exchange, routing_key='myapp.tasks.some_tasks.#'),
         binding(default_exchange, routing_key='myapp.tasks.more_tasks.#'),
    ]),
)

ask on 31 Oct 2016

👍8

Great! Just tested it and both bindings have been instantiated. Is this worth adding to the routing docs? I can create a pull request sometime tonight or tomorrow

sklarsa on 1 Nov 2016

Sure! That would be a great idea to have it in the docs :)

ask on 1 Nov 2016

Was this page helpful?

0 / 5 - 0 ratings