Celery: Get celery task id before calling `delay`

Created on 22 Jan 2014 · 3Comments · Source: celery/celery

In my application, I use a table to keep track and store additional information of launched tasks (call it application task table). In some cases, I need the ID of the created application task record in the actual celery task. The problem is that this application task record is only created once the task is launched (because the celery task id is part of the application task ID record) and that it'll be hard to pass in the task ID into a running task.

In essence, I need the celery task id to create the application task record, and I need the application task id to create the celery task.

Is there any robust way of resolving this chicken-and-egg problem? One way I can think of is to stop the execution somehow until I created the application task record (in the caller of task_function.delay()), but I can't think of a way to do this with celery out of the box or how to pass the application task id to the task.

Any thoughts, pointers, hints on that? - Thanks a lot!

Source

myisis

👍3

Most helpful comment

You can finalize signature objects by freezing them:

>>> sig = add.s(2, 2)
>>> sig.freeze()
<AsyncResult: e89ff8f1-a556-4bb7-8b9e-5db9806ca082>
>>> sig.delay()
<AsyncResult: e89ff8f1-a556-4bb7-8b9e-5db9806ca082>

>>> sig = group(add.s(i, i) for i in range(5))
>>> sig.freeze()
<GroupResult: 9b4a8bf0-2f81-43fe-a5a0-23563441b25d [6eb971bc-ad3b-4f8f-9724-5b2baa493369, 2390bada-76f7-4680-8013-e9da78bf6a68, c5c510c1-80ed-49bc-aaaa-7ddbd5d025d2, 9be6363e-7290-407b-8c54-d6baf90d99d6, 35502fc1-9ce9-44df-917b-a6026daed646]>
>>> sig.delay()
<GroupResult: 9b4a8bf0-2f81-43fe-a5a0-23563441b25d [6eb971bc-ad3b-4f8f-9724-5b2baa493369, 2390bada-76f7-4680-8013-e9da78bf6a68, c5c510c1-80ed-49bc-aaaa-7ddbd5d025d2, 9be6363e-7290-407b-8c54-d6baf90d99d6, 35502fc1-9ce9-44df-917b-a6026daed646]>

You can specify the id for any task:

 >>> from celery import uuid
 >>> task_id = uuid()
 >>> task_id
'c7f388e9-d688-4f1d-be22-fb043b93c725'
 >>> add.apply_async((2, 2), task_id=task_id)
 <AsyncResult: c7f388e9-d688-4f1d-be22-fb043b93c725>

To do something when a task is called you can use the before_task_publish or
after_task_publish signals (see Signal Basics)

ask on 23 Jan 2014

👍38 🎉8 🚀1

All 3 comments

You can finalize signature objects by freezing them:

>>> sig = add.s(2, 2)
>>> sig.freeze()
<AsyncResult: e89ff8f1-a556-4bb7-8b9e-5db9806ca082>
>>> sig.delay()
<AsyncResult: e89ff8f1-a556-4bb7-8b9e-5db9806ca082>

>>> sig = group(add.s(i, i) for i in range(5))
>>> sig.freeze()
<GroupResult: 9b4a8bf0-2f81-43fe-a5a0-23563441b25d [6eb971bc-ad3b-4f8f-9724-5b2baa493369, 2390bada-76f7-4680-8013-e9da78bf6a68, c5c510c1-80ed-49bc-aaaa-7ddbd5d025d2, 9be6363e-7290-407b-8c54-d6baf90d99d6, 35502fc1-9ce9-44df-917b-a6026daed646]>
>>> sig.delay()
<GroupResult: 9b4a8bf0-2f81-43fe-a5a0-23563441b25d [6eb971bc-ad3b-4f8f-9724-5b2baa493369, 2390bada-76f7-4680-8013-e9da78bf6a68, c5c510c1-80ed-49bc-aaaa-7ddbd5d025d2, 9be6363e-7290-407b-8c54-d6baf90d99d6, 35502fc1-9ce9-44df-917b-a6026daed646]>

You can specify the id for any task:

 >>> from celery import uuid
 >>> task_id = uuid()
 >>> task_id
'c7f388e9-d688-4f1d-be22-fb043b93c725'
 >>> add.apply_async((2, 2), task_id=task_id)
 <AsyncResult: c7f388e9-d688-4f1d-be22-fb043b93c725>

To do something when a task is called you can use the before_task_publish or
after_task_publish signals (see Signal Basics)

ask on 23 Jan 2014

👍38 🎉8 🚀1

Thanks a lot. I went with option task_id = uuid(). It works like a charm!

myisis on 25 Jan 2014

👍7

I know this is old, but it was awesome to see that I wasn't the only one with a use case like this - and also awesome to see that Celery supports several (good) solutions. Great library design.