Celery: why load ID in task_prerun signal rather than task_received?

[grahamlyons]

I'm using this middleware and associated Celery extension and so I've used the function load_correlation_ids to ensure that the IDs are passed from FastAPI to Celery.

I noticed that the IDs would be available in the logs for the task succeeded messages from Celery but not the task received ones, so I changed the signal where the ID gets loaded to task_received instead of task_prerun (https://github.com/snok/asgi-correlation-id/blob/57f4e40/asgi_correlation_id/extensions/celery.py#L38).

I'm not very familiar with the internal workings of Celery, but is there are reason for using one signal over the other?

Update:

I spent some time investigating this and managed to get fairly† consistent loading of correlation IDs using the following signals:

• task_internal_error
• task_retry
• task_revoked
• task_received
• task_rejected
• task_postrun
• task_prerun

If anyone has more insight I'd love to hear it.

† the values always seem to be missing for the first task received log message, which looks like it runs before the task_received signal

[sondrelg]

I think task_received sounds sensible. The earlier the better, as long as we're in the right process 👍 That said, I stopped using Celery in my async apps because code sharing was too difficult. How has your experience been? It's been a while, but had good experiences with https://github.com/tobymao/saq and https://github.com/python-arq/arq

[JonasKs]

We're testing out temporal now as an alternative too!