Advanced Celery Django

This is a kind of a QnA happen between me and my colleague, so chances are we could be wrong (probability of happening this 0.01), if you find this helpful please clap and if not please comment down below. Also, you can request your own Questions and i will try answer then and add here!

Q: what are tables that will created by celery when we install in django project?

A: Celery itself don’t create any table, this is django-celery-beat package, which is an extension to Celery that provides a database-backed task scheduler.

the tables that are created by Celery in a Django project as follows:

1. django_celery_beat_intervalschedule: This table stores information about the intervals at which tasks are scheduled to run.
2. django_celery_beat_crontabschedule: This table stores information about the cron schedules for tasks.
3. django_celery_beat_periodictask: This table stores information about the periodic tasks that have been registered with the scheduler.
4. django_celery_beat_clockedschedule: This table stores information about the ClockedSchedule tasks that have been registered with the scheduler.
5. django_celery_beat_solicitudeschedule: This table stores information about the SolicitudeSchedule tasks that have been registered with the scheduler.
6. django_celery_beat_periodictasks: This table stores information about the tasks that are currently scheduled to run.
7. django_celery_results_taskresult: This table stores information about the results of the tasks that have been executed.

Q: Tell me the difference between ClockedSchedule tasks vs SolicitudeSchedule tasks vs Intervalschedule task vs crontabschedule task.

A: The answer is …

1. IntervalSchedule: This type of schedule is used to execute a task at a fixed interval of time. It requires a every parameter, which defines the time interval in seconds, minutes or hours at which the task should be executed. For example, every=10 will execute the task every 10 seconds.
2. CrontabSchedule: This type of schedule is used to execute a task based on a cron-like schedule. A cron schedule is defined using a string that specifies the exact time and date when the task should be executed. For example, This will execute the task every day at midnight.
3. ClockedSchedule : This type of schedule is used to execute a task at a specific time of day. It requires a clocked_time parameter, which defines the exact time of day when the task should be executed. For example, clocked_time=datetime.time(10, 0) will execute the task at 10:00am every day. Found in from celery.schedules import ClockedSchedule
4. SolicitudeSchedule : This type of schedule is used to execute a task based on a specific condition. It requires a condition parameter, which defines the condition when the task should be executed. For example, condition=lambda: datetime.now().hour == 10 will execute the task only if the current hour is 10.

Q: How to limit the number of tasks executed concurrently ?

A: per task level:

from celery import Celery

app = Celery('my_app')
app.conf.update(
    worker_concurrency=3,
)

global level:

from celery import Celery

app = Celery('my_app')
app.conf.update(
    worker_concurrency=3,
)

Q: How can i use`Celery.beat_schedule` to pick up the task from db, whenever new entry create in table `TaskTable`

we can use signal `post_save`

app = Celery('your_app_name')

app.conf.beat_schedule = {
    'check_new_tasks': {
        'task': 'your_app_name.tasks.execute_new_tasks',
        'schedule': timedelta(seconds=5),
        'args': ()
    },
}

# signals.py
@receiver(post_save, sender=TaskTable)
def execute_new_tasks(sender, **kwargs):
    task = kwargs.get('instance')
    if kwargs.get('created'):
        app.send_task('your_app_name.tasks.execute_new_tasks', args=[task.id])