django - Celery group multiple tasks in one design -
i getting familiar celery , have question. setup django-redis-celery
lets take example of task sending email:
tasks
@task def send_email(message): mailserver.sendonemessage(message)
views
class newaccount(apiview): def post(self, request, format=none): send_email.delay(request.data.email)
this works perfectly, django sends messages redis , picked celery execute task. want improve system celery picks messages redis @ intervals , executes single task multiple messages. because, connecting email server slow , sending multiple messages single request result in faster process.
i want work:
tasks
@task def send_emails(messages): mailserver.sendmultiplemessages(messages)
thoughts?
since using redis cache (django-redis) implemented following workflow:
step 1. create task adds new emails cache
@shared_task() def add_email(user_id): cache.set("email#{}".format(user_id), none, timeout=none)
step 2. create periodic task runs every second , looks new emails in cache
class processemailstask(periodictask): run_every = timedelta(seconds=1) def run(self, **kwargs): call_email() def call_email(): item_exists = true ids = [] while item_exists: try: key = next(cache.iter_keys("email#*")) ids.append(key.split("email#")[1]) cache.delete_pattern(key) except: item_exists = false if len(ids) > 0: send_emails_to(ids)
step 3. run both celery workers , celery beat , profit!
Comments
Post a Comment