Recently, I've been working on a classic problem that every distributed system must tackle. Curious what it is and how I solved it?
Let me explain.
We're migrating our system to AWS ECS. We have several scheduled tasks that trigger at different times — some of which send emails to users. During testing, we noticed that our test users were receiving duplicate emails. As soon as I saw it, I realized we had missed a case that was already on our to-do list.
The problem: we were running multiple Celery workers, each with its own Beat scheduler.
For those unfamiliar, Celery Beat is a scheduler that periodically sends tasks to Celery workers — similar to a cron job in Python. The issue was that each Beat instance was independently triggering the same scheduled tasks, resulting in duplicate email deliveries.
I considered three possible solutions:
1. Run a single Beat instance
We could remove Beat from all Celery servers and run it on a dedicated instance.
Downside: If that server goes down, scheduled tasks won't run — introducing a single point of failure.
2. Implement leader election
Let the workers elect a leader responsible for running Beat. If the leader goes down, a new one is elected.
Challenge: What happens if a scheduled task is due during the election window? The new leader may miss it.
3. Use distributed locking
Let all instances attempt to schedule the task, but only one succeeds by acquiring a lock.
I implemented this using Redlock, a distributed locking algorithm backed by Redis. Before scheduling a task, each instance checks the cache. If the lock already exists, others skip execution. The lock expires after a certain time, ensuring the task can be scheduled again later.
This approach solved our duplication problem cleanly and without introducing a single point of failure.
If you have questions or suggestions, feel free to comment.
Since you've made it this far, we likely share similar interests — feel free to connect. I’d love to exchange ideas or even collaborate on something interesting.
Thanks for reading!