We are building a microservice in java. Responsibility of the microservice is to send emails by processing the incoming requests. To not impact the performance, we wanted to send the emails asynchronously. So we will queue the messages to a queue (in another microservice). We will have to support hundreds of tenants and have tens of queues per each tenant. So, the design should be scalable as the number of tenants and queues grow.
We are going to deploy in k8s. So we have multiple instances of service running in docker containers on pods. Assuming we have 100 queues created (in other microservice – lets say queue service),
We have a background job that runs periodically every 10 seconds on every pod to detect new queues.
For every queue, we spin up 2 threads –
a) polling thread – that is responsible for polling messages from queue-service and put it in a local queue
b) Processing thread – responsible to process messages from the local queue. Processing of message means sending an email in our usecase
For each of the queues, we end up spinning polling and processing threads
This solution is obviously not scalable as we cannot keep on creating new threads for each queue on each of the pods
I have come across a framework Akka, that seems to solve similar problems –
a) Sharing the workload across all available pods in cluster – Akka Cluster module
b) rebalancing the workload upon cluster topology changes
I am not familiar with Akka and actor based model and other concepts
Wanted to know if Akka is the right fit for the problem in hand? if yes, And how to model the design for my problem on a high level using Akka – Specially around how on each pod, I can allocate tenants and queues, and how to poll messages on each queue present in queue service and how to process the messages?