At some point you are going to need to scale your application – maybe it is doing something that take a while – like running reports, or generating something in which you have a large amount of work to do.
In the case of automating text generation on images – the problem was with the volume of work that had to be done. 40K images.
Enter RabbitMQ and Messaging.
RabbitMQ is really nothing more than a message broker. That is – a system that receives, messages, delivers them based on some rule to queues (Mailboxes), where connected consumers can pick them up and then do whatever with them.
It is a solution that already exists for messaging with a lot of high level features – plus a Module Pika – already exists for Python – so getting up and running with is a fairly simple process. Not only for Python but many other programming languages.
A message can be – whatever you design. For example a JSON String – that would contain information nessary to execute a job.
The consumer would be the system – job engine that is attached (connected) subscribed to a queue (Mailbox). Once a consumer is subscribed to a queue, they will get messages delivered to that queue.
How It Unfolds.
You have a producer – that generates the messages. Maybe it is from events, or something. This producer fills up the queue with jobs.
You have the consumers – that are subscribed to the job queue.
You want to make sure on the consumers the prefetch value for RabbitMQ is set to 1 – this means that a consumer can only grab one message at a time. This is important for long running jobs. If you don’t set the prefetch value, the default will deliver all messages to one consumer. Which would defeat the purpose of what we are doing.
You also want to set AutoAck on the consumer to false. What AutoAck does – is when the consumer gets a message, it is automatically confirmed and the message is removed from the queue. With a job engine, best would be to Manually Acknowledge the message – after the job is completed.
By manually Acknowledging the message, after the jobs is complete – in the event a consumer goes offline (crashes) – the message is placed back on the queue for another engine to pick up and process. This way all jobs will get processed and messages are not lost.
Have a Project or Idea!?
I am Available for Freelance Projects
My skills are always primed and ready for new opportunities to be put to work, and I am ever on the lookout to connect with individuals who share a similar mindset.
If you’re intrigued and wish to collaborate, connect, or simply indulge in a stimulating conversation, don’t hesitate! Drop me an email and let’s begin our journey. I eagerly anticipate our interaction!