I see here two responsibilities. One is trigger delayed jobs and second to complete it. You can use cron job to select jobs to execute and put it to message broker one by one using for example outbox pattern. It's should take less than second, typically miliseconds. In this approach you have control of trigger time for example, if it will change from 7days do 5 or 10 days
Absolutely. It's about being pragmatic. I was going to mention this in the video but it was getting a bit long. I do mention similar approaches in other videos related to isolating individual work. As you mentioned that's one good approach. The one aspect I never really focused on in terms of batch jobs, which I should of, is overlapping jobs. If you run a job every minute, but the processing time takes longer than a minute. You'll kick off the job again potentially doing overlapping work. Handling this concurrency can be a nightmare if you don't already have some infrastructure/code to do this.
Seeing this kind of code is such an inspiration for my own code writing. This type of code has seams defined in logical ways that afford for some interesting software testing. It also makes for a fantastic DX (developer experience) when building code (debugging, checking telemetry, exploratory/demo testing). Nice!
Ok so maybe noob question here, aren't you offloading the cron behaviour from your application domain to the queue as delayed delivery ? What kind of pattern or engineering is done inside the queue implementation to actually process any pending messages which need to be delivered on a particular day ? What I really want to know is whether we are just effectively doing the same thing the difference being that it is encapsulated from us.
Is this also usable for time based events. For example if I've a working day with a start and end time where it sends a working day started and ended event? Currently I would make a cronjob to fire such time based events, but as you said this can fail if the cron doesn't run for some reason.
I'm all against batch jobs myself, but I'm curious how this would be implemented to work better. If it's just a simple async sleep then we have to worry about process running constantly, databases for delayed events would help but it would definitely got really messy quick if we try to scale it horizontally. Oher problem with async is, what happens if our delay is very long and we get a lot of them. I'm pretty sure having a lot of threads sleeping at the same time is not a good idea. Other solution would be to essentially run "batch jobs" every second checking which expire events should be fired. This doesn't sound too efficient either, but at least solves the problem with doing everything as a batch. Any good ideas?
I'm not sure what you're referring to about threads sleeping? In my example of using a queue that supports delayed delivery, you're basically sending a message to the queue that won't be visible to consumers once the "delay" passes. Once the sender enqueues a message, it's done. There's no thread sleeping or anything of the sort.
I'm talking more low level solution and thread sleeping was just one idea how this could be implemented. The time delay inside queue has to happen somehow and I'm curious how it's done.
Oh I see. Well the issue with something like that is durability. Any process restart is going to lose all threads which means you'll lose everything you already defined to run in the future. Hence why using messaging and putting it into a durable queue. As for how a queue manages it, probably different per queue. You might want to check on how AWS SQS supports it. It supports delayed queues as well as delayed delivery per message.
If you don't have delayed queues, you can work with two queues. The "delay" queue, which is a regular queue where you configure a dead-letter-queue pointing to "main" queue, and que "main" queue where you put your consumer. So, the producer sends a message with a TimeToLive set to the "delay" queue. When that TTL expires, de message is sent to the "main" queue, where is consumed.... just an idea of how to setup.
A nice approach, but you have to double check if a particular queue supports that. e.g tried with MassTransit + SQS set up and after vague error messages we figured out that max delay is 15 minutes(SQS limit).. Then again, tried hangfire with postgree for delaying tasks and it worked pretty well. But recently we run running a performance with high load and exceeded the amount of allowed connections to the db...Finally we ended up with a batch job for some cases and hangfire approach for others..
Exactly, thanks for the comment. As you mentioned there are implications such as does your broker support it. With SQS the delayed delivery limit is low, which means you'd have to dequeue then enqueue again with delayed delivery, which will invoke cost/usage from SQS.
@@CodeOpinion thanks. Thats also an option indeed. Besides costs, it also Adds some extra complexity. Also what if you went down for more than 15 minutes...
One solution is you'd need to model that so that when your expiration handler runs it knows (query something) that it needs to re-enqueue another message with a delay for the required time.
Old video but how do you handle large processes of data that needs to execute on thousands of records contained in dbs with millions of records. Our company uses overly complex sql sprocs. This is highly efficient when it comes to performance, vs doing it 1 record at a time, but it comes at a cost.... maintaining and altering these sprocs is hell.
All context specific. Batch jobs in your situation might be the best solution. No one size fits all but one thing you can look at its fanning one huge job out into smaller chunks. Or specific parts of a job could be done asynchronously as a separate job.
Great solution, however i would only go for it if i wasn't on the cloud as AWS and Azure already provides those features out of the box (EventBridge linked to a lambda for ex)
How would you achieve recurring batch jobs? Would you have the consumer send another delayed message to itself when the job is complete? Or is there something more elegant I am missing.
The idea here is to actively avoid "batch" jobs when possible. There are plenty of ways to schedule recurrence of any software process. Cron, Windows Task Scheduler, big enterprise systems like Active Batch, AWS EventBridge, Azure Logic Apps, etc. There are other concerns with this type of scheduling, some are mentioned in another comment thread.
There is the temptation to send a cancel event, but this is usually unnecessary. As Derek mentioned, in most cases you just drop the event on the floor. Think of these events as reminders to check and act if necessary rather than commands to do something at a particular time. The assumption here is that it is easy to tell if the state of the entity in question is easy to determine.
Thanks for the replies, ok I was trying to see this as an alternative rather than for a very narrow use case. Which is fine, just wasn't sure if this was an approach for all use cases, which would allow competing consumers to spread the load etc.
Any recommendations for a service that allow a delay queue? About to start migrating cron jobs and using this will make our live easier.
2 ปีที่แล้ว +1
Azure Service Bus offers this out of the box, RabbitMQ has it by installing an additional plugin, SQS has it, but last time I checked it had a time limit. There are plenty of options out there
Ya there's a time limit for SQS, but depending what your doing, you can always re-enqueue it with another delay. Obviously you'll have to pay for that appropriately based on SQS pricing.
I think long term delaying is not quite as common a feature as one might hope for in all the usual suspects in the SaaS space. There seem to be asterisks. SQS has a relatively (in relation to Derek's example) short maximum delay. Step Functions have running limits (1 million per account?) which is at least something that needs to be on the radar. I'm not sure SaaS solutions like AWS MQ for Rabbit allow plugins like that, my guess is not. If you already host your own Rabbit maybe its not a big deal. I still wonder how it deals with a large volume of delayed messages coming in out of order under the covers. It's not a trivial problem if you need it to scale. More traditionally in my experience, a first step is to run your batch process far more often than 24 hours in the middle of the night. I think Derek's example is a bit too mundane to really shed light on some real-world problems that encounters. In my experience, that's often forced by less tech savvy business partners who want to drop batches files on your services, not self-imposed by making batches out of messages which strikes me as an antipattern in the modern async world. If your software can guarantee message order there may be other options that do not require managing that extra data store. Imagine simply shutting down or pausing your logical consumer once it encounters the first message that is "not ready". I can't say I've built such a thing, but in theory it might work. It's certainly a very critical feature just to guarantee message ordering in the first place.
So we replace the DB and batch jobs (with proper error handling) with another (externalized) data-store (backing the queue) and a message listener of some kind? Thank god that increasing the amount of different technologies used does not equal complexity and additional middleware never has any serious bugs you got to deal with. In the end it's between a rock and a hard place.
I haven't found this to be the case. We use a simple micro-service and the same technologies we are using throughout our systems to create a 'later' event. While we use a message broker, a webhook would work just as well to manage these events. If your message broker doesn't offer this, we've implemented it using a variety of databases to manage the timed queue events - sqlite, mongodb, dynamodb - whatever is easiest works. Batch Jobs are still a thing and your circumstances may not warrant this design intervention. Many systems, however, experience a lot of pressure and are running multi-hour batch jobs that could be avoided by addressing the problem using this pattern.
If you watch any of my other videos, I do emphasize messaging as a way to decouple, add resilience and fault tolerance to any large system. If you're not already using messaging and an message broker, do I think adding all of that to remove your existing batch processes? Probably not, but context specific. The point of the video is to illustrate an alternative to batch jobs.
What if the Reservation Expired event starts to be processed and at the same time the order is being picked up? Can Actors solve that problem or is there another simpler solution?
A simple solution for the example is to just have the expiration event fire after business hours (so instead of exactly 7 days to the minute, have it set to run maybe an hour after the store closes on the 7th day). That's a code-free way to avoid the problem in the first place, assuming it fits your business model (i.e. the customer isn't able to come in and pick up any time 24/7)
For a more general case (like in a 24/7 store where the business hour trick wouldn't work), record locking would do it. It could be pessimistic locking (actually locking the record), or optimistic locking (using version numbers on the record). The two events can't happen at the same time, one has to be first. Either the customer picks up the item first and the expiration fails/exits (ideal), or the expiration fires and then the customer pickup fails and they have to buy the item again (since it was just refunded). The second option is a worse experience for the customer, but it's also probably unlikely to be a common problem, at least not much more likely than the customer trying to pick up on the 8th day or even further after expiration. You could probably design a system to eliminate the potential completely, but if the issue results in a minor inconvenience and only happens to 1 in 100,000 customers, it likely wouldn't be worth the effort and complication. Also, if you were doing optimistic locking, you would probably want the expiration to be a 2-parter. First set the record to "expiring", then do the actual refund, then set the record to "expired", that way the customer can't pick up the item and then be refunded on their way out the door.
I see here two responsibilities. One is trigger delayed jobs and second to complete it. You can use cron job to select jobs to execute and put it to message broker one by one using for example outbox pattern. It's should take less than second, typically miliseconds.
In this approach you have control of trigger time for example, if it will change from 7days do 5 or 10 days
Absolutely. It's about being pragmatic. I was going to mention this in the video but it was getting a bit long. I do mention similar approaches in other videos related to isolating individual work. As you mentioned that's one good approach. The one aspect I never really focused on in terms of batch jobs, which I should of, is overlapping jobs. If you run a job every minute, but the processing time takes longer than a minute. You'll kick off the job again potentially doing overlapping work. Handling this concurrency can be a nightmare if you don't already have some infrastructure/code to do this.
Seeing this kind of code is such an inspiration for my own code writing. This type of code has seams defined in logical ways that afford for some interesting software testing. It also makes for a fantastic DX (developer experience) when building code (debugging, checking telemetry, exploratory/demo testing). Nice!
Glad it gives a different perspective.
Ok so maybe noob question here, aren't you offloading the cron behaviour from your application domain to the queue as delayed delivery ? What kind of pattern or engineering is done inside the queue implementation to actually process any pending messages which need to be delivered on a particular day ? What I really want to know is whether we are just effectively doing the same thing the difference being that it is encapsulated from us.
Yes, you're moving the concern out of your own application code. Infrastructure (broker) can offer the benefits of isolation, retry, DLQ, etc.
Came here to ask this question 😃
I think Derek's response is perfect perspective (i.e. it relocation of the concern)
@@drewjaqua2905 my question was more around how you would actually implement something like this if I have you a use case
Thank you, never used anything other than cronjobs until now
Hopefully the video gave some ideas on other strategies.
Interesting. I've never thought of this approach before...
Is this also usable for time based events. For example if I've a working day with a start and end time where it sends a working day started and ended event? Currently I would make a cronjob to fire such time based events, but as you said this can fail if the cron doesn't run for some reason.
I'm all against batch jobs myself, but I'm curious how this would be implemented to work better.
If it's just a simple async sleep then we have to worry about process running constantly, databases for delayed events would help but it would definitely got really messy quick if we try to scale it horizontally. Oher problem with async is, what happens if our delay is very long and we get a lot of them. I'm pretty sure having a lot of threads sleeping at the same time is not a good idea.
Other solution would be to essentially run "batch jobs" every second checking which expire events should be fired. This doesn't sound too efficient either, but at least solves the problem with doing everything as a batch.
Any good ideas?
I'm not sure what you're referring to about threads sleeping? In my example of using a queue that supports delayed delivery, you're basically sending a message to the queue that won't be visible to consumers once the "delay" passes. Once the sender enqueues a message, it's done. There's no thread sleeping or anything of the sort.
I'm talking more low level solution and thread sleeping was just one idea how this could be implemented.
The time delay inside queue has to happen somehow and I'm curious how it's done.
Oh I see. Well the issue with something like that is durability. Any process restart is going to lose all threads which means you'll lose everything you already defined to run in the future. Hence why using messaging and putting it into a durable queue. As for how a queue manages it, probably different per queue. You might want to check on how AWS SQS supports it. It supports delayed queues as well as delayed delivery per message.
Yeah that's what I meant. Thanks i'll check it out!
If you don't have delayed queues, you can work with two queues. The "delay" queue, which is a regular queue where you configure a dead-letter-queue pointing to "main" queue, and que "main" queue where you put your consumer. So, the producer sends a message with a TimeToLive set to the "delay" queue. When that TTL expires, de message is sent to the "main" queue, where is consumed.... just an idea of how to setup.
Thanks very much for this video. Never thought of this alternative way to solve this problem that I meet quite often.
Just another option and way at looking at the problem.
I was really interested in this idea - though this is not available or stable in most systems
A nice approach, but you have to double check if a particular queue supports that. e.g tried with MassTransit + SQS set up and after vague error messages we figured out that max delay is 15 minutes(SQS limit).. Then again, tried hangfire with postgree for delaying tasks and it worked pretty well. But recently we run running a performance with high load and exceeded the amount of allowed connections to the db...Finally we ended up with a batch job for some cases and hangfire approach for others..
Exactly, thanks for the comment. As you mentioned there are implications such as does your broker support it. With SQS the delayed delivery limit is low, which means you'd have to dequeue then enqueue again with delayed delivery, which will invoke cost/usage from SQS.
@@CodeOpinion thanks. Thats also an option indeed. Besides costs, it also Adds some extra complexity. Also what if you went down for more than 15 minutes...
RaabitMQ same concept as implemented in last year:
private static Dictionary CreateArguments(string exchangeName, string routingKey, int millisecondsDelay) =>
new Dictionary
{
{ "x-dead-letter-exchange", exchangeName },
{ "x-dead-letter-routing-key", routingKey },
{ "x-message-ttl", millisecondsDelay }
//{ "x-expires", millisecondsDelay + QueueExpirationTime }
};
Your PowerPoints look awesome, how to get a copy of your template?
It's not a template, just basic colors/themes already in PowerPoint. Nothing special!
What if you need an extension of the time, for example if the expiration is in 7 days and in 5 days the customer wants an extension of 4 more days?
One solution is you'd need to model that so that when your expiration handler runs it knows (query something) that it needs to re-enqueue another message with a delay for the required time.
Old video but how do you handle large processes of data that needs to execute on thousands of records contained in dbs with millions of records. Our company uses overly complex sql sprocs. This is highly efficient when it comes to performance, vs doing it 1 record at a time, but it comes at a cost.... maintaining and altering these sprocs is hell.
All context specific. Batch jobs in your situation might be the best solution. No one size fits all but one thing you can look at its fanning one huge job out into smaller chunks. Or specific parts of a job could be done asynchronously as a separate job.
Great solution, however i would only go for it if i wasn't on the cloud as AWS and Azure already provides those features out of the box (EventBridge linked to a lambda for ex)
Great video Derek
Thanks 👍
How would you achieve recurring batch jobs? Would you have the consumer send another delayed message to itself when the job is complete? Or is there something more elegant I am missing.
The idea here is to actively avoid "batch" jobs when possible.
There are plenty of ways to schedule recurrence of any software process. Cron, Windows Task Scheduler, big enterprise systems like Active Batch, AWS EventBridge, Azure Logic Apps, etc.
There are other concerns with this type of scheduling, some are mentioned in another comment thread.
There is the temptation to send a cancel event, but this is usually unnecessary. As Derek mentioned, in most cases you just drop the event on the floor. Think of these events as reminders to check and act if necessary rather than commands to do something at a particular time. The assumption here is that it is easy to tell if the state of the entity in question is easy to determine.
Thanks for the replies, ok I was trying to see this as an alternative rather than for a very narrow use case. Which is fine, just wasn't sure if this was an approach for all use cases, which would allow competing consumers to spread the load etc.
Any recommendations for a service that allow a delay queue? About to start migrating cron jobs and using this will make our live easier.
Azure Service Bus offers this out of the box, RabbitMQ has it by installing an additional plugin, SQS has it, but last time I checked it had a time limit. There are plenty of options out there
Ya there's a time limit for SQS, but depending what your doing, you can always re-enqueue it with another delay. Obviously you'll have to pay for that appropriately based on SQS pricing.
@@CodeOpinion "The maximum is 15 minutes." seems like there would be a lot of re-enqueues. We're using step-functions for such task.
You could use a database and implement it on your own.
I think long term delaying is not quite as common a feature as one might hope for in all the usual suspects in the SaaS space. There seem to be asterisks. SQS has a relatively (in relation to Derek's example) short maximum delay. Step Functions have running limits (1 million per account?) which is at least something that needs to be on the radar.
I'm not sure SaaS solutions like AWS MQ for Rabbit allow plugins like that, my guess is not. If you already host your own Rabbit maybe its not a big deal. I still wonder how it deals with a large volume of delayed messages coming in out of order under the covers. It's not a trivial problem if you need it to scale.
More traditionally in my experience, a first step is to run your batch process far more often than 24 hours in the middle of the night. I think Derek's example is a bit too mundane to really shed light on some real-world problems that encounters. In my experience, that's often forced by less tech savvy business partners who want to drop batches files on your services, not self-imposed by making batches out of messages which strikes me as an antipattern in the modern async world.
If your software can guarantee message order there may be other options that do not require managing that extra data store. Imagine simply shutting down or pausing your logical consumer once it encounters the first message that is "not ready". I can't say I've built such a thing, but in theory it might work. It's certainly a very critical feature just to guarantee message ordering in the first place.
So we replace the DB and batch jobs (with proper error handling) with another (externalized) data-store (backing the queue) and a message listener of some kind? Thank god that increasing the amount of different technologies used does not equal complexity and additional middleware never has any serious bugs you got to deal with. In the end it's between a rock and a hard place.
I haven't found this to be the case. We use a simple micro-service and the same technologies we are using throughout our systems to create a 'later' event. While we use a message broker, a webhook would work just as well to manage these events. If your message broker doesn't offer this, we've implemented it using a variety of databases to manage the timed queue events - sqlite, mongodb, dynamodb - whatever is easiest works.
Batch Jobs are still a thing and your circumstances may not warrant this design intervention. Many systems, however, experience a lot of pressure and are running multi-hour batch jobs that could be avoided by addressing the problem using this pattern.
If you watch any of my other videos, I do emphasize messaging as a way to decouple, add resilience and fault tolerance to any large system. If you're not already using messaging and an message broker, do I think adding all of that to remove your existing batch processes? Probably not, but context specific. The point of the video is to illustrate an alternative to batch jobs.
What if the Reservation Expired event starts to be processed and at the same time the order is being picked up? Can Actors solve that problem or is there another simpler solution?
A simple solution for the example is to just have the expiration event fire after business hours (so instead of exactly 7 days to the minute, have it set to run maybe an hour after the store closes on the 7th day). That's a code-free way to avoid the problem in the first place, assuming it fits your business model (i.e. the customer isn't able to come in and pick up any time 24/7)
For a more general case (like in a 24/7 store where the business hour trick wouldn't work), record locking would do it. It could be pessimistic locking (actually locking the record), or optimistic locking (using version numbers on the record).
The two events can't happen at the same time, one has to be first. Either the customer picks up the item first and the expiration fails/exits (ideal), or the expiration fires and then the customer pickup fails and they have to buy the item again (since it was just refunded).
The second option is a worse experience for the customer, but it's also probably unlikely to be a common problem, at least not much more likely than the customer trying to pick up on the 8th day or even further after expiration.
You could probably design a system to eliminate the potential completely, but if the issue results in a minor inconvenience and only happens to 1 in 100,000 customers, it likely wouldn't be worth the effort and complication.
Also, if you were doing optimistic locking, you would probably want the expiration to be a 2-parter. First set the record to "expiring", then do the actual refund, then set the record to "expired", that way the customer can't pick up the item and then be refunded on their way out the door.
Is there any technics to revoke delayed message from message broker?
They still can be triggered. Just add some condition to your handler and do nothing, if message should be revoked (just like in 6:45).