Thanks for making this video Dax. An interesting use case you may want to cover in one of these videos is ECS/Fargate servers run 24/7 and subscribing to a queue listening for tasks instead of having the containers spin-up on-demand. In our case, we have around 20-50 running tasks at any time which may take up to 20 minutes to complete. The tasks are mostly IO-bound (lots of API calls) rather than CPU bound so we run multiple tasks concurrently on the same container. We did not find a good way to do this with SQS in a way which was reliable and respected the 1 task per group ID rule, so we ended up using BullMQ.
Great video and awesome new features! This addresses exactly what I've been dealing with. One thing I would have loved to see is how AddTask works with a queue. Currently, I have a queue that's limited to 10 running tasks, which I'm sending to regular Fargate while maintaining 10 tasks running constantly. This new approach will be a big improvement. However, I have a question: When I have 30+ tasks in my queue, will SST/AWS reuse the already spun-up container, or will each task get its own container? Also I really appreciate that you use a quality mic
Good timing! I was exploring this and was also arriving at a similar stack for jobs. My query is in frameworks like Laravel and Python's Celery you can use the same codebase for DB schema models etc. that your web app code uses to do database changes and other things. How could you achieve that here to keep things DRY and maintainable? I know maybe package things in monorepo but are there other ways with less setup?
by event bus are you referring to the pipe thing in cloudformation? or something else? and is it in sst bc i cant seem to find any info about event bus in the docs aside from it's what the Cron component uses
1:44 this is the sst channel 5:23 there's all kind off cool settings you cant put here 6:09 let's import import let's import 6:27 resource is out best friend 9:42 you can spin up one you can spin up five 17:39 you should try to grow into it instead of starting there
Would love to see a video on the event bus option you mentioned!
He did do a stream a few years ago. It's on SST v2, but has good information: th-cam.com/video/iMsPztMiFIk/w-d-xo.html
Is EventBus ever coming to v3?
@@SR-ti6jj it's already there undocumented, I'm using it in my project. Check the terminal repo
@@magnusred2945 they're using it in a very narrow way versus what was available in v2 and there's a bunch of issues
Yes please on the eventbridge video! I use them heavily with AWS SAM but would like to see your version.
Thanks for this video Dax. The dev mode in fargate is really useful, Great work SST team
Great video! Please keep up with these tutorials, they're extremely helpful. Thanks Dax!
Thanks for making this video Dax.
An interesting use case you may want to cover in one of these videos is ECS/Fargate servers run 24/7 and subscribing to a queue listening for tasks instead of having the containers spin-up on-demand. In our case, we have around 20-50 running tasks at any time which may take up to 20 minutes to complete. The tasks are mostly IO-bound (lots of API calls) rather than CPU bound so we run multiple tasks concurrently on the same container. We did not find a good way to do this with SQS in a way which was reliable and respected the 1 task per group ID rule, so we ended up using BullMQ.
Seconded - would love to see that type of architecture in sst!
Great video and awesome new features! This addresses exactly what I've been dealing with. One thing I would have loved to see is how AddTask works with a queue. Currently, I have a queue that's limited to 10 running tasks, which I'm sending to regular Fargate while maintaining 10 tasks running constantly. This new approach will be a big improvement.
However, I have a question: When I have 30+ tasks in my queue, will SST/AWS reuse the already spun-up container, or will each task get its own container? Also I really appreciate that you use a quality mic
This is really neat.
I've always struggled with observability with these kind of jobs. What do you guys recommend for this?
Good timing! I was exploring this and was also arriving at a similar stack for jobs. My query is in frameworks like Laravel and Python's Celery you can use the same codebase for DB schema models etc. that your web app code uses to do database changes and other things. How could you achieve that here to keep things DRY and maintainable? I know maybe package things in monorepo but are there other ways with less setup?
When do I choose SQS and when do I choose EventBus?
Damn this was good
Would using a cluster incur additional costs compared to just functions?
yes you have to pay for vpc at minimum
Can you trigger Tasks using Queues?
Is the Cluster class is wrapping Pulumi code that creates an ECS cluster?
by event bus are you referring to the pipe thing in cloudformation? or something else? and is it in sst bc i cant seem to find any info about event bus in the docs aside from it's what the Cron component uses
What is your neovim color theme? 😲
🎉
Next.js mentioned lmao
1:44 this is the sst channel
5:23 there's all kind off cool settings you cant put here
6:09 let's import import let's import
6:27 resource is out best friend
9:42 you can spin up one you can spin up five
17:39 you should try to grow into it instead of starting there
please please please give us event bus