- 83
- 142 019
Bayonne Sensei
Portugal
เข้าร่วมเมื่อ 18 มิ.ย. 2020
Hey guys I am Bayonne and I'm a Software Engineer.
In this channel you will find Software development tutorials. Mostly Java and Spring Ecosystem.
Software Design focus on Event-Driven Architecture.
In this channel you will find Software development tutorials. Mostly Java and Spring Ecosystem.
Software Design focus on Event-Driven Architecture.
Spring Cloud Stream Kafka: The RIGHT Way to Handle Retries & Dead Letter Queue!
🚀 Are you retrying messages directly from the original Kafka topic? BIG MISTAKE! This can block newer messages, creating latency, bottlenecks, and chaos in your event-driven architecture.
In this video, I'll show you the correct way to handle retries using Spring Cloud Stream Kafka Binder:
✅ Retry failed messages from a dedicated retry topic (instead of blocking the original topic).
✅ Use exponential backoff to avoid hammering your downstream services.
✅ Send unrecoverable messages to a Dead Letter Queue (DLQ) for later analysis.
🔴 Watch now and stop making this critical mistake! 🔴
💬 What do you think about this retry strategy? Drop your thoughts in the comments! 👇
Github:
Decision Microservice (Consumer): github.com/PascoalBayonne/Decision-microservice/tree/feature/retry-topic-dlq
Customer: (Producer)
github.com/PascoalBayonne/Customer-Microservice/tree/feature/idempotent-producer-ms
#SpringCloudStream #Kafka #Microservices #EventDriven #KafkaRetry #SpringBoot #TechTips
In this video, I'll show you the correct way to handle retries using Spring Cloud Stream Kafka Binder:
✅ Retry failed messages from a dedicated retry topic (instead of blocking the original topic).
✅ Use exponential backoff to avoid hammering your downstream services.
✅ Send unrecoverable messages to a Dead Letter Queue (DLQ) for later analysis.
🔴 Watch now and stop making this critical mistake! 🔴
💬 What do you think about this retry strategy? Drop your thoughts in the comments! 👇
Github:
Decision Microservice (Consumer): github.com/PascoalBayonne/Decision-microservice/tree/feature/retry-topic-dlq
Customer: (Producer)
github.com/PascoalBayonne/Customer-Microservice/tree/feature/idempotent-producer-ms
#SpringCloudStream #Kafka #Microservices #EventDriven #KafkaRetry #SpringBoot #TechTips
มุมมอง: 47
วีดีโอ
🚀 Spring Boot + Kafka: Master Idempotent Producer to Guarantee Order & Prevent Duplicates! 🔥
มุมมอง 40221 วันที่ผ่านมา
🚀 Master Kafka Idempotent Producer with Spring Boot! 🔥 In this video, you'll learn how to guarantee message order and prevent duplicates in your event-driven architecture using Spring Boot and Spring Cloud Stream Kafka Binder. We'll dive deep into the idempotent producer, exploring how it ensures exactly-once delivery and handles failures without compromising data integrity. 🎯 What You’ll Learn...
Mastering Large File Uploads to AWS S3 with Spring Boot & Spring Batch - Multipart Made Easy! 🚀
มุมมอง 3772 หลายเดือนก่อน
In this video, we’ll show you the power of multipart uploads for handling large files using Spring Boot & Spring Batch in AWS S3! 🎉 And we’re taking it further by using the AWS CRT-based S3 client for ultimate performance and high throughput. 💨 🛠️ Why the CRT Client? Built on the AWS Common Runtime, it’s optimized for speed, automatically using multipart uploads to supercharge your upload proce...
Master Spring Batch: Effortlessly Upload Files to AWS S3 Buckets! 🚀 | Ultimate Step-by-Step Guide #1
มุมมอง 4273 หลายเดือนก่อน
🚀 Welcome to your ultimate guide on mastering Spring Batch for uploading files to AWS S3! 🚀 In this tutorial, I’ll show you step-by-step how to leverage Spring Batch to efficiently upload files to an Amazon S3 bucket. Whether you’re working with small files or looking for scalable solutions, this video has got you covered! 💡 In this video, you’ll learn: Setting up a Spring Batch job to handle f...
Spring Batch Explained: How to use Spring Batch to export data from a database to a file?
มุมมอง 3.5K5 หลายเดือนก่อน
In this video, you'll learn how to use Spring Batch to export data from a database to a CSV file. We'll cover the essential steps to set up your Spring Batch project, configure the job and step, and implement the item reader, processor, and writer. Additionally, we'll explore and compare two different reading strategies: a cursor-based item reader and a paging-based item reader. Please leave a ...
Microservices: How to Consume Messages in Batch Mode + Concurrency with Spring Cloud Stream Kafka?
มุมมอง 6798 หลายเดือนก่อน
Git: github.com/PascoalBayonne/Decision-microservice/tree/feature/consumer-batch-mode Unlock the power of efficient message consumption with Spring Cloud Stream Kafka! In this tutorial, learn how to leverage batch mode and concurrency to optimize your microservices architecture. Discover step-by-step guidance on consuming messages in batch mode while harnessing the benefits of concurrency, all ...
Spring Batch: How to send bulk emails using Spring Batch. Ep: 5
มุมมอง 72310 หลายเดือนก่อน
In this video we have seen How to send bulk emails using Spring Batch. Apart of that, we have shown also how we can pass data between steps of a job. We used Mailhog docker image in order to simulate our mail inbox. #springbatch #springbootemail #springframework
Spring Batch: Multi-File Input and Virtual Threads support - Share Data Between Steps. Ep: 4
มุมมอง 44311 หลายเดือนก่อน
In this video we are going to explore how to process multiple input files from a directory as a single unit of work. For example: Our business team provides daily reports which are composed by multiple files and for every single dat we need to process all the files as one atomic operation. In that case we are going to use different approaches that spring batch offers to us. We are going to use ...
Spring Batch: Multi-File Input (Upload PDF) and Virtual Threads support. Ep: 3
มุมมอง 537ปีที่แล้ว
In this video we are going to explore how to process multiple input files from a directory as a single unit of work. For example: Our business team provides daily reports which are composed by multiple files and for every single dat we need to process all the files as one atomic operation. In that case we are going to use different approaches that spring batch offers to us. We are going to use ...
Spring Batch: Multi-File Input (JSON) and Virtual Threads support. Ep: 2
มุมมอง 1.1Kปีที่แล้ว
In this video we are going to explore how to process multiple input files from a directory as a single unit of work. For example: Our business team provides daily reports which are composed by multiple files and for every single dat we need to process all the files as one atomic operation. In that case we are going to use different approaches that spring batch offers to us. We are going to use ...
Spring Batch: Multi-File Input and Virtual Threads support. Ep: 1
มุมมอง 4.3Kปีที่แล้ว
In this video we are going to explore how to process multiple input files from a directory as a single unit of work. For example: Our business team provides daily reports which are composed by multiple files and for every single dat we need to process all the files as one atomic operation. In that case we are going to use different approaches that spring batch offers to us. We are going to use ...
Microservices: Change Data Capture | Debezium Kafka Connect - Exatcly Once Delivery?
มุมมอง 482ปีที่แล้ว
Change Data Capture (CDC) is like having a system that watches your database and takes note of any changes (like inserts, updates, or deleting data | so Data Manipulation Language & Data Definition Language). As result we can use this information to keep other systems up to date, ensuring they always have the latest data. It's often used for things like copying data between databases or systems...
Microservices: Change Data Capture | Replace Transactional Outbox with Kafka Connect?
มุมมอง 752ปีที่แล้ว
Change Data Capture (CDC) is like having a system that watches your database and takes note of any changes (like inserts, updates, or deleting data | so Data Manipulation Language & Data Definition Language). As result we can use this information to keep other systems up to date, ensuring they always have the latest data. It's often used for things like copying data between databases or systems...
Microservices: Transactional outbox | Solving dual writes
มุมมอง 1Kปีที่แล้ว
- Dual writes is a situation when your application should change data in two different destinations or system, which in our case are database and kaki database. - Transactional outbox pattern solve these kind of problemas related to dual writes, in cases such as a transaction tries to update the database table then publish a message to middleware message system (kafka) and then commit the trans...
Microservices: Spring Cloud Contract | TestContainers - 06
มุมมอง 327ปีที่แล้ว
Understanding Synchronous vs. Event-Driven Applications is important but don't forget testing your event-driven microservices. This is what we are going to cover in this series. In this series, we will be talking about Microservices patterns associated with Event-Driven Architecture. We will also talk a little bit about Domain Driven Design (but not our focus). In this series, we will create a ...
Microservices: Spring Cloud Contract | TestContainers - 05
มุมมอง 435ปีที่แล้ว
Microservices: Spring Cloud Contract | TestContainers - 05
Microservices: Spring Cloud Contract | TestContainers - 04
มุมมอง 298ปีที่แล้ว
Microservices: Spring Cloud Contract | TestContainers - 04
Microservices: Spring Cloud Contract | TestContainers - 03
มุมมอง 268ปีที่แล้ว
Microservices: Spring Cloud Contract | TestContainers - 03
Microservices: Spring Cloud Contract | TestContainers - 02
มุมมอง 425ปีที่แล้ว
Microservices: Spring Cloud Contract | TestContainers - 02
Microservices: Spring Cloud Contract | TestContainers - 01
มุมมอง 813ปีที่แล้ว
Microservices: Spring Cloud Contract | TestContainers - 01
Spring Batch - Integration Tests | TestContainer Apache Kafka
มุมมอง 579ปีที่แล้ว
Spring Batch - Integration Tests | TestContainer Apache Kafka
Spring Batch: How to write items to multiple output sources?
มุมมอง 1.8Kปีที่แล้ว
Spring Batch: How to write items to multiple output sources?
Spring Batch - Integration Tests | TestContainer Database
มุมมอง 1.5Kปีที่แล้ว
Spring Batch - Integration Tests | TestContainer Database
End To End Testing of Batch Jobs | Spring Batch Testing 03
มุมมอง 1.7Kปีที่แล้ว
End To End Testing of Batch Jobs | Spring Batch Testing 03
End-To-End Testing of Batch Jobs | Spring Batch Testing - 02
มุมมอง 2.8Kปีที่แล้ว
End-To-End Testing of Batch Jobs | Spring Batch Testing - 02
End-To-End Testing of Batch Jobs | Spring Batch Testing - 01
มุมมอง 8Kปีที่แล้ว
End-To-End Testing of Batch Jobs | Spring Batch Testing - 01
Microservices : Event-Driven with Spring Cloud Stream - Message Router | Function Routing - PT-02
มุมมอง 671ปีที่แล้ว
Microservices : Event-Driven with Spring Cloud Stream - Message Router | Function Routing - PT-02
Microservices : Event-Driven with Spring Cloud Stream - Message Router | Function Routing - PT-01
มุมมอง 1.4Kปีที่แล้ว
Microservices : Event-Driven with Spring Cloud Stream - Message Router | Function Routing - PT-01
Microservices: Event Driven with Spring Cloud Stream (Apache Kafka) - Dead Letter Queue (DLQ)
มุมมอง 4.8Kปีที่แล้ว
Microservices: Event Driven with Spring Cloud Stream (Apache Kafka) - Dead Letter Queue (DLQ)
Microservices: Event Driven with Spring Cloud Stream (Apache Kafka) - Handling Failures (Retries)
มุมมอง 989ปีที่แล้ว
Microservices: Event Driven with Spring Cloud Stream (Apache Kafka) - Handling Failures (Retries)
Thanks for the video. Clear explanation. Keep up the good work. You rock man😊
Very happy to read such amazing comment. Let's Gooooo!❤🔥
Why we should get message from remote worker by another Kafka topic? I can't get it : if worker done well his job - why he should notice the master?
The manager should understand the state of the worker. For example how many lines were dropped? how many lines were written. That's important to worker to inform the manager through messaging system, it will let the manager decide if the entire job is completed or even proceed with some flows.
@@bayonnesensei8281 yes, I got it. At first glance I thought it some kind of "thebestpractice", but - it's obligated architecture to avoid the duplications. Thanks
@ amazing. Thanks for sharing your doubts.
Great video!
Glad you enjoyed it! Let's keep in touch. More videos coming. Thanks mate!
very good explanation.
Glad you liked it
It is very important concept. Thanks for sharing
Glad you think so!
According to this implementation (without OutboxMessagePublisher) the 'sent' column 'outbox_message' table will never be updated to 'true' value. Or maybe we even don't need such column in 'outbox_message' table?
how can I convert the data that from sql to JSON format before writing it to, AWS S3
Nice one... Got some ideas. You can just transform the data using for example Jackson in order to make it as json. You can even for example create an adicional tasklet step or step that uses the JsonItemWriter provided by spring batch. Let me know.
"Love your content! Informative and engaging-keep up the great work!"
Much appreciated! Will make more videos in next days.
Great job sensei. I mailed you regarding the CDCT, I would be happy to hear from you.
Hey my friend, happy new year. I took vacations and i was off. Feel free to contact me whenever you want to discuss about these topic. I AM BACK!
Great job, thank you sensei. I dropped you a mail few days ago, and would be happy to hear back from you.
Got it!
very nice! Thank u for your nice Videos 🙂
Thanks for your feedback mate. Glad it helped
How to write junit for below code method(){ Input stream file = new FileInputStream("path") Method2(file); file.close(); } Here path is not exists I want to mock or what I need to do here and I'm getting FileNotFoundException. I need file found instead like mock file.
Thank you Bayonne for this great content. Gratitudes!!
Thanks mate, happy that you guys are enjoying the content. Thank You!
good job man, thank you !!
You're welcome! Very glad you liked the video.
Good job. Thank you. It's sad of course, there isn't the Chunk size configuration option in Spring Boot Starter for S3. Then you have to implement your custom code instead of use the Spring's S3 Starter.
Thanks for sharing! Actually there is not any direct connection between S3 and Spring Batch, instead you have to rely on other frameworks for that. I am planning to cover this part soon.
Love the content ❤
Amazing, very glad you liked it.
This should be the default behavior. Good content with nice explanations. Thanks Sensei
Glad you think so!
I am waiting for the next video download file from S3
It's already coming. Thanks for your support
Please make a video on reading data from a csv file in S3 bucket.
Sure, the video is coming pretty soon my friend. We are going to talk about dowloading from S3. Thanks for your comment.
@@bayonnesensei8281 thanks mate 👍🏻
can you share the code
Nicely explained
Thank you so much 🙂
Cold emailing can be tricky, but with Mystrika, it feels manageable and efficient. The comprehensive analytics let me track everything in detail, and having a community of users to lean on is invaluable. If you are starting out or looking to improve, I cannot recommend Mystrika enough.
Just wrapping up a quarter using Filter Bounce, and I am convinced it is invaluable! The accuracy of the email verification for major domains is impressive. Plus, the fact that they provide 300 free verifications monthly is a fantastic perk. You really should consider this if you are looking to improve your cold email efforts.
I have been diving into cold emailing lately, and using Mystrika has been a breath of fresh air. The detailed analytics help me see exactly what is working. I previously struggled with email warmup, but Mystrika is warmup pool is unmatched. Seriously, if you are looking for a strong cold email strategy, you should definitely give it a shot!
Let is talk about my experience with Do You Mail. The unlimited sending from multiple email IDs has been a lifesaver. I used to worry about deliverability issues, but their system has nailed it. Honestly, for anyone pushing into cold emailing, just give it a go-you would not regret it!
Cold emailing is such a powerful tool if done right, and I cannot stress enough how much Do You Mail has transformed my approach. The scalability and unlimited sending from multiple domains have made life so easy. Plus, automatic SPF, DKIM, and DMARC configurations take away all the headaches. If you need a reliable solution, you ought to check this out!
I have crafted a few successful campaigns with Filter Bounce over the last few months, and I can say their tool is a game changer. The low bounce rate helps my emails land in inboxes instead of spam. Plus, they are super affordable! Seriously, check out their service if you want to refine your email approach.
I began using Do You Mail about two months ago and it has made a huge difference. The automatic configuration means I do not waste time on tech issues anymore. High delivery rates have boosted my campaigns significantly. If you are serious about scaling your cold outreach, you should definitely give it a look.
Hi, can you provide github link to the storage microservice? Because you didn't provide it in your videos. Thank you.
can u share source code
wish i knew your channel before ! best one ever for spring batch and kafka learning to beginners
Happy to hear that! I hope more videos will make such nice impression. Thanks
awsome thx , clear explanation
Glad it helped! Thanks for your feedback.
GitHub link please
Keep up the good job. Amazing content.
Thanks, will do!
good content
I appreciated, thanks for your support.
You continue to do videos man, i found it very useful even in my day to day work also. Don't think about views and subscribers. Thanks.
many thanks my friend! This is exactly what i think. Have a nice time.
Very good explanation. Thanks!!!
Glad it was helpful! Happy learning bro.
It's a great session on spring batch, I'm bit confused with thread concept, agreed threads got created but they are executed one after the other, there is no parallel processing isn't it?
Hello my friend, that is very nice question. For this particular video we are dealing with concurrency not parallelism
how use @Value("#{jobParameters['META_DATA']}") public String request; in ItemProcessor. i set but need @StepScope on ItemProcessor if set then Error scope on run app how fix it
Very nice tutorial, thank you Sensei
Very happy that you like. I am recording the same use-case but with better quality. Thanks a lot
Thanks for your sharing🎉
I really appreciate your support, thanks so much my friend.
These tutorials are awesome. Thank you.
Glad you like them! Planning to make more use case with different approach. Thanks for your feedback.
Very good informative tutorial. I am learning spring batch and I have question. As we are creating item reader and then marking it as bean in the same config class is any way I can have separate reader class where I will do the same thing and and just autowire in the config class and pass the autowired object in reader they way you did it for processor? Thank you in advance
Hey friend, thanks for your feedback. Well you can always separate the reader to its own class, i can say it is a good approach. This can be useful if you are creating a custom reader, so you can test it separately (Unit test). Take a look here th-cam.com/video/vLw39E-pIiA/w-d-xo.html
@@bayonnesensei8281 Thank you for your quick input. Just 1 input needed from you. I wanted to create a separate class for reader. As below I am marking it as bean in config file itself. Is there a way to create separate implementations for below @Bean public FlatFileItemReader<Person> reader() { FlatFileItemReader<Person> reader = new FlatFileItemReader<>(); reader.setResource(inputFile); reader.setLineMapper(new DefaultLineMapper<Person>() {{ setLineTokenizer(new DelimitedLineTokenizer() {{ setNames("firstName", "lastName", "age", "active"); }}); setFieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{ setTargetType(Person.class); }}); }}); return reader; }
What theme are you using in intellij?
Hey my friend, I am using the Light Theme on MAC, the affect is presentation mode.
very use full
Thanks a lot, You always supporting and that's amazing my friend.
Nov vídeo man. Was impressed and learned a lot after watching this video
Thanks sir!
That is amazing dude. I was really impressed and after watching this video l realized that l learned l lot. 🎉
Happy to hear that!
Very nice video, I would like to explore more partitions and concurrency with Spring Kafka.
More to come!