Spring Boot - Multithreading | Process Millions of Records in Batches | Hands-on Example

[@sahukarinaveenkumar3188]

We all can use this logic in realtime. Great Explanation. In most of the interviews these daya people are aksing scenario bases questions on Completedfuture. I think its a good example

[@Akash-tq1ui]

You are my teacher in springBoot microservices 😊 👩‍🏫 📚

[@MohamedKaanoun]

He is our teacher Lot of Love ♥

[@dailydoseoftrendings]

Good one, Basant. Way to GO !!

[@gopishettymahindra2713]

Thank you so much sir. I like it solving real time use cases.

[@vinodhreddy6600]

Thanks for this batch processing video.

[@yogeshchaudhary621]

Nice video! I like such kind of real problem applications.

[@aadithyagopalan2684]

Thanks. Nice one. Even my experience is that using custom executor service gives better performance when configured correctly. (Using the cores)

[@AsadOnline-sz3os]

Good approach . Java introduces a new way to implement fork join that's completble future

[@sujitkumar2196]

Awesome tutorial bro, thanks providing such type of video really it's helpful😀

[@arnabchowdhury4892]

@JavaTechiice Please use this logic for creating batchsize for (int i = 0; i < size; i += batchSize) {
batches.add(list.subList(i, Math.min(i + batchSize, size)));
}
return batches;

[@Rajivsingh-xb1nw]

Thanks for great knowledge.

[@hanumanthram6754]

Thank you for your hardwork ❤

[@rishiraj2548]

🙏🏻🙂👍🏻

[@deltamakuba]

Amazing content

[@sushant6815]

Waiting for such videos

[@grrlgd3835]

Thanks for this JT. I'm a bit confused how this overlaps with Spring Batch ?

[@Javatechie]

Yes, there isn’t much difference. You could use Spring Batch for this as well, and I’ve already covered it in my Spring Batch playlist. However, for this use case, Spring Batch isn’t the right choice since we’re not migrating data from a source to a destination. Instead, we’re performing operations on existing entities and publishing Kafka events.

[@Saravanan-lj9so]

Could you please Publish video for Rate limit using spring cloud gateway +RedisCache+Custom Ratelimit filter with , burstCapacity, replenishRate, requestedTokens

[@vinoth2819]

How about using parallel stream for each batch. Why was it changed. Could it process it much faster?

[@Javatechie]

Yes we can try that i believe

[@premierde]

👏 SpingBatch does the same thing. Can you please incorporate/integrate this in your SpringBatch and updtate there. I implemented via SpringBatch in our project.

[@erbayaskn271]

Thanks for the example, but if you increased the default thread pool settings in the second method, it would give the same result as the third method. ForkJoinPool customThreadPool = new ForkJoinPool(20);

[@hemangdave1038]

Completable future vs virtual thread what is the preferred approach....??

[@azharmobeen]

I have learned a lot from you, but this solution has several issues:
1. There’s no transaction management-what happens if some records are stored successfully while others are not?
2. There’s no tracking mechanism to identify which records succeeded and which ones failed.
3. There’s no retry mechanism in place to handle failures.
...and the list goes on.

[@Javatechie]

I completely agree with you buddy . This is just a simple demo to show how can we perform batch processing using thread

[@kaasiimaginarium]

Can we use virtual thread to bypass the thread count declaration

[@Javatechie]

Yes we can use VT instead of PT

[@kaasiimaginarium]

@@JavatechieHi, In one of my scenarios, we have a maximum list of a thousand.2 thousand records. I need to perform multiple operations on each record. Can I enable virtual threads and use parallel stream processing for this? In my case, do I really need to consider batch processing?

[@Javatechie]

Yes batch processing is optimize way to improve throughout in your case

[@kaasiimaginarium]

​Thanks for the quick reply@@Javatechie I'm trying to understand the consequences of implementing virtual threads with parallel stream processing. Typically, the number of records will be below 100, but in corner cases, it could reach up to 2000.

[@sathishspeaks]

Need more videos on spring batch
Asynchronous batch processing

[@Javatechie]

Satish It's already there please check spring batch Playlist

[@sumitsaurav4033]

db throughput is comparatively low if we save lakhs of record will it affect our db, how can we optimise it using kafka

[@Farm2Code]

Hi sir or brother, will you please let us know the job market in November and December. I was searching jobs I used to see most are asking for immediate joiners so i thought like putting resignation letter and search and it's 3 weeks since I applied resignation not even single call I'm receiving so please enlighten me , us .

[@henrivalencia21]

thank you ser

[@maheshkumar-sl8eq]

Can we use spring batch sir for same

[@Javatechie]

No buddy spring batch used for different context where you have source and destination to migrate and transform the data

[@maheshkumar-sl8eq]

@ thank you sir

[@premierde]

​@@Javatechieyes but in each steps of (ETL) SpingBatch, it also need all this concept of throughput etc. Cant this be integrated in the steps? in better way

[@rabindradocument8934]

I think a single machine is processing all requests here. So we cant load all data into memory for batch processing. If we dont load all data then it will be latency issue.

[@Javatechie]

Single machine means Single vm right? And each vm have multiple cores so each core will initiate a new thread to execute your task . Why there will be latency issue? Can you please brief

[@rabindradocument8934]

@Javatechie a single machine won't scale considering cpu and memory limits being constraint. Here vertical scaling required with enough memory and cpu to process the requests in batch.

[@rabindradocument8934]

If I consider it is a 4 core machine then at any instance of time only 4 instructions will be running. So processing millions of records is possible but it will be slow with a single machine. Let say my machine performs good with 50 threads. But no guarantee that it will perform good with 5000 threads. Even cpu may not able to schedule those 5000 threads. Even if they got scheduled all of them may not get cpu time. Running more threads in a 4 core machine will bring lot of switching between threads.

[@rabindradocument8934]

Also as we are seeing about million records I feel we should delegate heavy processing logic to multiple machines instead doing in a single machine.

[@rabindradocument8934]

It is a nice example on batch but I have only concern of NFR and millions records. Anyway it's an informative video saying about batch processing.

[@bala7965]

Hi bro please continue multitherading ecommerce website scenario .In big sale time or offer time release an 1000 quantity smart phone give 60 percent offer.How multiple users try to book this scenario how effectively handle this scenario.please use Atomic Integer concepts for this bro.. please make a video bro

[@Javatechie]

Okay got your point quiteinteresting. Sure I will try this usecase asap .

[@sahukarinaveenkumar3188]

You are spending ur values time for us. So great of you

[@rabindradocument8934]

Atomic integer won't work in a distributed system or platform

[@bala7965]

​@@rabindradocument8934then how you can tell bro? volatile is not in this case won't work?do you know what is Atomic Integer purpose?

[@bala7965]

@@rabindradocument8934 Bro you can tell how will handle this situation. i saw few solution which i ask above if you have better or right one you can tell me.We are learning from your words

[@sahukarinaveenkumar3188]

Just want to know in firm you work?

[@Javatechie]

I am a developer who loves to make videos and share knowledge with the community!