@Vishwas - AMazing as always. Wish you had added these 2 points. 1) Advance tab - Batch history - When and what is the use of it. 2) Scenario - when you should streaming vs when you should aggregator - When we use database - Bulk Update - Use Aggregator - When writing to file in S3 bucket use streaming.
Sir if we are writing files to some external system in the batch processing what happens if that external system is down for time what happens to output data from batch processing how do we handle it do we need to create a objectstore there to store the data until external system up and them pass It on object store data in to external system which is the output of the batch processing
Nice vidoe...have one doubt on job instance id... If say it's hardcoded, while executing it's stopped in midway, while restarting the batch job, y it will try to create new batch job instance, eventually fails since having duplicate instance id, rather it should continue the execution by taking job instance id in object store...pls help me here...
Hi, Good explanation. I have one doubt about Batch Block Size. From Mule's documentation (docs.mulesoft.com/mule-runtime/4.3/batch-filters-and-batch-aggregator) what I understand is that Batch Block Size means total no of records available in memory at a time for a single thread. For example if you have 10000 records to process with Batch Block Size as 100 and total max concurrency as 16 (assume we have 8 cors), that means 1600 records will go in memory but from your explanation what I understood is Batch Block Size mean that it represents no of records will be taken out in memory from then those will be processed by all threads. Please explain if possible, Thanks.
Your understanding is correct, the thread is per block and that is exactly why one should not set the block size to less than 100 as it will assign only a single thread to that block of 100 records and make the processing sequential. I tried to go through a general approach to explain but maybe that caused confusion. Thanks for asking this doubt, others might find this helpful.
@@Vishwasp13 Thanks for the clarification. So I believe the life span of a particular block for that single thread, would be completion of processing phase i.e. all batch steps. So if we have given some positive no to the failed record count; a condition might occur such as some blocks processed with all steps and others were only able to finish step-1 and failed records count limit has been breached !! Plz give your thoughts.
@@Vishwasp13 I also found it a bit confusing becasue of the "batch job working in brief section" as there you explain one record can go ahead of the other by being assigned to a different thread each . In this case it makes sense ifor a block size of 1. Like said in Mule documentation: "Within the Batch Step component, processing takes place in parallel at the block level, but records within each block process sequentially by default." "After processing records in a block, the Batch Step sends the records back to the stepping queue where the records wait for the next Batch Step component."
Really good article to understand Mule Batchjob concept... thanks for creating such content ..really appreciate it...
@Vishwas - AMazing as always. Wish you had added these 2 points.
1) Advance tab - Batch history - When and what is the use of it.
2) Scenario - when you should streaming vs when you should aggregator - When we use database - Bulk Update - Use Aggregator - When writing to file in S3 bucket use streaming.
streaming provides one-read forward-only iteration of records whereas the aggregation permits gives you random and/or sequential access to records
Thanks for the inputs, I thought of adding these points but the Batch History tab is not used much so thought of skipping it.
awesome explanation.
Highly appreciated
This is very helpful video 🤟
Great. Good presentation.
Thanks for the appreciation.
Great Video
Nice work👍
Very well explained
Brilliant
Sir if we are writing files to some external system in the batch processing what happens if that external system is down for time what happens to output data from batch processing how do we handle it do we need to create a objectstore there to store the data until external system up and them pass It on object store data in to external system which is the output of the batch processing
Nice vidoe...have one doubt on job instance id... If say it's hardcoded, while executing it's stopped in midway, while restarting the batch job, y it will try to create new batch job instance, eventually fails since having duplicate instance id, rather it should continue the execution by taking job instance id in object store...pls help me here...
Hi, Good explanation. I have one doubt about Batch Block Size. From Mule's documentation (docs.mulesoft.com/mule-runtime/4.3/batch-filters-and-batch-aggregator) what I understand is that Batch Block Size means total no of records available in memory at a time for a single thread. For example if you have 10000 records to process with Batch Block Size as 100 and total max concurrency as 16 (assume we have 8 cors), that means 1600 records will go in memory but from your explanation what I understood is Batch Block Size mean that it represents no of records will be taken out in memory from then those will be processed by all threads. Please explain if possible, Thanks.
Your understanding is correct, the thread is per block and that is exactly why one should not set the block size to less than 100 as it will assign only a single thread to that block of 100 records and make the processing sequential. I tried to go through a general approach to explain but maybe that caused confusion. Thanks for asking this doubt, others might find this helpful.
@@Vishwasp13 Thanks for the clarification. So I believe the life span of a particular block for that single thread, would be completion of processing phase i.e. all batch steps. So if we have given some positive no to the failed record count; a condition might occur such as some blocks processed with all steps and others were only able to finish step-1 and failed records count limit has been breached !! Plz give your thoughts.
@@Vishwasp13 I also found it a bit confusing becasue of the "batch job working in brief section" as there you explain one record can go ahead of the other by being assigned to a different thread each . In this case it makes sense ifor a block size of 1.
Like said in Mule documentation:
"Within the Batch Step component, processing takes place in parallel at the block level, but records within each block process sequentially by default."
"After processing records in a block, the Batch Step sends the records back to the stepping queue where the records wait for the next Batch Step component."