Spark Architecture in 3 minutes| Spark components | How spark works

Spark is one of the most prominent and widely used processing framework in Bigdata world. This videos explains the core components and architecture of spark with a real world example in just 3 minutes.

[@deepalirathod4929]

Finally it got cleared to me after reading here and there . thank you .

[@apurvgolatgaonkar6722]

Ma'am you taught amazing 😍😍 very less time consuming lecture but perfect... Keep it up

[@bitthal24]

I havent seen such a lucid way of explaining something this complex concept. Great work!

[@BigDataThoughts]

Thanks Bitthal

[@satishchippa]

Excellent way of explaining things in a most simplified manner.
Looking forward to more videos on Spark.

[@BigDataThoughts]

Thanks Satish

[@NIYANTAjmp]

very simple and nice explanation. Thank you for posting this video

[@SagarSingh-ie8tx]

Example was very good for beginners

[@showbhik9700]

This is the only video on TH-cam which clarified my doubts. Thanks!!

[@BigDataThoughts]

Thanks showbhik

[@BigDataThoughts]

Thanks showbhik

[@mehrozalam94]

Great Video, I have learned alot.
Thank you

[@theamithsingh]

finally, a video that simplifies spark, amazing, keep the videos coming please!!

[@BigDataThoughts]

Thanks

[@seemanthinin448]

Simple example and easy way of explaining an important concept.. thanks!

[@BigDataThoughts]

Thanks seemanthini

[@puneetojha1195]

This video is better than 1 hour course on spark . Thanks

[@BigDataThoughts]

Thanks puneet

[@desiengineerashish4908]

This made far easy to understand Spark Architecture. Thank u ma'am, you are great

[@BigDataThoughts]

Thanks Ashish

[@swapnilchilwant6867]

Thank you ma'am..👍

[@vsriga82]

Lot of information presented in simple way for everyone to understand. 👍

[@BigDataThoughts]

Thanks sriganesh

[@sandeepchoudhary4900]

Very nice video and it covers everything related to the Spark architecture in just 5 minutes. Keep sharing new videos.

[@BigDataThoughts]

Thanks sandeep

[@trainingt9855]

Great video

[@rahulkoley9447]

Very Informative, with full of clarity, Thank you.

[@BigDataThoughts]

thanks

[@SaurabhKumar-mc1is]

Vividly explained. Thanks mam

[@BigDataThoughts]

Thanks saurabh

[@ayyappareddymuthikepalli4261]

Nice explanation.. Pls keep videos 🎥 like this

[@dancingmoveswithdhruv3649]

Very clearly explained, really appreciate all your efforts

[@BigDataThoughts]

Thanks Dhruv

[@iamramanr4s]

way of explanation is .....just amazing

[@BigDataThoughts]

Thanks

[@gkethanvarma889]

Thank you so.................. much for this

[@mohnishverma87]

Just woow, very simple explanation of a complex cluster overview..
Thanks.

[@BigDataThoughts]

Thanks

[@samk_jg]

Amazing!

[@krishnavardhandasari2694]

Thank you mam for excellent way of teaching Spark.

[@BigDataThoughts]

Thanks

[@Azureandfabricmastery]

Simple and easy to understand. Thanks. I like doodle way of explaining concepts :)

[@BigDataThoughts]

Thanks sheik

[@askdoubts6359]

Grate nice explanation

[@minaksheebagul4938]

just love the way u explained it ;) Really appreciated mam.

[@BigDataThoughts]

Thanks minakshee

[@nahomg.4191]

I wish I could give 1000 likes. You’re an excellent teacher!

[@BigDataThoughts]

Thanks

[@ankitachauhan6084]

the best explanation ever great work !

[@BigDataThoughts]

Thanks

[@vikastangudu712]

one of the bests

[@BigDataThoughts]

Thanks vikas

[@vemulasunayana904]

Excellent example 👏

[@BigDataThoughts]

Thanks

[@lakshmikanthavilalakumar]

Beautifully explained short video 👏👏

[@BigDataThoughts]

thanks lakshmikanth

[@adityasisodiya3000]

Amazing content ! Really appreciate the understanding and approach to explain...looking fwd to more

[@BigDataThoughts]

Thanks Aditya

[@ThankGod143]

Extraordinary mam

[@BigDataThoughts]

Thanks harika

[@nareshkumarbattula5847]

It's the best video I've seen so far on spark architecture..awesome..keep going..

[@BigDataThoughts]

Thanks

[@athar5867]

Your videos really help me in clearing my interviews and getting the job, now I need some support for my new job. Can you either post videos regarding how pyspark class object work in backend, how parquet/csv reading writing work in distributed environment like which data read by each executor, how to do pagination of some order data etc.
If possible provide your LinkedIn profile url or suggest some way to connect

[@tanushreenagar3116]

VERY HELPFUL BEST EXPLANATION

[@BigDataThoughts]

Thanks

[@nishchaysharma5904]

Thank you for this video.

[@BigDataThoughts]

Thanks

[@hemanthkumarreddyedde]

Superb delivery

[@BigDataThoughts]

Thanks Hemanth

[@moughosh3640]

Extremely good explanation

[@BigDataThoughts]

Thanks mou

[@sanjoydas007]

Very helpful, it would be great if you can take an example and illustrate how the data chunking happens

[@sheereenhamza3700]

Thank you for such a goooood explanation :D

[@BigDataThoughts]

thanks sheereen

[@peekagyan]

Nice explanation. For 1 GB input data for a batch processing how can we decide how can we decide the cluster size, no. of nodes or no. of executors ? Could you please explain Thanks ma’am

[@neerajmishra6828]

Saw this video.. content looks promising... great job

[@BigDataThoughts]

Thanks

[@nareshb5859]

Very Nice Explanation

[@BigDataThoughts]

Thanks Naresh

[@shivratanmishra1459]

Great work ma'am

[@BigDataThoughts]

Thanks Shivratan

[@SandyRocker]

Thanks a lot mam

[@BigDataThoughts]

Thanks Sandy

[@SandyRocker]

Subscribed ✅

[@vidyaradhakrishnan5611]

Very nice explanation,

[@BigDataThoughts]

Thanks Vidya

[@RaviYadav-cx2pb]

Amazing explanation mam 😊😊👍

[@BigDataThoughts]

Thanks ravi

[@upskillwithchetan]

Great explanation Ma'am, please add more videos and arrange it in seq. under playlist

[@BigDataThoughts]

Thanks Chetan. yes there are more videos coming up. stay tuned

[@mohanmannam]

good explanation...

[@BigDataThoughts]

Thanks

[@iwonazwierzynska4056]

Excelent video :)!

[@BigDataThoughts]

Thanks

[@toandao7113]

This was marked to know that I'm here on 3/10/2023

[@rovashri566]

How did you make such a good visual explanation? Which tool you used to draw sketches ? Pls guide 🙏

[@hlearningkids]

Very nice 👍

[@BigDataThoughts]

Thanks

[@hlearningkids]

@@BigDataThoughts did you explained in this style big query also. ? improvement in this video can be summary in slow way. please dont get hurt because i gave comment. you did really well in video. excellent explanation.

[@shreyamoghe6893]

Great video!
I have one question though. Is it my correct understanding that each student which got the coin bag is same as how data is partitioned.
i.e. 1 student = 1 data partition?

[@BigDataThoughts]

1 data partition is operated on by 1 slot/task

[@Ramakrishna410]

One executor in one core and 2 partitions are assigned so one by one will execute. My que is if 10 tasks are there then these tasks wil execute parallelly or sequentially in partition level

[@BigDataThoughts]

A task operates on a partition of data. Tasks do run in parallel. If you have multiple cores you can specify how many cores will a executor use. The number of concurrent tasks an executor can run is equal to the cores assigned..

[@MrSmarthunky]

Really informative Shreya. One quick question. Stage will run sequentially depending on use case and Tasks will run in parallel?

[@BigDataThoughts]

Thanks Madhu. Stage may run sequentially or in parallel depending upon whether they have dependency or not. Typically a stage will have multiple tasks running in parallel on a different set of data but doing the same set of operations that the stage contains.

[@sandeepmullangi4413]

Nice video. Really liked it. So you said one node can act as driver. I want to know what is the best practise here? I usually submit jobs by doing SSH to master node (atleast in GCP dataproc) and then submit job. So should I consider my master node as driver? Is it right to do that way?

[@BigDataThoughts]

To give an example if we are using Yarn - when you are submitting a spark job in cluster mode. Container where the Application Master runs acts as Master node (driver) and the containers where all the executor process runs the tasks are called Slave Node. When the job gets submitted first the spark submit calls resource manager which in turn starts the application master and from there driver takes over.

[@vedantshirodkar]

Mam, I have one question.
If spark has to write a data to sql database and as our data is broken on to multiple worker nodes, so is it driver who establishes single connection with sql db or it is worker nodes who establishes multiple parallel connections ?

[@BigDataThoughts]

When Spark writes data to a SQL database, it is the driver program that establishes a connection with the database and manages the write process. Each worker node will write its portion of the data to the database through this single connection established by the driver

[@vedantshirodkar]

@@BigDataThoughts Thank You Mam for the elaborated explanation.

[@mdatasoft1525]

❤

[@ur8946]

could you pls explain more on parttiton

[@BigDataThoughts]

The Dataset is divided into partitions and each partition is the unit on which a task works. That's the input to task.

[@trainingt9855]

Can you help in understanding RDD

[@BigDataThoughts]

RDD are resilient distributed datasets and they are the lowest abstraction in spark. check this video - th-cam.com/video/DuWGBMF7ARc/w-d-xo.html

[@RameshKumar-ng3nf]

At the start of the video i was so happy seing all the diagrams..
Later got fully confused & felt complicated and i didnt understand well 😢

[@bintangmuhammad7082]

Can you please turn on the subtitles? thank you

[@SM-mq1iq]

Can you make it slow to follow. I felt this was fast to get to know the terms.

[@sumonmal009]

Good playlist for Spark
th-cam.com/play/PL1RS9FR9qIPEAtSWX3rKLVcRWoaBDqVBV.html

[@BigDataThoughts]

Thanks

[@vibhad-cv4sf]

Great Video....!! Appreciate your efforts!🎉
One question, Where does a cluster manager fit in in this architecture? What role does it play in comparison with driver?

[@BigDataThoughts]

Cluster manager's job is to provide resources for job execution. Ex - yarn, mesos etc. Driver is the one controlling the overall job execution and which executors take part in the job

[@vibhad-cv4sf]

@@BigDataThoughts ohh okay!! Thank you!!

[@harigovindk]

18/april/2024