25. Databricks | Spark | Broadcast Variable| Interview Question | Performance Tuning

#BroadcastVariable, #DatabricksOptimization, #SparkOptimization, #Broadcast, #DatabricksInterviewQuestions, #SparkInterviewQuestions, #DatabricksInterview, #DatabricksPerformance,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@ririraman7]

You should come in the top TH-camrs for Apache Spark PySpark tutorials. Awesome sir, brilliant. Thank You Thank You Thank You....

[@rajasdataengineering7585]

Thanks Ramandeep!

[@prabakaran-g5x]

A Passionate teacher,,,Hats off...Keep updating ...this is like contribution to Indians growth...Heart felt thanks

[@rajasdataengineering7585]

Thanks a ton!

[@shakthimaan007]

Finally found one person who can explain Broadcast variable in a clear and understandable way.
Huge respect bro.
Subscribed and off I go to other videos in the playlist :)

[@rajasdataengineering7585]

Thanks and welcome!

[@shakthimaan007]

@@rajasdataengineering7585 Do you have these notebooks saved somewhere in your git , etc

[@sasuki479]

Wonderful Raja!

[@rajasdataengineering7585]

Thank you! Keep watching

[@sivagssri]

Good job... Keep posting interview questions on Databricks and Spark... I have shared your channel in my group.

[@rajasdataengineering7585]

Thanks Siva...will post interview questions

[@vigneshgaming6286]

Hi sir,will you training on pyspark

[@irannamented9296]

Very useful nice explanations.

[@rajasdataengineering7585]

Glad it was helpful!

[@deepjyotimitra1340]

Thank you for your detailed video.

[@kartikjaiswal8923]

insightful and precise

[@rajasdataengineering7585]

Glad it is helpful! Thanks for your comment

[@roshankumargupta46]

Very useful..keep going!

[@rajasdataengineering7585]

Thank you Roshan

[@chessforevery1]

Great explained

[@rajasdataengineering7585]

Glad it was helpful!

[@swethakulkarni3563]

you are absolutely great!

[@rajasdataengineering7585]

Thank you!

[@RajBalaChauhan-b4w]

Thank you for such clarity. But I have a query - As Catalyst Optimizer will consider the broadcast join itself if a table is small enough to fit in memory, even if we haven't performed any broadcast join. So, is it really going to help us out in performance optimization? Or the performance will remain same only even after applying broadcast join?

[@rajasdataengineering7585]

Catalyst optimiser won't apply broadcast join by default. Either we need to apply manually or adaptive query execution needs to be enabled (AQE is enabled for recent spark versions)

[@gulsahtanay2341]

Good to know!

[@vishalaaa1]

excellent

[@rajasdataengineering7585]

Thank you! Cheers!

[@chidellasrinivas]

Hi Raja, i have few doubts. 1st Doubt - once data is cached in all worker nodes if there is any new records added to dim table. then do we need to broadcast again ?
2nd doubt - Once joining is completed can we clear data from each executors

[@himanshuchourasia8936]

Hi Raja, Could you please also make video on accumulator variable.

[@rajasdataengineering7585]

Hi Himanshu, sure will make a video on accumulator

[@nithinkatla-w6c]

sir, have a doubt broast variable and broad cast join are different or same

[@rajasdataengineering7585]

Both are same

[@sowmyakanduri-t8t]

Hi Raja, it covers only broadcast join part not the broadcast variables part. Please include that part also.

[@AmericaMuchatlu86]

Thank you for your wonderful playlist on Apache Spark. Can you please help on the difference between broadcast variable's and broadcast joins. Both are same?

[@rajasdataengineering7585]

Yes both are same

[@ElhamMirshekari]

Hi, thanks for the videos, can you explain about the checkpoints, what are they ? how they are useful in optimizations?

[@rajasdataengineering7585]

Checkpoint is mainly used in 2 places in spark. One is Spark optimization and another is Spark streaming.
Your question is related to spark optimization. It is quite similar to persist which stores the dataframe in disk. Only difference is persist would retain the lineage but checkpoint would remove the lineage once data is saved to disk

[@ElhamMirshekari]

@@rajasdataengineering7585 Thank you ! Please go ahead and explain the checkpoint in streaming as well, I really appreciate it!

[@rajasdataengineering7585]

Checkpoint is a location in streaming where spark maintains the metadata about processed data such as offset etc.
So when there is a failure in streaming execution, spark can understand till which data it has already processed and from where it needs to resume

[@sohelsayyad5572]

Hiii Raja, Good content !!
table is broadcasted nd stored on all nodes, but at what part of memory, is it on heap memory or off heap memory managed by OS ?
thank you

[@rajasdataengineering7585]

Thanks Sohel!
Its stored within on-heap memory

[@sohelsayyad5572]

​@@rajasdataengineering7585 thanks Raja 👍

[@sohelsayyad5572]

@@rajasdataengineering7585 IF we persist with storage level MEMORY_AND_DISK and offHeap.use enabled true. then data will spill to offHeap or directly to disk ?
Also that Data structure can't be split when its spilling somewhere. what does it mean.
I appreciate your response. thank you :)

[@prathapganesh7021]

Thank you

[@rajasdataengineering7585]

You're welcome

[@rahamanabdul6388]

Good Stuff. Can you please share or create a copy code in git so that we can use for our learning.

[@rajasdataengineering7585]

Sure, will do.

[@ADFTrainer]

it would be great if u provide script