22. How to select Worker/Driver type in Databricks?

1. Usually, drivers can be much smaller than the worker nodes.
2. More cores for your DBUs, is more parallelism per DBU (but on smaller partitions because of less memory/CPU)
3. Allows for data to be split into bigger == fewer partitions to allow less shuffling.
4. Storage Optimized is especially relevant for queries with many joins/groupBy’s.
5. Choose Memory Optimized for less intensive ETL tasks.
6. Use Compute Optimized for more Parallelism and iterative tasks.
7. Do not choose HDD Clusters as they are less performant.
Connect with me on Linkedin
/ bhawna-bedi-540398102
Instagram
www.instagram....
Data-bricks hands on tutorials
• Databricks hands on tu...
Azure Event Hubs
• Azure Event Hubs
Azure Data Factory Interview Question
• Azure Data Factory Int...
SQL leet code Questions
• SQL Interview Question...
Azure Synapse tutorials
• Azure Synapse Analytic...
Azure Event Grid
• Event Grid
Azure Data factory CI-CD
• CI-CD in Azure Data Fa...
Azure Basics
• Azure Basics
Data Bricks interview questions
• DataBricks Interview Q...

[@ayushsrivastava6494]

Honestly, I've been seeing your tutorials from a long long time and nothing beats the way you teach! You legit are the best out there. 🙌🏼

[@ajinkyadhoke4713]

One of the finest video Bhawana....more power to you...🔥🔥🔥🔥

[@naveenkuppili2889]

Love the way you explain, appreciate if you can put up videos on performance tuning (reading plans, DAGs, repartition vs coalesce ) etc. Thanks again.

[@cloudfitness]

Hi Videos on few of those topics are already posted here:- th-cam.com/video/sIlayzDrP48/w-d-xo.html th-cam.com/video/bKAneQeQe7s/w-d-xo.html th-cam.com/video/a2ehHq3DJrw/w-d-xo.html

[@ramaraju3273]

Awesome 👍👍👍... very informative video as always... keep up your great work... thank you.

[@Amin-tx4ku]

Never came across such an elegant explanation.Thank you.

[@tanushreenagar3116]

Best explanation 👌 ever it cleared 👏 many thing's. Thanks a lot.

[@kartikeshsaurkar4353]

Very informative ! Thanks for providing the details with real time scenarios

[@PavanKumar-tt8mm]

I am exploring more knowledge from you on Databricks. I am waiting for another concept from you.

[@deepjyotimitra1340]

Very useful information.

[@muthumulaprasanna]

Great explanation.. One question how to select memory? Like 14 gb Or 24gb etc

[@Akshay50826]

A big thank you☺️

[@shubhamaggarwal3676]

It would be nice if you make a video i.e. on how to choose optimal no of resouces like shuffle partitions, repartition and worker nodes + how to optimise the jobs by analysing spark ui

[@AmitKumar-kh4ht]

Very nice 👌

[@Itachi_88mm]

Delta cache also stores data from remote location to local node so that data is read from local node to avoid network traffic then how different is it from Broadcast variable kindly help us understand this difference

[@MohammedKhan-np7dn]

Hi Bhawana, Thank you alot for the knowledge sharing session. Can you share the link to go through the ppt for our reference.

[@nakkaeswaraoeswar2140]

Thanks a lot beauty.....Can you explain about what is Azure services and Azure Sql

[@pkrmettu28]

Can you please teach power bi and python real time scenarios covering basics and interview?!