Advancing Spark - Azure Databricks News Apr - May 2024
ฝัง
- เผยแพร่เมื่อ 5 มิ.ย. 2024
- With Data + AI Summit 2024 looming next week, we thought we'd best get everyone up top speed with all the platform changes that have been coming into Databricks over the past two months! Believe us, there's a lot to get through!
In this video, Simon whips through the release notes, pulling out the ones that he thinks are going to see the biggest difference for Databricks users, as well as a few gotchas to be aware of!
For the Databricks Release notes, see: learn.microsoft.com/en-us/azu...
Supposedly there are 4 major types of cluster in Datbricks that are- General Purpose, Storage Optimized, Memory Optimized and Compute Optimized Clusters but I'm not able to find detailed information as on which cluster to choose specifically in which scenarios. I've tried reading from multiple sources but am more confused than ever specially between Memory Optimized and Compute Optimized clusters. I'm talking of worker nodes.
And then there are Graviton Series & Delta Cache enabled clusters!!!!!! 😿😭
Could someone please help me out here explaining in as much detail and as layman as possible the difference between these.
Also, I've a requirement for simple lift and shift of a 9GB Parquet data stored in S3 to Databricks Delta Table in a Unity Catalog with no transformations at all. What should my ideal driver and worker configuration should be?? (I believe taking a smaller driver node and a heavy config storage/compute optimized worker nodes with lot of workers will speed up my process but any info on this would be greatly helpful! 🙏🏽
Type Widening sounds really interesting. I just wish there was a way to allow widening to string, just for those rare cases in which a numeric id becomes alphanumeric one day
Love your videos, they keep me up-to-date.
I want to become an engineer like you.
Working on it coming soon