66. Databricks | Pyspark | Delta: Z-Order Command

Azure Databricks Learning: Delta Lake - Z-Order Command
========================================================
What is Z-order Command in delta table and how to apply in delta lake development?
Z-order one of the performance optimization techinique used in delta lake. It is used along with optimize command and used to compact small files into optimal size and at the same time relevant data is co-located to improve the performance.
This video gives complete understanding of Z-order command
#DeltaZorder, #DatabricksZorder, #PerformanceOptimization, #Zorder,#Z-order, #Z-Ordering, #DeltaOptimize, #DeltaOptimizeZorder #DeltaCompactFiles, #DeltaSmallFileIssue, #DeltalakePerformance, #DeltaPerformanceImprovement ,#DeltalakeIntro, #IntroductionToDeltaLake, #Deltalake, #DeltaTable, #DatabricksDelta, #DeltaTableCreate, #DatawarehouseVsDataLakevsDeltaLake, #PysparkDeltaLake, #DeltalakevsDatalake, #SQLDeltaTable, #DataframeDeltaTable,#DeltaFormat ,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@shreeyashransubhe2537]

Sir, I have gone through lots of videos but never understood the concepts so simple yet very detailed manner. Thank you very much. I have shared your playlist with my colleagues too. They also liked it very much.

[@rajasdataengineering7585]

Thank you for your valuable comments. Really appreciated

[@pratikparbhane8677]

Great Explain , Understood OPTIMISE , VACCUM() AND Z-ORDERING in One Video

[@rajasdataengineering7585]

Glad it was helpful!

[@rohitdanda]

Your videos are so simple that a kid can also understand. Thanks and salute sir🖖 for putting so much effort and making videos and helping us!

[@rajasdataengineering7585]

Thanks for your comment. Glad to know it helps data engineers

[@SumitAmbatkar]

i watched your nearly all playlist i loved your teching style, how to ogrip on concept, your explaination are fabulous keeping doing sir, best of luck. we are always here for you
Thank you..:)

[@rajasdataengineering7585]

Thank you,Sumit! Keep watching

[@RanjeetkumarYadav]

Amazing and very intuitive example. Thank You!!

[@rajasdataengineering7585]

You're very welcome! Keep watching

[@YogeshBiguvu2208]

Excellent explanation with Examples.....Thank you so mcuh sir..

[@rajasdataengineering7585]

You are most welcome! Glad it helps

[@mukilanlakshmanan8968]

Sir, I love your teaching method, you have explained it in detail.

[@rajasdataengineering7585]

Thanks Mukilan! Glad to hear that

[@user-hj2nv8gt4o]

Sir, Thanks for explaining in a very simple manner.

[@rajasdataengineering7585]

Thanks and welcome

[@ajaykiranchundi9979]

A very well explained . The way you broke down the data to explain the same is amazing. I am sure it would have taken good time to put it together. Indebted to you brother.

[@rajasdataengineering7585]

Thanks Ajay👍🏻

[@saurav0777]

Thanks for uploading . Very nice explanation

[@rajasdataengineering7585]

Thanks

[@sraoarjun]

Indeed an awesome video !! Great explanation !!

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@3a8saisamireddi61]

detailed explanation👏

[@rajasdataengineering7585]

Thank you 🙂

[@vivek05117gece]

very well explained. Kudos to you.

[@rajasdataengineering7585]

Glad it was helpful!

[@shwetac2929]

you teaching methos is very good ....this video clear my all doubt

[@rajasdataengineering7585]

Glad to hear that

[@terrificmenace]

Thank you sir 🙏🏻 I went through many udemy courses but never understood these concepts. Ur explanation is very good and easy to understand many many thanks sir 🙏🏻 🙏🏻

[@rajasdataengineering7585]

Thank you 👍🏻

[@FreakONcW1]

Extremely helpful video.

[@rajasdataengineering7585]

Thanks Kinjal! Glad to know it was helpful!

[@dineshwaditake5248]

Nicely explained !!

[@rajasdataengineering7585]

Glad it was helpful!

[@tanushreenagar3116]

Very nice sir 👌 cleared my concept now

[@rajasdataengineering7585]

Thank you

[@ravulapallivenkatagurnadha9605]

Please continue this videos

[@AFSARAHMED4]

Excellent Explaination Sir

[@rajasdataengineering7585]

Thanks

[@gil.0007]

Very nicely explained 🎉

[@rajasdataengineering7585]

Thanks, glad it was helpful!

[@omprakashreddy4230]

Your videos are definitely creating great impact. Thank you for that.
Can you also please explain df.explain() command in great detail with examples.

[@rajasdataengineering7585]

Happy to hear that it's creating impact on data engineers. Thank you
Sure, will post a video on explain plan

[@rajasdataengineering7585]

Hi Omprakash, created a video on explain plan as per your request. Hope it helps you - th-cam.com/video/6NrVQTbkndU/w-d-xo.html

[@viniciusguimaraessantana5455]

thank you very much.

[@rajasdataengineering7585]

You are welcome!

[@manjit_singhh]

Very nice explanation 🙂

[@rajasdataengineering7585]

Thanks

[@mohitupadhayay1439]

Raja please try to create a full project where all these optimizations can be shown at full scale.

[@rajasdataengineering7585]

Sure Mohit, will do!

[@tanushreenagar3116]

PERFECT CONTENT SIR

[@rajasdataengineering7585]

Thanks Tanu!

[@venkatasai4293]

Thanks for the great explanation Raja. So are the statistics collected on all the columns ? What if we want to query on other columns ? Will it work ?

[@rajasdataengineering7585]

Yes Venkata, it will work first 32 columns. If your table contains more than 32 columns and you want to collect statistics for those columns, we can configure that separately

[@venkatasai4293]

@@rajasdataengineering7585 ok . So zorder is similar to bucketing right ? Colocating the data into same set of files ? If two tables contains same key and if we zorder them on the key While joining the data it will fetch only required files into the executor ?

[@sathyahisto]

good Explaination, liked it when you demonstrated with excel. Just one suggestion syntax for zorder seems to be changed to "Zorder by ()"

[@rajasdataengineering7585]

Yes, you are right. Thanks

[@sumiransinha3707]

Great!

[@rajasdataengineering7585]

Thanks

[@shankar1556]

Hi Azar,
Thank you for explanation.
I have a dought. in this example it shows that z-order create new partitions with sorting emp_id. Does z-order really create new partitions?

[@rajasdataengineering7585]

Hi Shankar, this is Raja.
When we perform z-order, data is being co-located within same set of files. It is not shuffling the data, nor creating new partitions

[@TheDataArchitect]

What about using multiple columns in z-order?

[@purnimasharma9734]

Hi Raja, how is the partition column determined e.g. how does it know that you have to use emp_id here? Is it based on the predicate column?

[@purnimasharma9734]

Never mind, when I watched your video completely, I found out.

[@rajasdataengineering7585]

Great

[@ravulapallivenkatagurnadha9605]

Please do video on how to convert pandas data pipilines to spark data pipiy

[@rajasdataengineering7585]

Sure will do

[@aswaniyettapu9992]

Can u do one video on lead and lag in pyspark..?

[@rajasdataengineering7585]

Sure, will post a video on lead and lag very soon

[@rajasdataengineering7585]

Hi Aswani, have posted a video on lead and lag function today as per your request

[@aswaniyettapu9992]

Tq so much