Hi Simon, I like your videos, they are super useful. Can you make some videos on how to optimize jobs and reduce the performance time or how to investigate when optimization is needed on the job?
Hi Simon, Nice video and is useful. I have a quick question, we are replicating huge data from MSSQL Datawarehouse to Deltalake using DLT(including CDC changes) with continuous mode .As part of that, i have specified my zorder is same as primary key; Does this increases the performance of merge operation in (apply statement) or not.How can i check this performance metrics.
Can you use both partition by and zorder by, on the same column or different columns. And if so, would it be beneficial? Also, why do you enclose spark.read with brackets?
Hey - so you /can/ z-order by a column you've partitioned on, but it'll give no benefit as your data is already sorted into those values by the partitioning! And brackets around the spark statement means you can span multiple lines without needing a line escape '\' for every line!
Great Videos Simon. One suggestion on background ribbons of slides. The ribbons on your slide templates keep moving and are bit uncomfortable to eyes. Request if this can be static
Simon is back!!!! Thank you for this awesome video :) Could you make one explaining how we can profile a spark script in order to identify optimizing tuning opportunities ? I always fo the the Spark UI but I'm completely lost. I know one thing for sure, too much is swapping between nodes is bad news :)!
Oooh, ok, so a quick tour of the Spark UI and "some things to look out for" when diagnosing spark performance problems? I'll add it to the list - need to thing about what the top ones would be or it'll be two hours long! Simon
Yes. You will have to z order on regular basis. And there is no guarantee that only new files will be re-written. Running optimize on big tables which get good size of incremental data can be counter productive.
Thanks Simon for the great video! I'm curious if I would have to achieve Z-ordering in Delta Lake Synapse, how would I be able to? As the Optimize command is only available on the Databricks runtime? Thank you :)
Hey! On the file optimisation level, you could maybe achieve something similar using bucketing - but you wouldn't get the same data skipping benefits. Probably easier to just spin up a databricks cluster over the same data and use that for maintenance jobs (again, Synapse wouldn't do the data skipping part, but your files would be arranged properly) For the indexing/query performance side - Microsoft have been building "Hyperspace", which is an indexing system separate to Delta. This might be the answer for where you can't optimize tables...but it's a very early product, I've not had a go at using it yet! Simon
Hi Simon, I like your videos, they are super useful. Can you make some videos on how to optimize jobs and reduce the performance time or how to investigate when optimization is needed on the job?
Super detail and clean explanation. Love this z-ordering concept.
Hi Simon, Nice video and is useful. I have a quick question, we are replicating huge data from MSSQL Datawarehouse to Deltalake using DLT(including CDC changes) with continuous mode .As part of that, i have specified my zorder is same as primary key; Does this increases the performance of merge operation in (apply statement) or not.How can i check this performance metrics.
Hi Simon - Thanks for this awesome video. One quick question , Do Optimize and Z-order disturb the checkpoint in case of Structured Streaming?
Can I have partition by date and Zorder by event name? Or partition and Z should be same columns?
Can you use both partition by and zorder by, on the same column or different columns. And if so, would it be beneficial? Also, why do you enclose spark.read with brackets?
Hey - so you /can/ z-order by a column you've partitioned on, but it'll give no benefit as your data is already sorted into those values by the partitioning!
And brackets around the spark statement means you can span multiple lines without needing a line escape '\' for every line!
Great Videos Simon. One suggestion on background ribbons of slides. The ribbons on your slide templates keep moving and are bit uncomfortable to eyes. Request if this can be static
Hey Simon .. Great explanation with proper use snd demo
good explanation! thanks.
Simon is back!!!!
Thank you for this awesome video :) Could you make one explaining how we can profile a spark script in order to identify optimizing tuning opportunities ? I always fo the the Spark UI but I'm completely lost. I know one thing for sure, too much is swapping between nodes is bad news :)!
Oooh, ok, so a quick tour of the Spark UI and "some things to look out for" when diagnosing spark performance problems? I'll add it to the list - need to thing about what the top ones would be or it'll be two hours long!
Simon
Advancing Analytics You’re the real MVP Simon! TY!!
Awesome-ly Explained !!!!
Can you share the NYTaxi notebook?
In production environments, do we have to zorder the tables weekly to improve performance?
Yes. You will have to z order on regular basis. And there is no guarantee that only new files will be re-written. Running optimize on big tables which get good size of incremental data can be counter productive.
1st :)
Thanks Simon for the great video! I'm curious if I would have to achieve Z-ordering in Delta Lake Synapse, how would I be able to? As the Optimize command is only available on the Databricks runtime? Thank you :)
Hey!
On the file optimisation level, you could maybe achieve something similar using bucketing - but you wouldn't get the same data skipping benefits. Probably easier to just spin up a databricks cluster over the same data and use that for maintenance jobs (again, Synapse wouldn't do the data skipping part, but your files would be arranged properly)
For the indexing/query performance side - Microsoft have been building "Hyperspace", which is an indexing system separate to Delta. This might be the answer for where you can't optimize tables...but it's a very early product, I've not had a go at using it yet!
Simon
hey Thanks its a great video.
Brilliant explanation, well done!
excellent
Thank you very much