Emil, finally I have fount one comprehensive tutorial about Spark architecture. I have subscribed your channel. Thanks a lot for your unvaluable support!
I just want to express my gratitude for this video, Emil. It's the first time I feel like I truly understand it, and it feels much less obscure now. I really appreciate the part about wide transformations, data shuffling, and why they are computationally intensive. I had a big "aha" moment when you described the process of saving data from memory to disk and then sending it over the network to another worker. Now it makes perfect sense why it is so slow. Pozdrawiam, Teodor
Amazing content! You've done absolute justice to explaining Spark! The breakdown of concepts is spot on and super easy to follow. Keep up the great work! Thank you!
My sincere thanks for this tutorial. It was of great help. I wish to see more videos from you. Keep up the good work.❤ The way you have explained things, I don't think any other tutorial has explained it with such simplicity and adeptness.
I came across your video while searching for spark architecture, and it's incredibly well explained. The content is delivered at a great pace, not overly complicated but perfect for a beginner. I'm expecting more videos like this in the future. great job
Very well planned and explained in details. It really helps me to uncover most of the doubts and gives me clear direction of how to and where to look as what is happening in background when a query is executed. Subscribing your channel for knowledge enrichment.
Thank you for the detailed explanation and well-crafted examples. It's rare to come across such high-quality content. Could you please share the notebooks containing these examples? Also, when can we expect a dedicated video on optimization techniques?
Thanks for the great explanation! It would be perfect to also hear about your production deployment experiences in Databricks Delta Lake projects, including CI/CD, deploying bundles, monitoring, etc.
Hi! Thanks … indeed super thanks for such learning session on Spark. I wonder how many months of learning and understanding been put into 1 hour! I recomend this to all #databricks users to have this understanding of #spark to make life easy with the #development of #datapipeline or #dataanalytics or #dataanalysis or #optimizing your #codes. Thanks !!
Hey Emil, I wanted to improve more on Databricks, PySpark and other essential technologies on a professional level. After watching your video, I felt I could reach out to you for further guidance to streamline my preparation. I wish you had more videos already in this channel, which would have helped me as a one stop get everything place. But given all this, Can you reply to this comment with some study references which you feel are really good that I can use to become better in these Data technologies.
The better way to learn and keep those information long term are practice and self notes. Treat this video as a source for learning and make your own notes.
52:47 here group by and then count is happening , but could you please tell that will happen when join operation instead of group by I'm totally confused.
query regarding cluster configuration -> while cluster configuration for example we specify node capacity as - 64 gb memory (it is RAM) - 8 cores how spark allocates -> storage (disk) ? from this 64 gb how it allocates for -> ram + disk?
Great video, Please im requesting you please where i get all this 5GB data please provide me link 🙏🙏🙏 I want to use your data which hii have used so i understand eusly 🙏 please provide all 5Gb data
An hour felt like a few minutes thanks to your teaching skills, thank you for this tutorial.
One of the best Spark introduction tutorials I have come across!
Thanks so much for putting this together
Emil, finally I have fount one comprehensive tutorial about Spark architecture. I have subscribed your channel. Thanks a lot for your unvaluable support!
Pleasure on my side. I am very happy you like it and find it useful!
Hands down the best tutorial. please provide an indepth tutorial on spark optimization mentioned in 1:04:07
Never understood the architecture so well.
Thanks !!!
The best spark tutorial on youtube!!
Thank you!!
I just want to express my gratitude for this video, Emil. It's the first time I feel like I truly understand it, and it feels much less obscure now. I really appreciate the part about wide transformations, data shuffling, and why they are computationally intensive. I had a big "aha" moment when you described the process of saving data from memory to disk and then sending it over the network to another worker. Now it makes perfect sense why it is so slow.
Pozdrawiam,
Teodor
Amazing! Such feedback means really a lot and motivates for furthrer work. Thank you so much.
Pozdrawiam
this is amazing, !! thanks alot, better than anything else i ever seen
Amazing content! You've done absolute justice to explaining Spark! The breakdown of concepts is spot on and super easy to follow. Keep up the great work! Thank you!
Thanks for feedback. I am really happy to see this being useful!
Amazing tutorial, I know so much about how spark works!!
My sincere thanks for this tutorial. It was of great help. I wish to see more videos from you. Keep up the good work.❤
The way you have explained things, I don't think any other tutorial has explained it with such simplicity and adeptness.
I came across your video while searching for spark architecture, and it's incredibly well explained. The content is delivered at a great pace, not overly complicated but perfect for a beginner. I'm expecting more videos like this in the future. great job
Thank you for feedback! Video about Delta Lake coming in 1-2 days. Hope you will like it as well.
@@DatabricksPro can u plz provide the link to ur git repo for the code
The best way to explain the Spark in action. great tutorial..
Very well explained and best part was to include examples for all the concept and explaination. Thanks for such great efforts.
Thanks! I am very happy that you liked it:)
This channel will explode in due time, great job! Looking forward to more such videos! (hopefully on spark optimisations soon). Thanks a lot!!
Thanks! Next week there will be a Delta Lake movie with a bit of optimizations approach.
That was really a great explanation, and covering each topic indeeply, I dont see anyone talk about sparkUI and the pwoer of it. Thanks really again
Thank you sou much for this informative tutorial. The best there is out there yet.
absolutely fantastic, you went through all the stages. Thank you!
You nailed it. Super useful video!!!
best one ever seen , thanks a lot :)
really good content, right on the money! thank you for your effort!
Great tutorial, thank you.
Best video on spark Architecture
Thanks a lot for the explanation
Thanks for this!!
Too good explanation!!
You nailed it perfectly Dimitri 😇
Very well planned and explained in details.
It really helps me to uncover most of the doubts and gives me clear direction of how to and where to look as what is happening in background when a query is executed.
Subscribing your channel for knowledge enrichment.
I happy it helps. Check out Delta Lake vid as well :)
@@DatabricksPro Sure. Thanks a lot.
Amazing tutorial!
Glad you think so!
🤙🤙💥Worthhhhhh spending time on this video. feels like zero to Hero
Spark 4.0 preview is out, would love to hear your take on it, great lecture!
Hi. This is a great idea! Thank you for that. I will take a look on it next week. Cheers!
@@DatabricksPro looking forward to it!
Brilliant content, thank you!
You are a legend 🔥🔥
Thank you for the detailed explanation and well-crafted examples. It's rare to come across such high-quality content. Could you please share the notebooks containing these examples? Also, when can we expect a dedicated video on optimization techniques?
so good ,Respect Man !!
Thanks for the great explanation! It would be perfect to also hear about your production deployment experiences in Databricks Delta Lake projects, including CI/CD, deploying bundles, monitoring, etc.
Yes. I also think its a good idea. it will come
Great one 😇
Hi!
Thanks … indeed super thanks for such learning session on Spark. I wonder how many months of learning and understanding been put into 1 hour! I recomend this to all #databricks users to have this understanding of #spark to make life easy with the #development of #datapipeline or #dataanalytics or #dataanalysis or #optimizing your #codes.
Thanks !!
Hey Emil,
I wanted to improve more on Databricks, PySpark and other essential technologies on a professional level. After watching your video, I felt I could reach out to you for further guidance to streamline my preparation. I wish you had more videos already in this channel, which would have helped me as a one stop get everything place.
But given all this,
Can you reply to this comment with some study references which you feel are really good that I can use to become better in these Data technologies.
thanks for your efforts
Wonderfully explained!!! I would it if you would have also provided Slides and Notebooks if possible?
The better way to learn and keep those information long term are practice and self notes. Treat this video as a source for learning and make your own notes.
52:47 here group by and then count is happening , but could you please tell that will happen when join operation instead of group by I'm totally confused.
Thank you so much for the great videos. Can I get access to the notebooks as well? Appreciate !!!😊
Thanks :) i am on business trip, but will upload in 2 days once back
AMAZING
Btw. Maybe you will also like the latest movie, about Delta Lake. Cheers
36:46 stopping here midway
Great work and thanks
Could you please make a similar video like this on delta lake
I am glad you like it :) Delta lake is actually really good idea for the next movie.
Thank you :)
You are welcome!
query regarding cluster configuration ->
while cluster configuration for example we specify node capacity as
- 64 gb memory (it is RAM)
- 8 cores
how spark allocates -> storage (disk) ?
from this 64 gb how it allocates for -> ram + disk?
Spark is legacy
That is a good idea for the movie :) thanks
@@DatabricksPro i mean Scala 2.13 and Java 11
Great video, Please im requesting you please where i get all this 5GB data please provide me link 🙏🙏🙏
I want to use your data which hii have used so i understand eusly 🙏 please provide all 5Gb data
Jiga bytes , airdd
i finished adv data eng plan in partner academy. but ur course is pretty good too
i’ve watched many videos and tutorials, none of them are in detailed. Awesome work🫡