Not sure why your channel does not show while searching pyspark tutorial. I spoke to a developer on linkedin and he suggested me your channel. Great work thank you Sir!
@@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks
This series and explanation is too good than other channels and I still don't know why this channel is not showing any recommendation when we search for databricks vidoes.
Thank you so much for putting such a great effort. I haven't gone thru all videos yet, but i can definitely imagine the hard work that you must have put behind this playlist.
I have searched lots of videos regarding spark arictecture and working but this is videos is awesome I realy appreciate for this video nice presentation and i understand very clearly complete concepts thank you so much🙏🙏
Very good explanation,I watched all your playlist gain knowledge about spark and writing code also,I hope to do more vedios on spark , I'm requesting you to upload vedios with subtitles too so we can make a note of the entire session, please add subtitles too for you old vedios.
Sir can you please give us the links of the notebooks used in this series. This has helped me and a lot of other people. Giving this link would be even more helpful 🙏🙏🙏
Hi Raja, One of the amazing explanation. I couldn't find these level of details in any of the source like - books, medium and other youtube. Amazing work !!. Could you share if there are any resource helped you for spark >
I have installed Jupyter with Pyspark...where should I run my command to see the Spark job like you are showing on 38:21sec? or should I install some other IDE for that?
hi, i am new to spark and your videos seem like a great resource to learn. i am wondering what is the best order to watch them? through the playlist pr just use the numbers like 1,2,3...
Thank you Raja for the detailed explanation. Do we have any video which is focusing on Worker Node and its details ? And as you were about to make a video regarding the memory management details, please also share that or the video title if already present. Thank you so much in advance!
Hi Raja, I have a question here, in terms of partitions when we will be loading a file of 2gb the minpartitionbyte of 128 mb makes the initial partitions to be 16 with the logic 2*1024 / 128 right ? and the minpartition property has a number 200, but isnt it that if there is any shuffle operation then only 200 partitions will be there but not while reading
I'm following this channel from couple of days now. The content and way of explanation is awesome. Good job my friend. keep up the good work. wishing you all the very best. one small suggestion, if possible, please try to avoid the initial music (which is played at the beginning of the video) at times it is annoying. thank you!
Excellent job. Can you please provide me the data set and code? Or please give me the Git link to download the dataset and code for your tutorials. Thanks.
Hi, here the executor mean processor (CPU) , Right? Because each node may have multiple CPU, and also each CPU can have multiple cores. So each cores has a separate executor.
@@rajasdataengineering7585 I went through complete video second by second. Video has lot of clarity than any other TH-cam channel. Keep up good work!! Have you experienced data skew issue, if yes can you point video or do video for us.
Thank you for your kind words. It gives a motivation to create more videos which can help genuine knowledge seekers like you. For data skew, have posted one video (though it does not cover advanced concepts) th-cam.com/video/EQhldyLWPwI/w-d-xo.html
Hi Raja u r teaching is awesome .... really help .. sir, just wanted to know .. for "STAGING" here u mentioned about "Nodes" .. here "Nodes" means No of worker Nodes or Partitions ??
Sir you said when read CSV file it's divided in default 200 partitions but default size of block is 128MB so it should decide into 16 partition please let me know am I correct or not ?
I do see it as more detailed explanation on spark , but not really on Databricks, missing many Databricks features like Unity catalog, DBFS, Vaccum, Liquid clustering etc..
Hi Ellie, I have already created videos for join and union. Will make a video for concat as per your request. Join : th-cam.com/video/nJGjFMPBlTg/w-d-xo.html Union: th-cam.com/video/FTTLMBLizV8/w-d-xo.html hope it helps you
Hi Raja, How do we set the number of executors in azure databricks ? like in this example the worker node is divided into 4 executors. Thanks in advance!
Hi, number of executors can be controlled using spark config parameter "spark.executor.instances". Number of cores per executor can be set by spark.executor.cores. Hope it helps
@@rajasdataengineering7585 thank you, so these parameters have to set while cluster creation meaning are these parameters at cluster level or can be changed/set by developers during etl/data processing?
Thank you ver much for such contents.. one request - Can you please make a video on real time executor number, core, memory allocation based on input data size like. 1. 1-5 gb 2. 5-15 gb 3. 15-25 gb 4. 25-50 gb 5. > 50gb = 1 tb
@@rajasdataengineering7585 yes but which playlist to follow as there are 7 playlists in the channel so it's Lil confusion. Say for example a playlist has 5 videos but when I open it I can see 17, 18, 19, 20 in the videos. In interview series it starts with 1 again
Not sure why your channel does not show while searching pyspark tutorial. I spoke to a developer on linkedin and he suggested me your channel. Great work thank you Sir!
Glad to hear it helps you! Thanks for visiting my channel
This is an absolute masterpiece on introduction of Spark and all it's internal structure.
Thank you for such a detailed video.
Thank you Abhinav👍🏻
@@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks
Can beginners learn these?
Simple. Clear . To the point stuff. Thanks. Love your series.
Glad you like them! Thanks for your comment
@@rajasdataengineering7585 Sir I would like to learn databricks fully .Please giude me
Pls go through all videos in this channel. You can learn databricks thoroughly
This series and explanation is too good than other channels and I still don't know why this channel is not showing any recommendation when we search for databricks vidoes.
Thank you 👍🏻
simple straight to the point absolute master class
i was searching for this for long time no one taught it this easily
thank you for this
Glad it helps! Thanks Prashanth, for your comment
I really appreciate the way you have explained the difference between in memory computation and using external system.
Thank you Ravi
Yes great explanation
@@rajasdataengineering7585hi may you share PPT and databricks file of the course
Thank you so much for putting such a great effort. I haven't gone thru all videos yet, but i can definitely imagine the hard work that you must have put behind this playlist.
Thank you so much!
I have searched lots of videos regarding spark arictecture and working but this is videos is awesome I realy appreciate for this video nice presentation and i understand very clearly complete concepts thank you so much🙏🙏
Glad it was helpful! Thanks for your comment
Thanks for the videos. Your videos really helped me to switch my job. Thanks for the great content. Your explanation is awesome. Thanks again.
Great to hear! Thank you so much!
Just started but I love clear and simple explanation, thanks a lot for your efforts
You're very welcome! Glad it helped
Best intro of spark which i've seen till now.......
Thank you
To the point and crisp yet detailed explanation, I've seen in a while, thank you so much!
Glad it was helpful!
you have put in a lots of effort and tried to cover all important points. thank you very much for your immense contributions
My pleasure! Thank you for your comment
Thanks Sir for Wonderful Explanation and provided in-dept Knowledge of Spark Architecture. Wonderful Resource for start SPARK Journey.
Thanks and welcome
Superb , Fantastic , Marvellous.....What a great teacher you are .
Thank you so much! Glad it helps
Yes with clarity you have explained architecture and specially the partitioning with diagram was really helpful.@@rajasdataengineering7585
Hi sir do you conduct any online coarses
Very well explained Raja, thanks for making this series brother.
Thanks Nasser
Appreciate your detailed explanation Sir... Really helpful
Glad it is helpful. Thanks for your comment
excellent exp;anation across all the youtube channels thanks
Much appreciated! Thanks for your comment
Just starting to watch yr playlist with the hope to learn spark, lets see how it goes. BTW thaks for the complete playlist mate!
Hope you enjoy it!
this explanation is great , very detailed
Thank you!
Raja, this is excellent way of explanation .
Thank you, Vydu!
when you mention referring the other videos, can you also keep mentioning those links in description. Thanks a lot for your explanation!!
Sure thing! Will add links
for sure best video ever seen before... Raja Great Presentation
Glad you liked it! Thanks for your comment
Thank you for great and fantastic master pieces
Thanks for listening!
What an explanation, thank you so much sir.
You are most welcome
Thankyou very much for the content Sir.
Glad it helps you gaining insight about spark internals!
Amazing explanation sir, I am able to understand it very clearly
Thanks. Glad you enjoyed this content!
@@rajasdataengineering7585 hi sir can i talk to you ..can i get ur contact num plzzz
Hi Raja, This is a great explanation. Appreciate your hard work.
Thank you!
Thanks alot for in depth explanation😊
Hope it helps! Thanks and welcome
Hi Raja, I was very much confused with the architecture, once I saw your video ,now its clear, Keep up the good work
Thanks Shekar!
Very good explanation,I watched all your playlist gain knowledge about spark and writing code also,I hope to do more vedios on spark , I'm requesting you to upload vedios with subtitles too so we can make a note of the entire session, please add subtitles too for you old vedios.
Thanks Naveen! Sure, will try to add subtitles
@@rajasdataengineering7585 Thanks,that will be great
Is this all tutorials related to spark only? Or it includes data bricks aswell? Please confirm
It's more on databricks which is internally using apache Spark
It is really helpful. Thank you.. Where will I get the complete PPT of this playlist?
Hi Raja, very nice explanation ...
Are you taking classes
Hi Lakshmi, thanks for your comment! I'm not taking classes
Thank you for making vedios
Actually helped a lot.....
Thanks Harsha, for your comment!
your explanation very excellent
Glad it was helpful!
Exactly what am looking for. Please let me know where I can find the datasets to practice.
Great video Raja!! Explained very well..Thanks
Thank you
Good explanation! keep up the good work.
Thank you
Yes, this is master piece. Thanks
Welcome, Glad you like it!
Could you please provide the slides used in all the lectures. This will be super useful. Thank you for this master pieces!!.
Watch in full screen, and take screenshots bro..
Excellent video 😀 thank you
nice pictorial representations bro keep gng
Thanks Anand
Sir can you please give us the links of the notebooks used in this series. This has helped me and a lot of other people. Giving this link would be even more helpful 🙏🙏🙏
Yes, please
Hi Raja, One of the amazing explanation. I couldn't find these level of details in any of the source like - books, medium and other youtube. Amazing work !!. Could you share if there are any resource helped you for spark >
Thank you! I don't have any other resources. I summarised these concepts based on my working experience
I have installed Jupyter with Pyspark...where should I run my command to see the Spark job like you are showing on 38:21sec?
or should I install some other IDE for that?
hi, i am new to spark and your videos seem like a great resource to learn. i am wondering what is the best order to watch them? through the playlist pr just use the numbers like 1,2,3...
Hi, for all videos, I have given serial number. You can follow the order based on that serial number
great explaination sir
Thanks! Hope you find it helpful
Just starting to learn spark and databrics. Is this resource is enough to crack entry level data engineering job?
Yes definitely, these videos are more than enough to crack entry level job
Thank you Raja for the detailed explanation. Do we have any video which is focusing on Worker Node and its details ? And as you were about to make a video regarding the memory management details, please also share that or the video title if already present. Thank you so much in advance!
Please watch videos
th-cam.com/video/cTjHokox5Is/w-d-xo.html
th-cam.com/video/A80o9WGXK_I/w-d-xo.html
Awesome!! Could you please attach the PPT for Each Video.. Thanks
Great explanation.
Glad it was helpful!
Excellent video
Thank you very much!
Hi Raja, I have a question here, in terms of partitions when we will be loading a file of 2gb the minpartitionbyte of 128 mb makes the initial partitions to be 16 with the logic 2*1024 / 128 right ?
and the minpartition property has a number 200, but isnt it that if there is any shuffle operation then only 200 partitions will be there but not while reading
Yes, that's shuffling partition parameter which is not applicable for reading partition
So you mentioned 200 in your example that's why my doubt arose
Thanks for the content!
My pleasure! Welcome
I'm following this channel from couple of days now. The content and way of explanation is awesome. Good job my friend. keep up the good work. wishing you all the very best.
one small suggestion, if possible, please try to avoid the initial music (which is played at the beginning of the video) at times it is annoying. thank you!
Hi Shiv, thank you for your valuable comments.
I already removed this initial music. May be it is still there for only few initial videos.
Great video raja!! Wanted to know how you have such in depth knowledge.. did you learn from spark docs ?
Thanks Morgan. Yes spark documents and working experience helped me to understand concepts
Excellent job. Can you please provide me the data set and code? Or please give me the Git link to download the dataset and code for your tutorials. Thanks.
Hi, here the executor mean processor (CPU) , Right? Because each node may have multiple CPU, and also each CPU can have multiple cores. So each cores has a separate executor.
Executor means logical division of nodes. That means combination of processor + memory + network
Well Explained
Thanks
perfect video sir
Thank you!
Very good course🎉
Glad you think so! Thanks
Excellent
Thank you! Cheers!
Very Nice
Thanks
Started your videos!! All are great
Thank you Ninja
@@rajasdataengineering7585 I went through complete video second by second. Video has lot of clarity than any other TH-cam channel. Keep up good work!!
Have you experienced data skew issue, if yes can you point video or do video for us.
Thank you for your kind words. It gives a motivation to create more videos which can help genuine knowledge seekers like you.
For data skew, have posted one video (though it does not cover advanced concepts)
th-cam.com/video/EQhldyLWPwI/w-d-xo.html
@@rajasdataengineering7585 thanks a lot again Raja!! Will go through it. I am looking for airflow training, I have sent mail to you, kindly respond.
Hi Raja u r teaching is awesome .... really help ..
sir,
just wanted to know .. for "STAGING" here u mentioned about "Nodes" .. here "Nodes" means No of worker Nodes or Partitions ??
Thank you Arup.
Node means no of worker nodes in the cluster
@@rajasdataengineering7585 thanks raja ..💐🌸
How was 50 Partitions per executor has been calculated Raja?
Nice explanation...
good explanation
Thanks for liking
Sir you said when read CSV file it's divided in default 200 partitions but default size of block is 128MB so it should decide into 16 partition please let me know am I correct or not ?
Are any prerequisites required to this pyspark series?
No, nothing needed. I have covered from basic
Thank you!
there are some videos missing? like 27 , 28, 29 etc?
I do see it as more detailed explanation on spark , but not really on Databricks, missing many Databricks features like Unity catalog, DBFS, Vaccum, Liquid clustering etc..
Soooo good explanation
Glad it was helpful!
Hi Do you offer any course private in detail? with real time project
Hi , is there any email to directly reach out for questions ?
Hi Raja ,please do azure synapse analytics,eagerly waiting
Sure Krishna, will create videos on synapse analytics
In order to store data in JVM memory we need to do serialization and deserialization. Why ?
Hi Raja,I want to learn databricks for azure data engineer. Will this playlist help me for interview??
Hi Sravanthi, yes this playlist definitely helps you to perform well in interview
Great Video! 👍
Thanks 👍🏻
Is the master node same as cluster manager? or they are two different concepts?
Master node is driver and different from cluster manager
@@rajasdataengineering7585 Thanks for the prompt response .
Master node = Driver
Can we create multiple executor node on a worker node?
Yes we can. Executor is logical division of computing resources
Yun number of worker nodes?? What is Yun or yum??
HI Raj, Videos are Missing from 27 to 30 Could you Please Upload them.
Hi Srikanth, those 4 videos are related to Azure Synapse analytics. Its still available under all videos section
i need prepare for apache spark programming with databricks certification,will this videos help me?
Yes, it will help
Nice info
Raja could you kindly make a video on these three functions and compare them: Join, Union, Concat
Hi Ellie, I have already created videos for join and union. Will make a video for concat as per your request.
Join : th-cam.com/video/nJGjFMPBlTg/w-d-xo.html
Union: th-cam.com/video/FTTLMBLizV8/w-d-xo.html
hope it helps you
can I have document for reference of this playlist
Hi Raja,
How do we set the number of executors in azure databricks ? like in this example the worker node is divided into 4 executors.
Thanks in advance!
Hi, number of executors can be controlled using spark config parameter "spark.executor.instances".
Number of cores per executor can be set by spark.executor.cores.
Hope it helps
@@rajasdataengineering7585 thank you, so these parameters have to set while cluster creation meaning are these parameters at cluster level or can be changed/set by developers during etl/data processing?
It can be set at cluster level using init scripts or at notebook level using syntax spark.config.set()
@@rajasdataengineering7585 thank you 🙏
Welcome
This is the masterpiece
Thanks Ajay
Thank you ver much for such contents.. one request -
Can you please make a video on real time executor number, core, memory allocation based on input data size like.
1. 1-5 gb
2. 5-15 gb
3. 15-25 gb
4. 25-50 gb
5. > 50gb = 1 tb
Sure Neel, will make a video on this requirement
@@rajasdataengineering7585 Waiting for this!
excellent explanation!!
Thank you Sudip
Can you provide documentation of this video
Bro ...i have some doubts and need your guidance on how to proceed with spark ..is there a way to contact you ? ☹️
FROM WHERE TO START? IS THERE ANY SEQUENCE TO FOLLOW??
Playlist is already sorted in correct order. You can follow the playlist
inside the worker node how many executor will be there who will decide this
Can you please guide me how to start your videos I mean the order I can see many playlists in the channel. I want to learn spark and data bricks
Sure bro, let me give serial number to my videos so that you can follow the structured learning list
@@rajasdataengineering7585 yeah thanks, also please comment here that which playlists we shld follow for the order to learn spark and data bricks 🙂
Sure
@@saikrishna1939 The videos are given with serial number. You can follow with that sequence
@@rajasdataengineering7585 yes but which playlist to follow as there are 7 playlists in the channel so it's Lil confusion. Say for example a playlist has 5 videos but when I open it I can see 17, 18, 19, 20 in the videos. In interview series it starts with 1 again
What is the total duration of the playlist
Do you have python videos as well .
Hi Jas, not yet. Will create python series soon if needed
@@rajasdataengineering7585can u please provide python videos
Nice Videos! can you share the slides?