Very neat and clear explanation. Thank you so much.!! .**SUBSCRIBED** one more thing ..how can someone dislike anyone's efforts to produce such helpful content. please respect the hard work.
Hi Ranjan, thank you for the great series and excellent explanations. I have two questions: 1) In the video at 5:05, you mention that PySpark requires a cluster to be created. However, we can create Spark Sessions locally as well if I am not mistaken. When we run spark locally, could you please explain how PySpark would outperform pandas? I am confused about this concept. You can process data using various cores locally, but your ram size will not change right? 2) In the previous video you mentioned that Apache Spark computing engine is much faster than Hadoop Map Reduce because Hadoop Map Reduce reads data from the hard disk memory during data processing steps, whereas Apache Spark loads the data on the node's RAM. Would there be a situation where this can be a problem? For example, if our dataset is 4TB and we have 4 nodes in our cluster and we assign 1TB to each node. How will an individual node load 1TB data into RAM? Would we have to create more nested clusters in this case?
@@RanjanSharma I just got a job offer for a data engineer working with databricks spark. Your video definitely helped me in the interview. Thank you again.
@9:19 min RDD in memory computation? Panda does in memory isn't it? do RDD also do in-memory.. may be i lost somewhere with point can you explain this minute difference please?
Great and Clear expression..the one and only best playlist for Pyspark in youtube
that's what I call quality content. Very logically presented and instructed.
Great sir ...happy for clearing the concepts
Keep watching..thanks bro . Keep sharing and Exploring bro :)
Very neat and clear explanation. Thank you so much.!! .**SUBSCRIBED**
one more thing ..how can someone dislike anyone's efforts to produce such helpful content. please respect the hard work.
thanks So nice of you :) . Keep sharing and Exploring bro :)
excellent explanation,, strong hold on concepts,,
Glad you liked it! thank you :)
Excellent Content, Thank you Ranjan.. Subscribed :D
Hi Ranjan, thank you for the great series and excellent explanations. I have two questions:
1) In the video at 5:05, you mention that PySpark requires a cluster to be created. However, we can create Spark Sessions locally as well if I am not mistaken. When we run spark locally, could you please explain how PySpark would outperform pandas? I am confused about this concept. You can process data using various cores locally, but your ram size will not change right?
2) In the previous video you mentioned that Apache Spark computing engine is much faster than Hadoop Map Reduce because Hadoop Map Reduce reads data from the hard disk memory during data processing steps, whereas Apache Spark loads the data on the node's RAM. Would there be a situation where this can be a problem? For example, if our dataset is 4TB and we have 4 nodes in our cluster and we assign 1TB to each node. How will an individual node load 1TB data into RAM? Would we have to create more nested clusters in this case?
I've same doubt. How spark would store TB's of data in ram
great explanation
Glad you think so! Buddy keep exploring and sharing with your friends :)
Nice explanation
Great thank you for this explanation
Thanks :) Keep Exploring :)
@@RanjanSharma I just got a job offer for a data engineer working with databricks spark. Your video definitely helped me in the interview. Thank you again.
@@JeFFiNat0R Glad i could help you 😊
Content wise great videos.. way of explaining can be improved.
Glad you think so!Thanks :) Keep Exploring :)
@9:19 min RDD in memory computation? Panda does in memory isn't it? do RDD also do in-memory.. may be i lost somewhere with point can you explain this minute difference please?
you are too fast in explaining things. Could you please slow down a bit ? btw, good work
Thanks for your visit .. Keep Exploring :)
in my further videos , i have decreased the pace.
Why are you still using RDDs and not the Spark SQL Dataframe API?
This video was just for explanation of RDD. In next video, I will be explaining SQL DataFrame.
Audio is low compared previous 2 videos.
**gj**