Spark Runtime Architecture (Cluster Mode) |

#pyspark #azuredataengineer #databricks
Use the below link to enroll for our free materials and other course.
www.cleverstud...
You can talk to me directly on Topmate by using the below link:
topmate.io/nar...
Follow me on LinkedIn
/ nareshkumarboddupally
-----------------------------------------------------------------------------
Clever Studies Official WhatsApp Group joining link:
Clever Studies 2.0: chat.whatsapp....
Clever Studies: chat.whatsapp....
--------------------------------------------------
Follow this link to join 'Clever Studies' official telegram channel:
t.me/+eMaiZNWT...
--------------------------------------------------
Facebook: www.facebook.c...
Instagram: / cleverstudiesindia
PySpark by Naresh playlist:
• PYSPARK BY NARESH
--------------------------------------------------
Realtime Interview playlist:
• How To Explain Project...
--------------------------------------------------
Apache Spark playlist:
• How Spark Executes A P...
--------------------------------------------------
PySpark playlist:
• PySpark | Tutorial-9 |...
Hello Viewers,
We ‘Clever Studies’ TH-cam Channel formed by group of experienced software professionals to fill the gap in the industry by providing free content on software tutorials, mock interviews, study materials, interview tips, knowledge sharing by Real-time working professionals and many more to help the freshers, working professionals, software aspirants to get a job.
If you like our videos, please do subscribe and share within your circle.
Contact us: [e-mail]
Thank you!

[@user-js8zh7ge7r]

very good job done

[@mitrasubham]

Nice explained

[@2412_Sujoy_Das]

Needed this one badly... Thanks Naresh

[@PavanKumar-vi7hd]

Hi Naresh
your way of explanation is excellent.
this is first time i understand spark architeecture is very easy way in Cluster Mode

[@cleverstudies]

Thank you Pavan.❤

[@pankajchikhalwale8769]

Please make a short video on the relationship between stages, node, executor, dataframe/dataset/RDD, and core, partition, and task. Want to know what consists what ? And what contains what.

[@user-tl7sh4tm8x]

How cluster manager can create any of the nearest worker as application master because the configuration of the master can be different. So will it not go and create master to the machine that is configured for master role. With memory allocated to master depending on the type of task.

[@TechGuyTelugu]

What about the node manager do in this architecture

[@vivekmerugu6711]

Hi Naresh,
Thank you so much,
This help me a lot. ❤
Naresh, I've execute my spark application in cluster mode (yarn) in emr cluster,
My spark application is failing with an exception saying application master container failed 2 times, exists with 137 code.
This exceptional is occuring for only one dataset which I'm processing with spark application.
For other datasets, my spark application is working fine.
The dataset for which spark application is failing having large input payload, ( one record with 25000+ characters ).
I tried increasing the driver memory and executor memory, now this time , I'm getting an exception while deserialization of input payload.
Any suggestions how to resolve this issue.
It will be helpful, please