4 Recently asked Pyspark Coding Questions | Apache Spark Interview

[@adityatomar9820]

One of the great explanation so far on youtube. I wish i could afford your course :(

[@sopankardile2603]

One of the best interview series Thank you sumit sir .

[@sumitmittal07]

glad to know that you liked it.

[@singhjirajeev]

00:03 Recently asked Pyspark Coding Questions
02:37 Writing and executing Pyspark pseudo code
05:21 Creating a Spark dataframe from input and performing group by aggregation
08:04 Using aggregation functions and collect list in Pyspark.
11:15 Spark SQL solution for creating DataFrame and running queries.
14:18 Understanding the data frame reader API for reading JSON and the usage of explode function
17:11 Creating a Spark dataframe and performing operations on it.
19:44 Converting string to date and performing group by in Pyspark DataFrame
22:32 Finding the average stock value using PySpark
25:38 Practice more on data frames for interviews
28:15 Practice more to gain confidence in writing correct syntax for Pyspark coding

[@souradeep.official]

Need more Pyspark Interview Solutions like this 😊

[@abhyaravya421]

Thanks a lot, Sumit! I am a senior data engineer with 5 years of exp but since we don't work with dataframes or pyspark mostly I am not able to do these simple things.

[@ritikadamani2008]

Best selection of questions and very good explanation.

[@praptijoshi9102]

You are doing a great job posting these❤

[@veerugandhad3437]

Very useful informative video which gives more confidence to the bigdata aspirants. Thanks Sumit.

[@2412_Sujoy_Das]

Much needed sir.....!!!

[@sumitmittal07]

Sujoy, I am sure you will enjoy watching this.

[@venugopal-nc3nz]

It will be great if you put questions in comment . Others can try without looking at solution first

[@satishutnal]

Best explanation sir thanks

[@sumitmittal07]

I am happy to hear this

[@gudiatoka]

Sir...Share need more .. please continue this playlist

[@rohit-ll3rj]

We can apply distinct() too I guess for avoiding duplicate values in df.

[@shashankgupta2776]

Thank you Sir greatly explained, would be good if you can post data/schemas also in the decription box for us to query and do hands on. Thanks.! :)

[@TheUMESH34]

This is great!

[@sumitmittal07]

thank you Umesh

[@anjibabumakkena]

Nice explanation sir, kindly post scenario based questions

[@sumitmittal07]

yes for sure

[@SusheelGajbinkar]

Thank you sir😄

[@prasoonvijay5775]

Hi Sumit,
Could you please create Video explaining pipelines on AWS Databricks End-End along with Orchestration of those.

[@NextGen_Tech_Hindi]

thanks sumit make videos like this .

[@sumitmittal07]

definitely

[@sravankumar1767]

Superb

[@Nikhil-qi4oz]

Amazing sir

[@sumitmittal07]

Nikhil, I am sure you will find it useful.

[@NextGen_Tech_Hindi]

What about remaining 10 questions on pyspark you told we are covering it in next video but still you not uploaded on TH-cam and when you will upload it on TH-cam we are waiting for remaining 10 questions on pyspark
Thank you ❤

[@mdasif2411]

Hi Sir, can we not write in Spark sql in interview? As there is no difference in performance.

[@sharankarchella2688]

Nice video

[@sumitmittal07]

thank you

[@rudrakasha-t1v]

in question number 2 = do we not need to remove duplicate as last can you please clear me on it ?

[@VinodKumarChouhan-o8c]

Hello sir, how can I run pyspark code online, are you also using any online utilty to run pyspark code as shown in this video , could you please share the source, it would be very helpful.

[@sonurohini6764]

Sir create coding interview playlist

[@RAHULKUMAR-px8em]

Q2.
Data=[('a','aa',1),
('a','aa',2),
('b','bb',5),
('b','bb',3),
('b','bb',4)]
data_schema= "col1 string, col2 string, col3 int"
df_data=spark.createDataFrame(data=Data,schema=data_schema)
df_data.display()
from pyspark.sql.functions import *
from pyspark.sql.types import *
result = ( df_data.groupBy(col('col1'),col('col2'))\
.agg(collect_set(col('col3')))
)
result.display()