We cannot use more than one explode in a spark sql statement, we have work around for this, will discuss in upcoming videos.. Thanks for your support 🙂
Sure, Please have a look at our Playlist, we have 40+ interview scenario question. We will add more :). Please do try on your own machine for better understanding.
Hey Azar, I need your help here. I am trying this example but getting this issue below This is the input file I am using input file - explode_posexplode_ex.csv Name|Age|Education Azar|25|MBA,BE,HSC Sangam|27|BE,HSC Dilip|MBA,BBA,HSC Ajay|BE,HSC then I created df as below df = spark.read.option("delimeter","|").csv("/FileStore/tables/explode_posexplode_ex.csv",header=True) But I am getting only one col instead of 3. When i execute below command output is shown below df.show() df.columns (1) Spark Jobs +------------------+ |Name|Age|Education| +------------------+ | Azar|25|MBA| | Sangam|27|BE| | Dilip|MBA| | Ajay|BE| +------------------+ Out[42]: ['Name|Age|Education']
Small correction: explode() won't take Struct types; It takes only array and map types
Superb explanation 👌
I checked and posexplode is still not working with withColumn in spark 3. Analysis exception
Difficult question. Easy solution 👍
Nice
Nice explanation bro
Thank you for the series, it is very useful
Glad to hear that!
Hi bro in pyspark dataframe how to find each column null values count
Hi Shahul , in this example we have only one array , now we have multiple struct array . can you pls explain how to do that ? thanks in advance mate
Thanks , can you please make some samples using agg with struct of arrays??
Could you please create a video on "how to implement SCD type 2 using pyspark"?
What if there are two or more columns that need to be exploded ??
We cannot use more than one explode in a spark sql statement, we have work around for this, will discuss in upcoming videos..
Thanks for your support 🙂
can you post all possible interview questions
Sure, Please have a look at our Playlist, we have 40+ interview scenario question. We will add more :). Please do try on your own machine for better understanding.
Hey Azar,
I need your help here.
I am trying this example but getting this issue below
This is the input file I am using
input file - explode_posexplode_ex.csv
Name|Age|Education
Azar|25|MBA,BE,HSC
Sangam|27|BE,HSC
Dilip|MBA,BBA,HSC
Ajay|BE,HSC
then I created df as below
df = spark.read.option("delimeter","|").csv("/FileStore/tables/explode_posexplode_ex.csv",header=True)
But I am getting only one col instead of 3. When i execute below command output is shown below
df.show()
df.columns
(1) Spark Jobs
+------------------+
|Name|Age|Education|
+------------------+
| Azar|25|MBA|
| Sangam|27|BE|
| Dilip|MBA|
| Ajay|BE|
+------------------+
Out[42]: ['Name|Age|Education']
bro how to remove delimeter from spark datafrme?????? i need answer brooo
Can u share some sample inp and out here or in mail
How to do with Scala please help me
Please mail me the error that you are facing