79. Databricks | Pyspark | Split Array Elements into Separate Columns

Explode Function == • 11. Databricks | Pyspa...
Azure Databricks Learning: Pyspark Transformation
=============================================
How to split elements of a array column of a dataframe into Separate columns in Databricks development?
In Databricks pyspark development, we used to come across various unique requirements and one of them is splitting the array elements of a dataframe column into separate columns.
To get through understanding of this concept, please watch this video
#PysparkArrayElement,#PysparkSize,#DatabricksSplitElementsIntoColumns,#PysparkSplitElementsIntoColumns,#SparkArrayDataType, #DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@lukusausten2174]

Why cant we just explode and pivot afterwards taking the first value?
Avoid UDF no?

[@code_nation]

In case, if name of array_column is different then we have to change the array column name in function too. Rather than that, we can use this function:
def splitArrayToColumns(df, array_column, max_value_in_array):
df_size = df.withColumn("NoOfArrayElements" ,size(array_column))
max_value = df_size.agg({"NoOfArrayElements": "max"}).collect()[0][0]
for i in range(max_value_in_array):
df = df.withColumn(f"new_col_{i+1}", df[array_column][i])
return df

[@rajasdataengineering7585]

Thanks for sharing a solution

[@akash4517]

Great Videos Raja .

[@rajasdataengineering7585]

Thanks Akash!

[@SuganyaVinayak]

I have doubt. Daily basis i am getting the column and appended to existing table. But day to day col count diff. Then arrayofstruct, arrayof string those datatype. How i resolve.

[@RakeshGandu-wb7eu]

Nice video. Any examples on other way to convert single column into array columns

[@भ्रमंती-ज5ज]

Hi Raja, How can we use this for arrays which contains json with some key value pairs seperated by "=" ..for eg [{name = Joseph, city = newyork },{org = xx}]

[@travelfoodlife1449]

I m a new learner .. what u
Is the meaning of using f in df=df.withcolumn(f"new_col_{i} ???????

[@QuanNguyen-z2g]

Hi Raja, please share git-repo storing all the source codes of this learning series. Much appreciated!

[@ajanthraju]

Can't we use explode() or explode_outer() function?

[@rajasdataengineering7585]

Explode will flatten to new rows, not new columns