12. StructType() & StructField() in PySpark |

In this video, I discussed about StructType() and StructFiled() Classes to create schema for dataframe.
Link for PySpark Playlist:
• 1. What is PySpark?
Link for PySpark Real Time Scenarios Playlist:
• 1. Remove double quote...
Link for Azure Synapse Analytics Playlist:
• 1. Introduction to Azu...
Link to Azure Synapse Real Time scenarios Playlist:
• Azure Synapse Analytic...
Link for Azure Data bricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure Logic Apps playlist
• 1. Introduction to Azu...
#PySpark #Spark #databricks #azuresynapse #synapse #notebook #azuredatabricks #PySparkcode #dataframe #WafaStudies #maheer #azure

[@manu77564]

My humble request... please continue..

[@WafaStudies]

Sure. Thank you ☺️

[@peerkhaja2700]

Ur always giving informative videos.. Keep it up maheer

[@WafaStudies]

Thank you ☺️

[@durgasiresh]

Good explanation and great effort & very useful videos Thank you!!

[@babarhassan7726]

Thank you I needed this video 👍

[@polakigowtam183]

Good Vedio . Thanks Maheer

[@WafaStudies]

Welcome

[@adityashrivastava860]

Beautiful explaination.

[@jeevaraj815]

Hi Sir your videos are helpful for me.I learned very much with your videos.... One humble request if is possible means you can do it, Atleast one video per day or 5 videos per week. Thanks in advance

[@ITKaksha]

Good explanation. I have one query, I other videos, you have also used below format
StructType().add(field='id',data_type=IntegerType())
In this video, you have slightly format
StructType([StructField(name='id','dataType=IntegerType())
Are both these same ?

[@sahildhar6805]

Yes

[@srinureddy378]

Yes, but different syntax, and we have few more ways to define schema

[@vutv5742]

Completed🎉🎉🎉

[@subhanishaik8163]

Hi i have one question.how to convert 11/11/2022 1102 to YYYY-MM-DD HH:MM:ss in pyspark

[@menaga.g]

Hi @subhanishaik8163
By using date_format():
df = df.withColumn('date_time_str' , lit('2022/11/11 1102'))
df1 = df.withColumn('New', date_format(to_timestamp(df.date_time_str, 'yyyy/MM/dd HHmm'), 'yyyy-MM-dd HH:mm'))
OUTPUT:
date_time_str New
2022/11/11 1102 2022-11-11 11:02
2022/11/11 1102 2022-11-11 11:02

[@Ali-q4d4c]

👍🏻

[@VinayKumar-st9iq]

Abyone help me. For me getting type error while excecutung below code
error
TypeError: __call__() takes 1 positional argument but 2 were given
Code:
from pyspark.sql.types import StringType, StructField, StringType, IntegerType
data = [(1,'Narendra',2000),(2,'Modi',5000)]
schema = StringType([\
StructField(name='id',dataType=IntegerType()),\
StructField(name='Name',dataType=StringType()),\
StructField(name='Salary',dataType=IntegerType())])

df = spark.createDataFrame(data,schema)
df.show()

[@prasanthrajagopal158]

You are using "schema=StringType", I think thats a typo. Use "StructType()"