Data validation between source and target table | PySpark Interview Question |

Hello Everyone,
source_data = [(1,'A'),(2,'B'),(3,'C'),(4,'D'),(5,'E')]
source_schema = ['id','name']
source_df = spark.createDataFrame(source_data,source_schema)
source_df.show()
target_data = [(1,'A'),(2,'B'),(3,'X'),(4,'F'),(6,'G')]
target_schema = ['id','name']
target_df = spark.createDataFrame(target_data,target_schema)
target_df.show()
This series is for beginners and intermediate level candidates who wants to crack PySpark interviews
Here is the link to the course : www.geekcoders...
#pyspark #interviewquestions #interview #pysparkinterview #dataengineer #aws #databricks #python

[@beingnagur]

At 6.04 instead of copying the same statement you can use .otherwise("not matching")

[@rishabhkesarwani-br2rx]

I do below steps to compare source vs target table
1) Count should be matching in source and target table
2) Schema should be matching in source and target table
3) Use the except and to check if any records are there which are present in source and not in target or vice versa.
4) Use the left anti join to find out the records which are not matching.
5) Trying to debug why there is record mismatch

[@GeekCoders]

Nice

[@gudiatoka]

exceptAll can be usefull too or anti join

[@GeekCoders]

Except all may miss the null value sometime

[@nishirajnikku969]

I request you to please create a playlist for Pyspark Unit testing .

[@jhonsen9842]

Main Problem i found in learning Pyspark is brackets every time it gives me some error.

[@GeekCoders]

Yes

[@CeejayPTcoach]

wont the join be a costly operation

[@VinodKumar-gz8bk]

What are the most challenging thing that you faced in your project & how you overcome?

[@shivamchandan50]

plz make video on pyspark unit testing