วีดีโอ

I would have appreciated if you would have run the code of salting and showed us on spark UI for better clarity what is happening internally within spark

Dude you are awesome. I didn't know Delta lake is a thing until now. Surprising you posted a video about it 4 years ago

Thank you so so much Man!!!

beautifully explained, thank you very much :)

Hello sir I was trying to resolve multiple delimiter issue from two days ,my spark version is 2.x your vedio has helped me alot

What if we dont have primary key in the datasets??pls let me know

Hi I have one doubt. how i contact you sir?

Wonderful video

Using row number messes up the data for some reason.You will always get the correct count. But If you do group by on random columns and take a count ,you will see a difference in count for each group. Although the sum of all group will be rqual to count .But each groups count will.vary

Nice video

Nice explanation 👌 perfect

Hi Sir... Perfect Great Explanation... Thank you for your effort... I have a doubt :-- After joining The Salting step should be - unsalted and then grouped by has to be applied, Right...? .....

Thank you. Helpful content!

Need to Display Primary key also along that. how we can do that

Thanku Sir 🙏

best

Little confused, Its mentioned that GlobalTempView is application scoped. How can we share dataframe through different spark applications. I tried using global temp view from a spark application but couldn't access it from the different application. Is there any other way to achieve that?

nice

so nice thanks

How to display unique keys along with mismatch records. we are unable to identify which primary key got mismatches. Can u pls help us on this..

but the join output will not be correct because in previous scenario it would have joined with all the matching ids but with new salting method it will join with only newly slated key, that's weird

I have 2 questions: First one: I think that is wrong on your visual presentation of table 2 after salting. Why don't you have z_2 und z_3 there? Also why are you using capital letters sometimes, that's confusing. Secone question: I don't get the benefit of Key Salting in general. How is this different from broadcasting you second table? Because you explode it and then you will end up with sending the whole table to every executor anyway? No one can give an answer to this question.

Hi, this video will help us. but As you mentioned in video when you will be post the next video? how to get the mismatching records with the uniques id? can you please try that?

Hello Bro I have some 3lakh data i want to include the unique id also in the quey output so that will help to find exact where is the mismatch can u help me how to do that

Nice bro. Why have you stopped putting videos in this channel Seriously it's of quality content which helps many

Superb ,I was searching this specific answer from 2 hours and finally found.

Column rename name is not reflected in parquet file

Well, I must say, thanks a lot.....have been searching for this kind of explaination.

Great Video!! Whats is the comand "cache" for??

Nice explanation... 👍 Informative

Thanks but if we have multiple columns as KEY how to handle it ?

its great video - can you help me to get key column as well in result along with colName and mismatched ?

Very well explained.

How can we see dirty data ? I think the explanation is wrong!!!

Good work, its better you show the ourput after the salting dataframes and explain udf more detail.

Hey great video, could you also link the associated resources you referred to while making this video?

@Tech Island hi is there a performance difference with and without AQE

U r producing quality content. Kudos. Thanks.

Can we achieve similar performance in Spark for DB writes also basically an upsert query on an already existing huge postgres table?

Thank you so much really appreciate your nice clear way of simplifying things.

Very informative. Thank you!!!

Excellent Description

I loaded data from MySQL to dataframe using spark.read.format , after that I did some transformation and created final dataframe to load in to same table in MySQL.....so I did truncate and used overwrite using write.mode() but I am not able to see data in MySQL but if I load to different table it works but not on same table .....please provide your suggestions.

The concept is explained very clearly. Do you have any video for Coalescing Shuffle and optimized join strategy?

Thank you very much bro! Nicely explained. Please do explain one live project end to end with data bricks.

Diagram is not clarify the things like what is unresolved logical plan, how it becomes logical plan and after optimized logical plan this is something on which if you can provide detailed information that will be a good understanding for all of us.

This really great and crystal clear explanations....thanks a lot for sharing and spreading knowledge!

Not that much clear.. Moving the slides and explaining will not help people to understand the concept..

Great !

Amazing!