วีดีโอ

Great video... Will dynamic partiton pruning work if both the datasets are partitioned on a column and same column is the joining key?? I have seen that in such cases, spark does a full shuffle , so was curious to understand why is spark behaves like this when both tables are partitioned on the joining key

Hi Afaque, what if i found the size from Optimized plan :: catalyst_plan = df._jdf.queryExecution().optimizedPlan() df_size_in_bytes = catalyst_plan.stats().sizeInBytes() df_size_in_bytes_in_mb = df_size_in_bytes / (1024 * 1024) print(f"DataFrame size in MB: {df_size_in_bytes_in_mb}") -- will that be accurate replication?

These are gold standard advices... specially on the part on how to keep practicing for data modelling. It will be really great if you can also point towards some resources for other topics

Hi Afaque, I have doubt over here When you mention memory which memory it holds cached data and also disk means where it holds data exactly? can you elaborate

Crisp and simple explanation

This is the most detailed video I found on salting. Afaque, are you on TopMate ? Would love to get your mentorship

how much they were offering you for this role ?

Hi, was this role Data engineer 2 or senior data engineer. Kindly confirm.

Hi great video. I have a confusion about something the partition of the original parquet files was based on timestamp and not date, that is why you create a date out of that but after reading the parquet, but because the filter was based on a date (not a timestamp), how spark manage to achieve this dynamic parition pruning ?

Have you worked on building microservices and rest apis ? and do they ask them in interview ?

Thanks for the amazing video. Very well explained. Great Job !

Hi, apart from Spark, SQL, DSA, DM/DWH & System design, do they ask other stuffs as well like jenkins, github, scala, flink, kakfka etc?

Hi Afaque, in this video you talked about the GC cycle, basically when on heap memory is full, then GC happens. From where can we get to know that GC cycle has happened and now its time to use some Off heap memory? I mean do we check DAG/Query Plan or somewhere else.

how much ctc apple was offering ?

share some good books which you read for data engineeing

how did you apply for this role, careers or referral ?

How much they were offering ? location india or singapore ? also, does apple singapore hire from india , please tell !

How did you apply for this role ? careers or referral ?

Thank you for this Series! Exactly what I needed, Amazing tutor!

Thanks for the informative video. I have one major doubt: We are doing 25KB and 9KB of IO operation consecutively to use the bucketing and writing the data into tables. Spark is meant to use in memory processing. If we compare the duration of this bucketing IO operation with shuffling what will the cost and the execution time difference we'll get. Is it worth to use bucketing as it comprise IO operation.

Amazing video!

wow

Congratulations man🎉 Keep posting these kind of videos with more source of resources which would helpful to get into those positions. You guys are real light house in the sea of job searchers ❤

You explains these concepts very easily. Thanks you so much

Thanks a lot for the video. Afaque. Just have a quick question: you weren't asked anything related to the cloud?

You are simply amazing for sharing these tutorials, thank you very much!

There are some cases where we have huge data in a day or week or month. In that case we make partition and then bucket. So in that case there can be shuffling but still a better performance. Am I correct?

Good but would have been more appropriate to cover the join , how to parition data before join to make sure join works properly.

Congratulations Afaque.....doing a great job sharing all this precious information👍🙌

Hi @Afaque , Is atlassian still supporting WFH model or have shifted back to WFO i..e specifically for DE roles?

Is it possible to shift from QA backgroung to DE ?

This has been really insightful, Thank you!!

can you please add sql questions somewhere in description?

can you list sql questions?

This playlist is awesome, watched all the videos on the trot. Keep producing great content bro!

What will happen to the bucketed tables which were created during the spark execution?, is it going to drop once after execution completes or do we need to drop it and how ?

really great videom thanks! can you also explain how bucketing works with several columns

Wonderful video, first of all. I have a question. I understand that 5 cores per executor is better for HDFS throughput. I don't understand how is this beneficial in any way. Because whether I have 100 cores per executor in just one node or 5 cores per executor across 20 different nodes, if I want to read 1 TB of data from HDFS, it will read 1 TB itself right? How does 5 cores per executor keep throughput in place?

Finally re-watched the whole playlist after a year for my spark revision :) I can surely say these are the best videos on these topics. Please keep making such in-depth videos on important concepts and other DE trending technologies as well based on your experience, that you feel are important for DEs. I really appreciate your work Afaque!

Awesome 👍

Great in depth information

Hi Afaque, can you please tell why GC increases in case of FAT Executors?

This is amazing content, so helpful. You gained a subscriber!

I love your videos man, I'm watching these again after a year to revise the concepts. Please keep making such in-depth videos, these are really helpful, noone explains these concepts at this level like you. :)

It is very descriptive and can you pls share the roadmap and proper materials and how much python and SQL required for cracking product based companies

When can we expect more videos on Spark ;)

I love to work

Inspiring and informative.

please share some resources you follow for preparation

But u recently joined DataBricks i think