video from 11:30, we are adding random key to exiting towerid key for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right. what in case of partition column is string datatype. ??
If the partition key is non numeric then how to perform salting? like your tower ids were numeric, but if instead of being 1, 2, .. they are to be A, B, ...
thank you so much , really good , so what is the difference b/w isolation salting and salting ? and what is difference b/w , isolation map join & map join ??
Nice explanation.A couple of questions 1) Repartitioning does ensure the data distribution is not skewed (unlike coalesce) 2) You said repartitioning uses the hash value to distribute the data (are you talking about bucketing ?)
@TechWithViresh I simply love your videos. I have watched your other tutorial videos too. They are awesome. I am interested in knowing how to do Iterative Broadcast Join with the SQL API. Any help is highly appreciated. Can you pls advise.
We could loose our key join by Salting key adding random numbers If we want to do join with the same key then problem May be join key could be the different on other than salted column
Thank you so much for the video. I seek some clarification though. In your example you did mapPartition. Means for each partition of different keys, you updated the key with salt. But still the records remained in the respective partitions only. How will those records be shuffled across partitions for equal distribution?
@@TechWithViresh I tried it so I believe a new DF will have to be created and REPARTITIONED again! in order for the records to be shuffle by updated salted keys. It wont just trigger shuffle on key update in mapPartition function! That only makes sense.
7:24, repartition does full shuffling and hence creates equal size partitions. i.e It always guarantess the equal sized partitions.
Very informative video 👍🙂
video from 11:30, we are adding random key to exiting towerid key
for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right.
what in case of partition column is string datatype. ??
Incase of strings, we can add surrogate keys, based on string column values and then do the salting.
Great video, like the pace, like the presentation.
Glad you liked it!
Hi Viresh, Thanks for the video.... How can we achieve salting technique in Pyspark?
thank you so much for this video, but i am not able to find the 2nd part of this video.. Can you please comment the link for the part 2 video
Hi Viresh, can you please share the link for part 2
Hello,
Your videos are very good,
Can you please do a video on incremental data load and full data load by taking an example?
If the partition key is non numeric then how to perform salting? like your tower ids were numeric, but if instead of being 1, 2, .. they are to be A, B, ...
Thank you for making this video. Could you suggest on which column mean, medium and the mode are calculated?
The columns are those that are being shuffled by such as the join columns or group by columns. There is data skew when the distribution is not normal.
thank you so much , really good , so what is the difference b/w isolation salting and salting ? and what is difference b/w , isolation map join & map join ??
Thanks for the video, no part 2 tho?
Nice explanation.A couple of questions 1) Repartitioning does ensure the data distribution is not skewed (unlike coalesce) 2) You said repartitioning uses the hash value to distribute the data (are you talking about bucketing ?)
There are two provided partitioners in Spark 1. Hash partitioner and 2. Range partitioner.Default is Hash one.
If you repartition on column, there you can get skewed data. If you repartition by number of parts then distribution may be almost equal.
@TechWithViresh I simply love your videos. I have watched your other tutorial videos too. They are awesome. I am interested in knowing how to do Iterative Broadcast Join with the SQL API. Any help is highly appreciated. Can you pls advise.
We could loose our key join by Salting key adding random numbers
If we want to do join with the same key then problem
May be join key could be the different on other than salted column
have u uploaded part 2 of this
Check out other videos in the playlist for performance optimization and executor tuning.
Thank you so much for the video. I seek some clarification though.
In your example you did mapPartition. Means for each partition of different keys, you updated the key with salt. But still the records remained in the respective partitions only. How will those records be shuffled across partitions for equal distribution?
Partition will change with the change in the key, as it is essentially the hascode of key+salt now.
@@TechWithViresh I tried it so I believe a new DF will have to be created and REPARTITIONED again! in order for the records to be shuffle by updated salted keys. It wont just trigger shuffle on key update in mapPartition function! That only makes sense.
Hi sir, pls upload the spark interview question videos which were present earlier.. I'm not able to find them in your playlist
All the videos are uploaded, please check:)
where is Part 2 ?
Awesome video 🙏...can you pls share part2 video link
Coming Soon!! , Thanks :)
@@TechWithViresh when ?
can u share part 2 video
Coming Soon!! , Thanks :)
please give some solid coding example with explaination