True! in every video of the Big Data playlist there will be a moment where you feel like you just learnt something that will blow away your senior's mind!!!1
Bro ur videos are genuinely so helpful.. they way you give actual examples are really nice.. easy to understand.. thanks to you.. keep up the good work we would watch the inbetween youtube ads 👍
what if the bucketed column is a string how will it perform the hash partition on it using Modulus division ...will it take the ASCII value of string into consideration?
Thank you very much Gowtham for making Big Data concepts so easy and understandable. The contents are great and help millions of people like me who are moving toward Big Data. Keep on good work :)
While scanning the buckets for the records , does it scan all the buckets one by one until the record is found and will break out once the record is found ?
I am working in internal audits ( 2 years exp..) - role is to be checking the business transactions with large data and making the observations... Companies can consider the internal audit Experience for data analyst jobs ???
excellent work and explanation but i have a question. what if the remainder is bigger than the number of buckets ? for example if we have 3 buckets and hash % 3 is 6 . in which bucket that record will be ? thanks.
If i get it correctly you are assuming what happen any hash value hash (x) % (mod a) > (mod a) and then whil bucket will it go? The simple answer is no this is not possible in current math . What you are assuming is the qoutient and this is separte than remainder . Lets take an example of hash value 95 and mod value 3 . Here qoutient would be 31 that is (95 - (31*3)) = 2. If you have 10001 also in hash value still you will get qoutient 3333 that is (10001-(3333*3)) = 2. Remainder will always be
When you have taken 2 buckets and if you insert values in same order 1. 9876 2.1234 3.5678 4.0014 Then, if we are searching for 0014 Now it come under Full Scan, So that is the reason, you have taken 1&4 in 1st bucket and the remaining in 2nd bucket. Hahaha....good cover drive
Ha ha it's not like that , and the records will not store in the order the way we insert in the bucket , the hash partition algorithm will decide the buckets destinations for the records , 0014 can be in 1st bucket too . Using bucket we are not completely avoiding full scannn... 🙂
![How to Decide [Bucket Count] in Hive #hive #apachehive](http://i.ytimg.com/vi/pHvyhfWK43s/mqdefault.jpg)
![How to Decide [Bucket Count] in Hive #hive #apachehive](/img/tr.png)
![Hive Partition [ Static Vs Dynamic]](http://i.ytimg.com/vi/RFS4JJFlzGc/mqdefault.jpg)
You nailed the game, I do not need expensive courses to learn when I can have a brother like you.
Thank you so much bro 🙏
True! in every video of the Big Data playlist there will be a moment where you feel like you just learnt something that will blow away your senior's mind!!!1
Love the way you explain. Completed whole hive series. Keep posting more.
great explanation. I was having hard time understanding why bucketing concept is required if partitioning is there in hive. Now i got the idea
Bro ur videos are genuinely so helpful.. they way you give actual examples are really nice.. easy to understand.. thanks to you.. keep up the good work we would watch the inbetween youtube ads 👍
Thanks for the series, gives deep understanding and clear picture on topics. Waiting for Pyspark videos.
Outstanding explanation..Thanks much..-Mohamed
This explaination on hive bucketing was really useful👍it has really helped me to clear technical interview
Keep updating this playlist. Very useful Thanks!
what if the bucketed column is a string how will it perform the hash partition on it using Modulus division ...will it take the ASCII value of string into consideration?
Thank you very much Gowtham for making Big Data concepts so easy and understandable. The contents are great and help millions of people like me who are moving toward Big Data. Keep on good work :)
It was a great session.thank you for your efforts.
Good explanation. thanks for the video
While scanning the buckets for the records , does it scan all the buckets one by one until the record is found and will break out once the record is found ?
Bro, your interview questions vd was very useful. Can you do the same for each topic separately pls?
I am working in internal audits ( 2 years exp..) - role is to be checking the business transactions with large data and making the observations... Companies can consider the internal audit Experience for data analyst jobs ???
Easily understandable explaination❤
excellent work and explanation
but i have a question. what if the remainder is bigger than the number of buckets ?
for example if we have 3 buckets and hash % 3 is 6 .
in which bucket that record will be ?
thanks.
Thanks
So you mean hash is 3 and number of bucket is also 3
Therefore 3 mod 3 right ?
If i get it correctly you are assuming what happen any hash value hash (x) % (mod a) > (mod a) and then whil bucket will it go? The simple answer is no this is not possible in current math . What you are assuming is the qoutient and this is separte than remainder . Lets take an example of hash value 95 and mod value 3 . Here qoutient would be 31 that is (95 - (31*3)) = 2. If you have 10001 also in hash value still you will get qoutient 3333 that is (10001-(3333*3)) = 2. Remainder will always be
It is property of natural numbers that any number n divided by x will have remainder
I also had the same doubt came to my mind. But that is not the case. Because reminder of 3 can not be more than 3...
Very nicely explain ...keep posting new videos pls
Good content as always, thank you.
Found a gem in you tube!💎💎
hi can you post video on how to prepare and crack faang interviews?
Nice explanation brother.
How bucket works on top of partition?
for the begginer hard to find the order, to follow, would appreciate if you rename it with part-1,part-2...
what is that 's' at the end of the command SELECT avg(amt) FROM bk_test_data TABLESAMPLE(BUCKET 1 OUT OF 3 ON rand()) s;
Great bro thanks
Nice explanation 😁
Great info bro
Glad you liked it
Thank you brother
When you have taken 2 buckets and if you insert values in same order
1. 9876
2.1234
3.5678
4.0014
Then, if we are searching for 0014
Now it come under Full Scan,
So that is the reason, you have taken 1&4 in 1st bucket and the remaining in 2nd bucket.
Hahaha....good cover drive
Ha ha it's not like that , and the records will not store in the order the way we insert in the bucket , the hash partition algorithm will decide the buckets destinations for the records , 0014 can be in 1st bucket too . Using bucket we are not completely avoiding full scannn... 🙂
Thankyou
First view 😀