Correction: 10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42. NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent. NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps. NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: th-cam.com/video/_L39rN6gz7Y/w-d-xo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct. Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
AdaBoost: Forest of Stumps 1:30 stump: a tree just with 1 node and 2 leaves. 3:30 AdaBoot: Forest of Stumps; Different stumps have different weight/say/voice; Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting) 6:407:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples)) 7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1. (0 means perfect stump; 1 means horrible stump) --[class notes]
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send. Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
Dude anything I try to learn related to machine learning or statistics, your video pops up at the top. Thanks a bunch for making all these fun videos! Using your video not only to understand stuff but also to explain it to other people!
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
Take the example of Chest Pain Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index. Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8 No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8 Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42. I also join others in asking to talk about Gradient Boosting next time. Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice. PS, how to PIN a comment (I paste it here to save your precious time ^_^) : - Sign in to TH-cam. - In the comments below a video, select the comment you want like to pin. - Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ... - Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3) How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately TH-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
Hi Josh, great video as always! Questions: 1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating? 2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps? 3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different? Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with. 2) Yes (so, consider setting the seed for the random number function first). 3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
Hi Josh, I love your videos so much! You are awesome!! A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label? Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy? Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
Hello Sir, I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc. Thanks!
Thanks a lot, I didn't understand my UCSD lecture but, thanks to you (and your team mate), It's now super clear! I've bought AdaBoost and Classification tree study guide, great help! 1/ Could you consider making a study guide about the 4 gradient boost videos? 2/ Did you ever think writing a book with your team? Have a great day.
I have some questions. 1. When do we stop calculating new stumps and their amount of say ? I mean how do we know when we have enough stumps to work with ? 2. You mentioned in the comment below that "once an observation is omitted from a bootstrap dataset, is it not lost for good. You just lose it for one stump. After that it goes back in the pool" When do we come back to an earlier data set in order to make use of a "omitted observation" ? As far as I understood, we only work with the modified dataset which keeps getting smaller and smaller (in terms of unique items)
1) You keep adding stumps until predictions no longer improve. 2) After each stump is created, we return all of the omitted samples back to the dataset.
I swear that is the greatest channel about machine learning and statistics , Great job josh I just have a quick question : what if we have stump that both children say (left children get 2 yes 0 no ) and (right children get 2 yes and 1 no) and that is the best we could come with so what should i do ? i saw a different video and i found that he classify left children as yes and right children with no and he say the first stump make 2 errors but how !!!!!!!!!!!!!!!!!! we say that the node vote for a majority so it should be both say yes ,and the right child get 1 error so first stump made one error , right ?
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Thanks for all these wonderful videos. They make ML easy for many new learners! I just have few quick questions: 1. No of stumps depend on the no of features in the dataset, right? If there are only 5 features, only 5 stumps can be created, right (or less than that if we don't plan to use all stumps) 2. As "Amount of Say = 1/2*log[(1-Total error)/Total error]", we get positive or negative values for "Amount of Say" depending upon "Total Error". So when we try to calculate "New sample weight" do we need to change the sign (+/-) of the exponent explicitly? Doesn't it appear in the "Amount of Say" correctly? 3. Can you kindly show how a new test sample get evaluated in Ada boost? Appreciate your thoughts on these. Thanks
1. Each time we create a stump, we test all of the features (so, if we have 5 features, we'll test 5 candidate stumps), but we can create as many stumps as we would like. 2. What time point in the video, minutes and seconds, are you asking about? 3. This is illustrated at the end of the video.
@@madaramarasinghe829 If the sample was correctly classified we use the negative version to decrease its weight. If the sample was incorrectly classified we use the positive version to increase its weight.
For my future reference. 11:36: If prediction for a sample was wrong, then increase its weight for future correction. If Amount Of Say is high (the tree is good), increase the weight more. Wrong - > Increase weight. Better Tree -> more amount of say -> adjust weight more
May I ask a couple of questions please? Hopefully, you can help shed some light. 1. What is the threshold for a prediction flip? Do we flip as long as total error rate is larger than 0.5, i.e., amount of say is negative? Or we only flip if total error rate is larger than 0.6 or 0.7? Or it is a hyper-parameter to be tuned? 2. Can we say that weighted Gini index and sample weights update achieve the same goal but through different mechanisms? Recall that in order to mitigate class imbalance, we could either apply class weights or sample weights. I see the resemblance here. Thank you!
@@jiayiwu4101 Presumably. In theory, AdaBoost is intended to be used with _any_ weak learner, and maybe there is some weak learner that is really, really bad. However, AdaBoost is almost always used with classification tree stumps. And with stumps, I don't think it is possible to have error > 0.5.
Hello Josh. Thanks for an awesome video on Adaboost. Can you make a video of the difference between Natural log and common log, when should we use each of them and the properties and real life applications? I had hard time figuring out why the amount of say was 0.97 at 9:22 in the video. I was calculating it using the log key on my calculator whereas in reality it was the natural log. If I take 0.5*log(7) then my amount of say would be 0.422 whereas in reality you have used 0.5*ln(7) which gives 0.97 as the amount of say. I believe not many people would come to know that the log is natural and that I think it will be great if you use ln for natural log henceforth to avoid confusion. That being said, I found all your videos super exciting and thanks for sharing the knowledge.
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
Pretty good explanations, given you survive the deadly level of condescension. I almost switched to some indian dude explaining the same thing, but he exceeded my max limit of "basically"s, and I switched back just to get the grasp of the subject. While at it, I realised what this statquest tone reminds me of. The Fun with Flags Show of Sheldon Cooper. But when he goes "Question!", it is definitely Dwight Schrute from The Office. The two of my favorite sitcoms characters!
You are correct! However, this typo has been spotted before and is mentioned towards the top of the video's Description. Unfortunately, youtube makes it close to impossible to edit videos once they are posted, so for now, the best I can do is leave the note in the description.
Dear Josh, I have 2 more questions. For Random Forest, we use bagging to generate many trees. But you haven't introduced this idea for AdaBoost. 1. Does that mean, the `amount of say` of all trumps is determined by the whole data set (corpus)? 2. How many AdaBoost stumps we should generate? What's the rule to determine the number? 3. Could a stump share the same features with other stumps? And, as you mentioned, the creation of the first stump will effect the second stump. My second question is: 1. How to determine which feature we should use for the first stump? By entropy/Gini comparison between the features? 2. If it is, is the idea of the first stump creation applied to the second stump, thus third, fourth, etc. Thanks.
1) The amount of say is determined using the current dataset, which can change for each stump, as explained at 15:32 using the sample weights. 2) You should generate enough so that classification stabilizes. You can draw some metric you are interested in (for example, "accuracy") on the y-axis and the number of stumps on the x-axis. Create stumps until you see that predictions are no longer improving. 3) You can use the same feature over and and over. 1) You can use whatever metric you want to you. I believe GINI is frequently the default setting. 2) All stumps are made the same way, however the dataset can change for each stump due to changes in sample weights as explained at 15:32.
Correction:
10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42.
NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest
NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent.
NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps.
NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: th-cam.com/video/_L39rN6gz7Y/w-d-xo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct.
Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
Isn't it be 0 .1109?
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
you should pin this to the top
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
First, thank you for those great videos. I have the same question that Tim asked. How does omitted observations get "back into the pool"?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Thank you! :)
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
Awesome, thank you!
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
bam!
TRIPLE BAM!!!
Man right here just clarified my 2-hour lecture in 20 mins. Thank you.
Bam! :)
Wow. I cannot emphasize on how much I'm learning from your series on machine learning. Thank you so much! :D
Hooray! I'm glad the videos are helpful. :)
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
Hooray!!! I'm so happy you like them. As soon as I finish my videos on Neural Networks, I'll start making more study guides.
This video is just beyond excellent. Crystal clear explanation, no one could not have done it better. Thank you, Josh.
Thank you!
Best video of Ada Boost on the TH-cam, watched it two times to understand it fully.
It's such a beautiful explanation...
Thank you! :)
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
BAM! :)
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
Hooray!! Thank you very much! :)
This is by far the best explanatory video on "AdaBoost" that I have come across.
Thanks!
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
Thanks! :)
You are THE BEST, can't tell how much i've got to learn from statquest!!!
Awesome! Thank you so much! :)
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
Thank you very much!!! I'm glad you like my videos and good luck with your studies!
I should have checked here instead of everywhere else. Josh sings a song and explains things so clearly. Love the channel. Thanks again!
Thanks!
AdaBoost: Forest of Stumps
1:30 stump: a tree just with 1 node and 2 leaves.
3:30 AdaBoot: Forest of Stumps;
Different stumps have different weight/say/voice;
Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting)
6:40 7:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples))
7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1.
(0 means perfect stump; 1 means horrible stump)
--[class notes]
bam!
Thank you for this. These videos are concise and easy to understand. Also, your humor is 10/10
Thanks! :)
You're my hero, Josh! This is so much more understandable than twisted formulas.
Thank you! :)
How come I missed this channel for so long? Absolutely brilliant.
Thank you!
I always love Josh's Videos. There is a minor calculation error while calculating amount of say for chest pain stump. (1-3/8)/(3/8) = 5/3, not 7/3
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
Merci!
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send.
Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
Hi Josh Starmer ,
A huge BAM for this video.
The best explanation I have ever seen for Adaboost.
Keep helping people.
Glad it was helpful!
I love this format, you're great.
RiTeh strojno mafija where you at?
Wow, you explained the concept of bootstrapping so easily without even mentioning it! Impressive!
Bam! :)
AdaBoost -> Gradient Boosting -> XGBoost series will be awesome! First step AdaBoost clearly explained : )
I'm just putting the finishing touches on Gradient Descent, which will come out in a week or so, then Gradient Boosting and XGBoost.
That sounds great@@statquest! I guess you are the Machine Teaching
@@statquest I'm waiting this as well!
@@statquest when will you post Gradient Boosting and XGBoost?
@@statquest waiting for Gradient Boosting and XGBoost
I get impressed by each video of yours..and in free time recapitulated what you taught in the videos, sometimes. Awesome Josh!!!
Awesome! Thank you!
"Biiii dooo dooo bo di doo di boo diiiii doooo" This is exactly how my brain reacts when it sees mathematical calculations.
bam! :)
Dude , you are brilliant brilliant brilliant , how did you come with this kind of teaching style , Clearly Explained !!
Thanks a ton!
Your tutorials are simply awesome Josh! You are a great help!
Glad you like them!
Your channel is the best one about Stats I found so far
Wow, thanks!
Please, Can anyone make 10 hours version 'dee doo dee doo boop'?
You made me laugh out loud! :)
sounds good, i want it too
@@statquest I would seriously play it for my background music during work... Please make one lol.
I also want some 'dee doo Dee doo boop '
@StatQuest how to apply adaboost for regression?
Even somebody who doesn't know English, could understand machine learning with your videos
Thank you!
That opening scared me..😅😅
You were scared to learn that ML is not so complicated? BAMM!
Lolo
This is very enjoying and yet understandable to watch, best adaboost explanation, I've watch almost all the video here
Wow, thanks!
Josh, you're the best. Your explanations are easy to understand, plus your songs crack my girlfriend up.
That's awesome!! :)
Dude anything I try to learn related to machine learning or statistics, your video pops up at the top. Thanks a bunch for making all these fun videos! Using your video not only to understand stuff but also to explain it to other people!
Awesome, thank you!
HAHA love your calculation sound :D :D :D
I will recommend this channel for as many as I can
Thank you very much! :)
The real question is: Is there a model which can predict the volume of "bam" sound ?
Great Question! :)
@@statquest 😆😆
The Bam has total error 0, so the amount of say will freak out :)
Thank you Statquest. Was eagerly waiting for Adaboost, Clearly Explained.
Hooray!!! :)
I wish math in real life happened as fast as 'dee doo dee doo boop' :D
Me too! :)
pee dee doo poo dee deee poop.
I am in love with this channel. I think the main reason is the Josh explanation style :D
Thanks! 😃
The explanation is brilliant, thank so much for keeping things so simple
Thank you!
u just cant imagine how great this way .. this could not be learnt better than this video
Thanks again! :)
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
I am confused as well :(
It is same as Gini Impurity in Decision Tree video.
Take the example of Chest Pain
Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category
Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category
Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index.
Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8
No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8
Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
@@DawFru thanks buddy
No wonder why AdaBoost takes looong time to run! Thank you for the nice explanation as always!
Thanks!
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42.
I also join others in asking to talk about Gradient Boosting next time.
Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice.
PS, how to PIN a comment (I paste it here to save your precious time ^_^) :
- Sign in to TH-cam.
- In the comments below a video, select the comment you want like to pin.
- Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ...
- Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
such an amaizng explanation, intuitvely shows how Ada boost helps in making the model better than decision tree.
Thanks!
such a complex concept you explained with ease.. Awesome video
Thank you! :)
had to watch two times to fully grasp the concept.. Worth every minute :)
Awesome! Thanks for giving the video a second chance. :)
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
You are correct - the mistakes are additive.
@@statquest Please fix the video because that's the confusion I came here to rectify. That's a big mistake.
Actually having read the original AdaBoost authors now, I don't think the training model is a sum of the previous models..?
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
Awesome! Gradient Boosting should be available soon.
Thanks, that will be very helpful!
@@atinsingh164 I'm working on it right now.
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
I'll make a special dedication to this man on the day of my graduation.
bam! :)
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3)
How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately TH-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
Hi Josh, great video as always!
Questions:
1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating?
2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps?
3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different?
Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with.
2) Yes (so, consider setting the seed for the random number function first).
3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
Hi Josh,
I love your videos so much! You are awesome!!
A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label?
Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy?
Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
@@jinqiaoli8985 I can't wait to release the next XGBoost video. I just have a few more slides to work on before it's ready.
Hello Sir,
I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc.
Thanks!
Thank you so much! I'm working on Gradient Descent right now. I hope it is ready in the next week or two.
Hi Josh you are doing great job. Can you please make a video on Xgboost. That will be very helpful
the most underrated channel
Thank you! :)
Hi Josh,
Love your videos from India,
Can you please tell me how to calculate the amount of say in regression case and also the sample weights?
Thanks
Did you get your answer? If yes, could you please explain
Thanks a lot,
I didn't understand my UCSD lecture but, thanks to you (and your team mate), It's now super clear!
I've bought AdaBoost and Classification tree study guide, great help!
1/ Could you consider making a study guide about the 4 gradient boost videos?
2/ Did you ever think writing a book with your team?
Have a great day.
Thank you very much for supporting StatQuest!!! I'm writing a book right now. It should be out later this spring.
@@statquest I'll buy it as fast as a bam!
"Devmaanush" hai ye banda!
Translation: This dude has been sent by God!
Thank you very much! :)
Understood clearly, it took 2 hours to complete this, but now i know start to finish
Hooray!
Ammount of say for chest pain how 7/3 i think it will be 5/3
I have some questions.
1. When do we stop calculating new stumps and their amount of say ? I mean how do we know when we have enough stumps to work with ?
2. You mentioned in the comment below that "once an observation is omitted from a bootstrap dataset, is it not lost for good. You just lose it for one stump. After that it goes back in the pool"
When do we come back to an earlier data set in order to make use of a "omitted observation" ? As far as I understood, we only work with the modified dataset which keeps getting smaller and smaller (in terms of unique items)
1) You keep adding stumps until predictions no longer improve.
2) After each stump is created, we return all of the omitted samples back to the dataset.
Love the opening music, make me laugh at machine learning course. What an odd!
Hooray! :)
Thanks, Josh, your explanation is amazing. Greetings from Egypt
Thank you! :)
10:15 Warning wrong calculation alert
it is 5/8 not 7/8 since it is 1-3/8
and your remaining part fxxk up!
There is also another error in the formula. The formula should be with ln instead of log!
I swear that is the greatest channel about machine learning and statistics , Great job josh
I just have a quick question :
what if we have stump that both children say (left children get 2 yes 0 no ) and (right children get 2 yes and 1 no) and that is the best we could come with so what should i do ? i saw a different video and i found that he classify left children as yes and right children with no and he say the first stump make 2 errors but how !!!!!!!!!!!!!!!!!!
we say that the node vote for a majority so it should be both say yes ,and the right child get 1 error so first stump made one error , right ?
I believe they should both vote yes.
It'd be really refreshing to hear an actual model make dee doo dee doo boop' sounds while training.
:)
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Glad it was helpful!
Hey Josh, I'm back to study Machine Learning for Final Exams 😂😂😂
Good luck and let me know how they go! :)
Thanks for all these wonderful videos. They make ML easy for many new learners!
I just have few quick questions:
1. No of stumps depend on the no of features in the dataset, right? If there are only 5 features, only 5 stumps can be created, right (or less than that if we don't plan to use all stumps)
2. As "Amount of Say = 1/2*log[(1-Total error)/Total error]", we get positive or negative values for "Amount of Say" depending upon "Total Error". So when we try to calculate "New sample weight" do we need to change the sign (+/-) of the exponent explicitly? Doesn't it appear in the "Amount of Say" correctly?
3. Can you kindly show how a new test sample get evaluated in Ada boost?
Appreciate your thoughts on these. Thanks
1. Each time we create a stump, we test all of the features (so, if we have 5 features, we'll test 5 candidate stumps), but we can create as many stumps as we would like.
2. What time point in the video, minutes and seconds, are you asking about?
3. This is illustrated at the end of the video.
@@statquest
Thanks for the quick resopnses.
Re 2: At 12:43 you talk about e^(amount of say) and at 13:13 you talk about e^(- amount of say)
@@madaramarasinghe829 If the sample was correctly classified we use the negative version to decrease its weight. If the sample was incorrectly classified we use the positive version to increase its weight.
I love your Songs..
Please make a video on XGBoost .
Thanks! I'm working on Gradient Descent and then Gradient Boost. Those should be out soon.
Thank you man. Appreciate such thorough but concise explanation.
Glad it was helpful!
tripple baｍ
Thanks! :)
Man, you are too good at explaining things!
Thanks!
OMG! I'm completely delighted with your with your didactics! Awesome! Your students are very privileged! Gosh! Regards from Brazil fan :D
Muito obrigado!!!! BAM! :)
@@statquest BAM!!! :D
Josh and Kahn, the greatest teachers of all time. They should have their own University.
bam! :)
Thank you so much Josh... u filled josh in me ..(josh means happy in hindi) love from Hyderabad.INDIA....❤
Thank you very much! :)
For my future reference.
11:36:
If prediction for a sample was wrong, then increase its weight for future correction.
If Amount Of Say is high (the tree is good), increase the weight more.
Wrong - > Increase weight.
Better Tree -> more amount of say -> adjust weight more
Noted
May I ask a couple of questions please? Hopefully, you can help shed some light.
1. What is the threshold for a prediction flip? Do we flip as long as total error rate is larger than 0.5, i.e., amount of say is negative? Or we only flip if total error rate is larger than 0.6 or 0.7? Or it is a hyper-parameter to be tuned?
2. Can we say that weighted Gini index and sample weights update achieve the same goal but through different mechanisms? Recall that in order to mitigate class imbalance, we could either apply class weights or sample weights. I see the resemblance here.
Thank you!
Presumably you flip if the error rate is larger than 0.5 because flipping will improve predictions.
@@statquest thank you. ❤️
@@statquest So the error rate should not be larger than 0.5. Is this statement correct?
@@jiayiwu4101 Presumably. In theory, AdaBoost is intended to be used with _any_ weak learner, and maybe there is some weak learner that is really, really bad. However, AdaBoost is almost always used with classification tree stumps. And with stumps, I don't think it is possible to have error > 0.5.
This guy's voice is so calming.
:)
Thank you Josh for all of your great videos. You are a good Samaritan!
Awesome! Thank you! :)
Hello Josh. Thanks for an awesome video on Adaboost. Can you make a video of the difference between Natural log and common log, when should we use each of them and the properties and real life applications? I had hard time figuring out why the amount of say was 0.97 at 9:22 in the video. I was calculating it using the log key on my calculator whereas in reality it was the natural log. If I take 0.5*log(7) then my amount of say would be 0.422 whereas in reality you have used 0.5*ln(7) which gives 0.97 as the amount of say. I believe not many people would come to know that the log is natural and that I think it will be great if you use ln for natural log henceforth to avoid confusion. That being said, I found all your videos super exciting and thanks for sharing the knowledge.
Thanks for the note. In all of my recent videos I explain what I mean when I write "log()".
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
The output from the leaves is always the classification that gets the most votes. So this stump would classify everything the same way.
i love your intros and outros, are simply awesome!!! i really enjoy learning with your channel!!!!
Thanks! :)
Very nicely explained. I have never seen such a good explanation. Love U ♥♥♥
Thank you! :)
Pretty good explanations, given you survive the deadly level of condescension. I almost switched to some indian dude explaining the same thing, but he exceeded my max limit of "basically"s, and I switched back just to get the grasp of the subject. While at it, I realised what this statquest tone reminds me of. The Fun with Flags Show of Sheldon Cooper. But when he goes "Question!", it is definitely Dwight Schrute from The Office. The two of my favorite sitcoms characters!
I think I have "resting condescension" voice. I swear, it's not something I'm trying to do to make people annoyed - it's just my normal voice.
Hi Josh , very nice video i did not find such a nice explanation. One feedback in calculation at 10:18 , the numerator should be 5/8 not 7/8
You are correct! However, this typo has been spotted before and is mentioned towards the top of the video's Description. Unfortunately, youtube makes it close to impossible to edit videos once they are posted, so for now, the best I can do is leave the note in the description.
hi smal mistake: 1- 3/8 should 5/8 at 10:31 rest all good, love your videos!
Thanks! That error is noted in a pinned comment and in the video's description.
This is very informative for me. I skipped the Decision Tree but... but... I can understand it! Love your vids!
Thanks!
Please never stop making videos
Thank you! :)
The best! The simplest! The most informative!
Thank you! :)
Your channel has saved me a looooooot of time. Thanks!
Glad to hear it!
Dear Josh,
I have 2 more questions.
For Random Forest, we use bagging to generate many trees. But you haven't introduced this idea for AdaBoost.
1. Does that mean, the `amount of say` of all trumps is determined by the whole data set (corpus)?
2. How many AdaBoost stumps we should generate? What's the rule to determine the number?
3. Could a stump share the same features with other stumps?
And, as you mentioned, the creation of the first stump will effect the second stump. My second question is:
1. How to determine which feature we should use for the first stump? By entropy/Gini comparison between the features?
2. If it is, is the idea of the first stump creation applied to the second stump, thus third, fourth, etc.
Thanks.
1) The amount of say is determined using the current dataset, which can change for each stump, as explained at 15:32 using the sample weights.
2) You should generate enough so that classification stabilizes. You can draw some metric you are interested in (for example, "accuracy") on the y-axis and the number of stumps on the x-axis. Create stumps until you see that predictions are no longer improving.
3) You can use the same feature over and and over.
1) You can use whatever metric you want to you. I believe GINI is frequently the default setting.
2) All stumps are made the same way, however the dataset can change for each stump due to changes in sample weights as explained at 15:32.
@@statquest Thank you very much, Josh. Now I understand the text description at 15:32.
thank you for making these videos, they are really helping me with my ML class!
Hooray! And good luck with your class.