Correction: 10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42. NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent. NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps. NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: th-cam.com/video/_L39rN6gz7Y/w-d-xo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
AdaBoost: Forest of Stumps 1:30 stump: a tree just with 1 node and 2 leaves. 3:30 AdaBoot: Forest of Stumps; Different stumps have different weight/say/voice; Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting) 6:407:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples)) 7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1. (0 means perfect stump; 1 means horrible stump) --[class notes]
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
Hi Josh, great video as always! Questions: 1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating? 2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps? 3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different? Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with. 2) Yes (so, consider setting the seed for the random number function first). 3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
Take the example of Chest Pain Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index. Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8 No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8 Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send. Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42. I also join others in asking to talk about Gradient Boosting next time. Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice. PS, how to PIN a comment (I paste it here to save your precious time ^_^) : - Sign in to TH-cam. - In the comments below a video, select the comment you want like to pin. - Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ... - Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
May I ask a couple of questions please? Hopefully, you can help shed some light. 1. What is the threshold for a prediction flip? Do we flip as long as total error rate is larger than 0.5, i.e., amount of say is negative? Or we only flip if total error rate is larger than 0.6 or 0.7? Or it is a hyper-parameter to be tuned? 2. Can we say that weighted Gini index and sample weights update achieve the same goal but through different mechanisms? Recall that in order to mitigate class imbalance, we could either apply class weights or sample weights. I see the resemblance here. Thank you!
@@jiayiwu4101 Presumably. In theory, AdaBoost is intended to be used with _any_ weak learner, and maybe there is some weak learner that is really, really bad. However, AdaBoost is almost always used with classification tree stumps. And with stumps, I don't think it is possible to have error > 0.5.
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3) How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately TH-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
Hi Josh, I love your videos so much! You are awesome!! A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label? Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy? Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
Q1. 11:58 since amt of say can be negative as well, shouldn't the graph and x-axis extend towards the left? Q2. If a tree has a negative amount of say then a correctly classified sample will be assigned a higher weight than the sample incorrectly classified. Looks confusing why you would assign a higher weight to a sample that has been correctly classified even if it the tree overall has a negative amount of say.
A1. Sure, you can extend the graph in the negative direction. You get values closer and closer to 0 the more negative the amount of say is. A2: If a tree has a negative amount of say, that means it said most of patients in the training dataset with heart disease did not have have heart disease and most of the patients without heart disease had heart disease. Thus, if this tree "correctly" classifies a new sample, it grouped it with the observations with the opposite value, which means it did a bad job categorizing the samples. Thus, we need to spend more effort trying to group it with the same value.
Hi Josh, love the videos. Just one question. In the video of decision tree, when checking what feature should be at the root node, we just classified, then calculated the gini impurity(gini index, from this video as I am aware), but in 5:52 , by classifying the samples by the condition 'Has chest Pain', if the sample has chest pain, the 'correct' means that you have disease, and if you don't have chest pain, the 'correct' means that you don't have heart disease? but why do we set the conditions of 'correct' differently here? Did I miss anything?
In the stump that uses "chest pain" to classify people, people with chest pain go to the left and people who do not have chest pain go to the right. Now, if most of the people that go to the left have heart disease, than that node will classify everyone who goes to the left as someone who has heart disease. This means that it will correctly classify most of the people who go to the left, but it will not correctly classify everyone. Does that make sense?
Thanks Josh! I have 3 questions: 1. @3:43 you say weak learners are "almost always stumps" is there a case where it is not a stump? 1.a. also what is adavtage of using stump over bigger trees? 2. Does boosting algorithm only use decision trees?
1) I don't know. However, it's possible and easily done (i.e. in sklearn you just change the default value for "depth"). 2) It probably depends on your data. 3) In theory, no. In practice, yes.
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
One thing I am not really sure about is weather when creating stump 3, stump 1 and 2 should be considered or just stump 2. I watched a video on Boosting where they mentioned that the previous stumps are combined in order to create the new one. The combined wrong classifications are then used to sample the new set which is used to create the new stump. Could someone please clearify
@@statquest I understand but when would we prefer one over the other? (Considering all previous weak learners over just the previous stump). Thank you for taking the time to reply.
@@anelm.5127 Unfortunately I don't understand your question because in adaboost, there is no choice to be made. The weights assigned to each sample a shaped by every single stump we make, never just the last one.
Hello Sir, I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc. Thanks!
Hello Josh, Thank you for the amazing videos. I had a couple of questions on stumps that are created after Sample weights are updated at time 16:00. We continue with sampling from the full set assuming new weights. This means the new set will be a subset of the original dataset as you explained in 18:32 . Going forward all subsequent stumps will only work on smaller and smaller subsets, making it a bit confusing for me on how to ensure good randomness. 1. Do we also restart from the updated sample weights at 16:00 and redo sampling, thereby creating multiple different datasets but using the same sample weight values? This will probably ensure we use all the data from the original data set in some instances. 2. As a follow-up to Q1, do we perform multiple sampling at different levels to get more variations in the dataset before creating stumps? In the video, you described only one instance of sampling using new weights but I assume it needs to be performed multiple times to get variations in datasets. 3. Do we not sample from distribution at the very beginning also? All sample weights will have the same value but random sampling would mean some might not make it to the first round of ( stump creation + Amount of say + New sample weights ) itself to get unequal sample weights at 16:00. In the video you take all the samples by default hence the query.
18:07 do we use bootstrapping in Adaboost? so if we're sampling with replacement there's no gaurantee that the sample with the larger weight is duplicated more
Excellent tutorial, thanks! Would like to ask: 1) after recalculating and updating weights for each sample, the process repeats. And in first step we found the 'weight' has the lowest Gini, then in next round we exclude the 'weight' and only consider the remaining criteria? or not? I see the 'weight' is reused when bootstrapping is used, but how about if we still using Gini? 2) if same set of criteria is re-used, then we would get a bunch of stumps with same criteria but differ only with the amount of say. Why is that useful in this case, other than simply using each criteria once only? 3) if the same set of criteria is re-used, then how can we determine when to stop the stump building process and settle down to calculate which classification result group has larger amount of say? Thx~
Every stump selects from the exact same features (in this case, the features are "chest pain", "blocked arteries" and "patient weight"), however, the sample weights are always changing and this results in bootstrap datasets that contain different samples for each stump to divide. That said, while "Patient weight" might work well on the first stump, it might not work well in the second stump. This is because every sample that "Patient weight" misclassified will have a larger weight and thus, a larger probability of being included in the next bootstrapped dataset.
Many thanks for explaining the Adaboost in such a easy manner. I have one doubt at 19:39 minutes, where you have calculated the amount of say for "Does not have heart diseases". How you are calculating 0.41 and 0.82 values. Kindly explain.
I used the exact same method as demonstrated at 6:27. However, in order to save time, I did not show every single step of the process like I did the first time. However, you can imagine that that is what I did.
Correction:
10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42.
NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest
NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent.
NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps.
NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: th-cam.com/video/_L39rN6gz7Y/w-d-xo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Isn't it be 0 .1109?
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
you should pin this to the top
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
First, thank you for those great videos. I have the same question that Tim asked. How does omitted observations get "back into the pool"?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Thank you! :)
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
bam!
TRIPLE BAM!!!
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
Awesome, thank you!
Man right here just clarified my 2-hour lecture in 20 mins. Thank you.
Bam! :)
Wow. I cannot emphasize on how much I'm learning from your series on machine learning. Thank you so much! :D
Hooray! I'm glad the videos are helpful. :)
Best video of Ada Boost on the TH-cam, watched it two times to understand it fully.
It's such a beautiful explanation...
Thank you! :)
This video is just beyond excellent. Crystal clear explanation, no one could not have done it better. Thank you, Josh.
Thank you!
AdaBoost: Forest of Stumps
1:30 stump: a tree just with 1 node and 2 leaves.
3:30 AdaBoot: Forest of Stumps;
Different stumps have different weight/say/voice;
Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting)
6:40 7:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples))
7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1.
(0 means perfect stump; 1 means horrible stump)
--[class notes]
bam!
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
BAM! :)
You are THE BEST, can't tell how much i've got to learn from statquest!!!
Awesome! Thank you so much! :)
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
Hooray!!! I'm so happy you like them. As soon as I finish my videos on Neural Networks, I'll start making more study guides.
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
Hooray!! Thank you very much! :)
This is by far the best explanatory video on "AdaBoost" that I have come across.
Thanks!
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
Thank you very much!!! I'm glad you like my videos and good luck with your studies!
I should have checked here instead of everywhere else. Josh sings a song and explains things so clearly. Love the channel. Thanks again!
Thanks!
Thank you for this. These videos are concise and easy to understand. Also, your humor is 10/10
Thanks! :)
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
Merci!
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
Thanks! :)
How come I missed this channel for so long? Absolutely brilliant.
Thank you!
I always love Josh's Videos. There is a minor calculation error while calculating amount of say for chest pain stump. (1-3/8)/(3/8) = 5/3, not 7/3
Hi Josh Starmer ,
A huge BAM for this video.
The best explanation I have ever seen for Adaboost.
Keep helping people.
Glad it was helpful!
AdaBoost -> Gradient Boosting -> XGBoost series will be awesome! First step AdaBoost clearly explained : )
I'm just putting the finishing touches on Gradient Descent, which will come out in a week or so, then Gradient Boosting and XGBoost.
That sounds great@@statquest! I guess you are the Machine Teaching
@@statquest I'm waiting this as well!
@@statquest when will you post Gradient Boosting and XGBoost?
@@statquest waiting for Gradient Boosting and XGBoost
Wow, you explained the concept of bootstrapping so easily without even mentioning it! Impressive!
Bam! :)
You're my hero, Josh! This is so much more understandable than twisted formulas.
Thank you! :)
Even somebody who doesn't know English, could understand machine learning with your videos
Thank you!
The real question is: Is there a model which can predict the volume of "bam" sound ?
Great Question! :)
@@statquest 😆😆
The Bam has total error 0, so the amount of say will freak out :)
This is very enjoying and yet understandable to watch, best adaboost explanation, I've watch almost all the video here
Wow, thanks!
I love this format, you're great.
RiTeh strojno mafija where you at?
Your channel is the best one about Stats I found so far
Wow, thanks!
That opening scared me..😅😅
You were scared to learn that ML is not so complicated? BAMM!
Lolo
Dude , you are brilliant brilliant brilliant , how did you come with this kind of teaching style , Clearly Explained !!
Thanks a ton!
"Biiii dooo dooo bo di doo di boo diiiii doooo" This is exactly how my brain reacts when it sees mathematical calculations.
bam! :)
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
The output from the leaves is always the classification that gets the most votes. So this stump would classify everything the same way.
Please, Can anyone make 10 hours version 'dee doo dee doo boop'?
You made me laugh out loud! :)
sounds good, i want it too
@@statquest I would seriously play it for my background music during work... Please make one lol.
I also want some 'dee doo Dee doo boop '
@StatQuest how to apply adaboost for regression?
I get impressed by each video of yours..and in free time recapitulated what you taught in the videos, sometimes. Awesome Josh!!!
Awesome! Thank you!
Your tutorials are simply awesome Josh! You are a great help!
Glad you like them!
I will recommend this channel for as many as I can
Thank you very much! :)
Josh, you're the best. Your explanations are easy to understand, plus your songs crack my girlfriend up.
That's awesome!! :)
had to watch two times to fully grasp the concept.. Worth every minute :)
Awesome! Thanks for giving the video a second chance. :)
The explanation is brilliant, thank so much for keeping things so simple
Thank you!
Hi Josh, great video as always!
Questions:
1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating?
2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps?
3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different?
Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with.
2) Yes (so, consider setting the seed for the random number function first).
3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
HAHA love your calculation sound :D :D :D
such an amaizng explanation, intuitvely shows how Ada boost helps in making the model better than decision tree.
Thanks!
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
I am confused as well :(
It is same as Gini Impurity in Decision Tree video.
Take the example of Chest Pain
Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category
Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category
Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index.
Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8
No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8
Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
@@DawFru thanks buddy
No wonder why AdaBoost takes looong time to run! Thank you for the nice explanation as always!
Thanks!
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send.
Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
Thank you Statquest. Was eagerly waiting for Adaboost, Clearly Explained.
Hooray!!! :)
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42.
I also join others in asking to talk about Gradient Boosting next time.
Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice.
PS, how to PIN a comment (I paste it here to save your precious time ^_^) :
- Sign in to TH-cam.
- In the comments below a video, select the comment you want like to pin.
- Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ...
- Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
May I ask a couple of questions please? Hopefully, you can help shed some light.
1. What is the threshold for a prediction flip? Do we flip as long as total error rate is larger than 0.5, i.e., amount of say is negative? Or we only flip if total error rate is larger than 0.6 or 0.7? Or it is a hyper-parameter to be tuned?
2. Can we say that weighted Gini index and sample weights update achieve the same goal but through different mechanisms? Recall that in order to mitigate class imbalance, we could either apply class weights or sample weights. I see the resemblance here.
Thank you!
Presumably you flip if the error rate is larger than 0.5 because flipping will improve predictions.
@@statquest thank you. ❤️
@@statquest So the error rate should not be larger than 0.5. Is this statement correct?
@@jiayiwu4101 Presumably. In theory, AdaBoost is intended to be used with _any_ weak learner, and maybe there is some weak learner that is really, really bad. However, AdaBoost is almost always used with classification tree stumps. And with stumps, I don't think it is possible to have error > 0.5.
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
u just cant imagine how great this way .. this could not be learnt better than this video
Thanks again! :)
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
You are correct - the mistakes are additive.
@@statquest Please fix the video because that's the confusion I came here to rectify. That's a big mistake.
Actually having read the original AdaBoost authors now, I don't think the training model is a sum of the previous models..?
I am in love with this channel. I think the main reason is the Josh explanation style :D
Thanks! 😃
I wish math in real life happened as fast as 'dee doo dee doo boop' :D
Me too! :)
pee dee doo poo dee deee poop.
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Glad it was helpful!
such a complex concept you explained with ease.. Awesome video
Thank you! :)
Thank you so much Josh... u filled josh in me ..(josh means happy in hindi) love from Hyderabad.INDIA....❤
Thank you very much! :)
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3)
How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately TH-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
Thank you for your effort, but I have a question: how do we calculate the "Gini index"? (6:21)
Gini is explained in my video on decision trees: th-cam.com/video/_L39rN6gz7Y/w-d-xo.html
Hi Josh,
Love your videos from India,
Can you please tell me how to calculate the amount of say in regression case and also the sample weights?
Thanks
Did you get your answer? If yes, could you please explain
Man, you are too good at explaining things!
Thanks!
Hi Josh,
I love your videos so much! You are awesome!!
A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label?
Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy?
Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
@@jinqiaoli8985 I can't wait to release the next XGBoost video. I just have a few more slides to work on before it's ready.
Thanks, Josh, your explanation is amazing. Greetings from Egypt
Thank you! :)
Hi Josh you are doing great job. Can you please make a video on Xgboost. That will be very helpful
"Ohh NOOO!! Gratitude alert! " -> "Thanks a lot for your videos. They are just awesome.!"
You're the best!
"Devmaanush" hai ye banda!
Translation: This dude has been sent by God!
Thank you very much! :)
the most underrated channel
Thank you! :)
Ammount of say for chest pain how 7/3 i think it will be 5/3
Thank you man. Appreciate such thorough but concise explanation.
Glad it was helpful!
10:15 Warning wrong calculation alert
it is 5/8 not 7/8 since it is 1-3/8
and your remaining part fxxk up!
There is also another error in the formula. The formula should be with ln instead of log!
Q1. 11:58 since amt of say can be negative as well, shouldn't the graph and x-axis extend towards the left?
Q2. If a tree has a negative amount of say then a correctly classified sample will be assigned a higher weight than the sample incorrectly classified. Looks confusing why you would assign a higher weight to a sample that has been correctly classified even if it the tree overall has a negative amount of say.
A1. Sure, you can extend the graph in the negative direction. You get values closer and closer to 0 the more negative the amount of say is.
A2: If a tree has a negative amount of say, that means it said most of patients in the training dataset with heart disease did not have have heart disease and most of the patients without heart disease had heart disease. Thus, if this tree "correctly" classifies a new sample, it grouped it with the observations with the opposite value, which means it did a bad job categorizing the samples. Thus, we need to spend more effort trying to group it with the same value.
Love the opening music, make me laugh at machine learning course. What an odd!
Hooray! :)
Thank you Josh for all of your great videos. You are a good Samaritan!
Awesome! Thank you! :)
I love your Songs..
Please make a video on XGBoost .
Thanks! I'm working on Gradient Descent and then Gradient Boost. Those should be out soon.
Very nicely explained. I have never seen such a good explanation. Love U ♥♥♥
Thank you! :)
Hey Josh, I'm back to study Machine Learning for Final Exams 😂😂😂
Good luck and let me know how they go! :)
Hi Josh, love the videos. Just one question. In the video of decision tree, when checking what feature should be at the root node, we just classified, then calculated the gini impurity(gini index, from this video as I am aware), but in 5:52 , by classifying the samples by the condition 'Has chest Pain', if the sample has chest pain, the 'correct' means that you have disease, and if you don't have chest pain, the 'correct' means that you don't have heart disease? but why do we set the conditions of 'correct' differently here?
Did I miss anything?
In the stump that uses "chest pain" to classify people, people with chest pain go to the left and people who do not have chest pain go to the right. Now, if most of the people that go to the left have heart disease, than that node will classify everyone who goes to the left as someone who has heart disease. This means that it will correctly classify most of the people who go to the left, but it will not correctly classify everyone. Does that make sense?
It'd be really refreshing to hear an actual model make dee doo dee doo boop' sounds while training.
:)
This is very informative for me. I skipped the Decision Tree but... but... I can understand it! Love your vids!
Thanks!
tripple baｍ
Thanks! :)
Thanks Josh! I have 3 questions:
1. @3:43 you say weak learners are "almost always stumps" is there a case where it is not a stump?
1.a. also what is adavtage of using stump over bigger trees?
2. Does boosting algorithm only use decision trees?
1) I don't know. However, it's possible and easily done (i.e. in sklearn you just change the default value for "depth").
2) It probably depends on your data.
3) In theory, no. In practice, yes.
I'll make a special dedication to this man on the day of my graduation.
bam! :)
thank you for making these videos, they are really helping me with my ML class!
Hooray! And good luck with your class.
The best! The simplest! The most informative!
Thank you! :)
Your channel has saved me a looooooot of time. Thanks!
Glad to hear it!
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
Awesome! Gradient Boosting should be available soon.
Thanks, that will be very helpful!
@@atinsingh164 I'm working on it right now.
I just want to say, THANK YOU. You video really helped me to understand the equations.
BAM! :)
Understood clearly, it took 2 hours to complete this, but now i know start to finish
Hooray!
One thing I am not really sure about is weather when creating stump 3, stump 1 and 2 should be considered or just stump 2. I watched a video on Boosting where they mentioned that the previous stumps are combined in order to create the new one. The combined wrong classifications are then used to sample the new set which is used to create the new stump.
Could someone please clearify
In this case, when we create a new stump, the effects of all previous stumps are felt by the weights each sample is given.
@@statquest I understand but when would we prefer one over the other? (Considering all previous weak learners over just the previous stump).
Thank you for taking the time to reply.
@@anelm.5127 Unfortunately I don't understand your question because in adaboost, there is no choice to be made. The weights assigned to each sample a shaped by every single stump we make, never just the last one.
Hello Sir,
I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc.
Thanks!
Thank you so much! I'm working on Gradient Descent right now. I hope it is ready in the next week or two.
Hello Josh, Thank you for the amazing videos. I had a couple of questions on stumps that are created after Sample weights are updated at time 16:00. We continue with sampling from the full set assuming new weights. This means the new set will be a subset of the original dataset as you explained in 18:32 . Going forward all subsequent stumps will only work on smaller and smaller subsets, making it a bit confusing for me on how to ensure good randomness.
1. Do we also restart from the updated sample weights at 16:00 and redo sampling, thereby creating multiple different datasets but using the same sample weight values? This will probably ensure we use all the data from the original data set in some instances.
2. As a follow-up to Q1, do we perform multiple sampling at different levels to get more variations in the dataset before creating stumps? In the video, you described only one instance of sampling using new weights but I assume it needs to be performed multiple times to get variations in datasets.
3. Do we not sample from distribution at the very beginning also? All sample weights will have the same value but random sampling would mean some might not make it to the first round of ( stump creation + Amount of say + New sample weights ) itself to get unequal sample weights at 16:00. In the video you take all the samples by default hence the query.
You just add the excluded samples back to the pool with their old weights and continue from there.
Josh and Kahn, the greatest teachers of all time. They should have their own University.
bam! :)
18:07 do we use bootstrapping in Adaboost?
so if we're sampling with replacement there's no gaurantee that the sample with the larger weight is duplicated more
no, we don't use bootstrapping since we use the weights to bias how samples are selected.
Excellent tutorial, thanks! Would like to ask:
1) after recalculating and updating weights for each sample, the process repeats. And in first step we found the 'weight' has the lowest Gini, then in next round we exclude the 'weight' and only consider the remaining criteria? or not? I see the 'weight' is reused when bootstrapping is used, but how about if we still using Gini?
2) if same set of criteria is re-used, then we would get a bunch of stumps with same criteria but differ only with the amount of say. Why is that useful in this case, other than simply using each criteria once only?
3) if the same set of criteria is re-used, then how can we determine when to stop the stump building process and settle down to calculate which classification result group has larger amount of say?
Thx~
Every stump selects from the exact same features (in this case, the features are "chest pain", "blocked arteries" and "patient weight"), however, the sample weights are always changing and this results in bootstrap datasets that contain different samples for each stump to divide. That said, while "Patient weight" might work well on the first stump, it might not work well in the second stump. This is because every sample that "Patient weight" misclassified will have a larger weight and thus, a larger probability of being included in the next bootstrapped dataset.
@@statquest this is a great explanation. Thank you very much!
The Phoebe of Machine Learning! Excellent videos! Thanks!
I have a song called "Smelly Stat" that always goes over well at coffeeshops. ;)
Hahaha!You’re the coolest! U must include that one in any of your future videos!
OMG! I'm completely delighted with your with your didactics! Awesome! Your students are very privileged! Gosh! Regards from Brazil fan :D
Muito obrigado!!!! BAM! :)
@@statquest BAM!!! :D
Many thanks for explaining the Adaboost in such a easy manner. I have one doubt at 19:39 minutes, where you have calculated the amount of say for "Does not have heart diseases". How you are calculating 0.41 and 0.82 values. Kindly explain.
I used the exact same method as demonstrated at 6:27. However, in order to save time, I did not show every single step of the process like I did the first time. However, you can imagine that that is what I did.
Please never stop making videos
Thank you! :)
i love your intros and outros, are simply awesome!!! i really enjoy learning with your channel!!!!
Thanks! :)