Great vid. Want to add few more points: Platt Scaling is used when the Calibration Plot looks like a sigmoid curve. Platt Scaling uses a modified Sigmoid function and solves an optimization problem to get A and B. Isotonic Regression is used when the calibration plot does not look like a sigmoid curve. Isotonic Regression breaks the curve into multiple linear models. Thus it needs more points than Platt Scaling.
The combination of your status as a working Data Scientist and your decision to dive into the technical details of your chosen topics makes your videos match the definition of what I see as the ideal content for AI youtubers .. thanks!
I watched several videos with explanation of Model Calibration and this is the best one. You've shown the practice not the theory and this is exactly what most Data Scientists need. Thank you and that TH-cam for recommending this video. I subscribed your channel immediately.
Thank you so much for making such great videos. It really helps for someone new to DS to quickly understand all the concepts. Appreciate explaining with actual codes and going through each step!
I wanted to ask you, how do you go about educating yourself on ML? Do you read papers regulary or are you researching specifically the topics your need for your work? I find it difficult to find projects to learn from that aren't so constructed that they don't teach much anymore. Thanks for your video, I liked the part were you showed your code, it would be really great, if you can comment your juypter notebook and put it in the description, if you don't mind, I would like to test and learn with the code myself.
I'll tell you how I came across this topic. I was solving a basic classification problem on Kaggle. I wanted to make my predictions as accurate as possible to avoid opportunity costs. Also my data set was imbalanced. Hence, I was looking on various ways to deal with imbalanced data and adjusting thresholds and finally came across model calibration. So, I guess the key is to expose yourself to various problems and make your model work better.
If you didn't put in the class_weight parameter into your LR for the imbalanced dataset, the scores would still be pretty much in a "straight" line. What created your skewed calibration curve is the class_weight parameter. You only care about the class_weight parameter if you want to use a 0.5 as a threshold to label your predictions as a 1 or 0. If you're only looking at probabilities, you should NOT touch the class_weight parameter.
Hello sir, thank you for your excellent explanation. I have inquiry regarding calibration model for transition probabilities. However I don't understand how it works and I would like to learn and know more about it. I would be really appreciate, if you can assist me on this matter. Thank you.
I think the effect of the calibration in 2.2 is clearer when looking at the mean: it's now 10% which corresponds to the 0.9 ratio you used to define the unbalanced dataset (it was 23% on the uncalibrated model). Or I'm missing something? Anyway, thank you for this video, I wasn't aware of this calibration concept and it's quite clear now!
Yeah. You could look at the mean. But we are typically dealing with long tailed distributions (lots of samples have small values and a few samples have large values). This upwards skews the mean. Hence i talk in medians instead. But either way, you can see the effects of the "reduction" in probability values
I believe the strong use case would be in the medical industry where you need to "support" for the model's estimated probability say for having cancer as an example. I am not sure calibration is always required even for imbalanced dataset.
Calibration is also required for credit risk model. In this case the actual event rate is used to calculate the unit economics and that's why it might be important that predicted and actual probabilities are close to each other.
I feel like I run into this issue all the time, but what do you even mean when you say the "true" probability in a situation like classification of something as fraud or not fraud? The "true" probability of any specific example is either 100% fraud or 100% not fraud, as something's identity as something fraudulent isn't an inherently probabalistic thing. This is in contrast to something like, say, whether the outcome a given plate appearance in baseball will be a home run or not. You could repeat the identical same plate appearance many many times and only some of them would result in a home run, most would not. That, at least, is what comes to my mind when we talk about predicting the "true" probability of unbalanced data events, that could be calibrated to. What am I missing?
1. At 0:46 you mentioned probabilities may be skewed higher as a result of balancing, any resources explaining why this happens? (this video didn't demonstrate balancing) 2. At 4:24, why does half below 47% and half above 47% make sense, why mention "that's correct"? Is there an ideal y_pred.describe() output you have in mind for any problem that comes? What would be an incorrect .describe output? (At this moment we have no knowledge of calibration plots yet, just interpreting based on .describe output) 3. Is it fit calibration on valid and plot reliability curves on test, or vice-versa? (th-cam.com/video/AunotauS5yI/w-d-xo.html did it the opposite way from your video, he used test set to train calibration layer and plotted curves on validation.)
When you do undersampling/oversampling, the "probabilities" from a binary classifier are inflated to represent the minority class. I.e. they don't represent true probabilities
Well done, but calibration is completely useless. In Case 2, turn off class_weight='balanced' and you wont need to calibrate anything. The only time you need to calibrate a model is when you play with the class weights or up-sample the minority class, both of which are things you should NEVER do in any circumstance.
Great vid. Want to add few more points: Platt Scaling is used when the Calibration Plot looks like a sigmoid curve. Platt Scaling uses a modified Sigmoid function and solves an optimization problem to get A and B. Isotonic Regression is used when the calibration plot does not look like a sigmoid curve. Isotonic Regression breaks the curve into multiple linear models. Thus it needs more points than Platt Scaling.
The combination of your status as a working Data Scientist and your decision to dive into the technical details of your chosen topics makes your videos match the definition of what I see as the ideal content for AI youtubers .. thanks!
Great explaination. Good to learn something new everyday.
I watched several videos with explanation of Model Calibration and this is the best one. You've shown the practice not the theory and this is exactly what most Data Scientists need.
Thank you and that TH-cam for recommending this video. I subscribed your channel immediately.
Thanks a lot! In more recent videos, I am doing a lot more math. But I’ll pivot back to programming too soon enough :)
Thank you so much for making such great videos. It really helps for someone new to DS to quickly understand all the concepts. Appreciate explaining with actual codes and going through each step!
I wanted to ask you, how do you go about educating yourself on ML? Do you read papers regulary or are you researching specifically the topics your need for your work? I find it difficult to find projects to learn from that aren't so constructed that they don't teach much anymore. Thanks for your video, I liked the part were you showed your code, it would be really great, if you can comment your juypter notebook and put it in the description, if you don't mind, I would like to test and learn with the code myself.
I'll tell you how I came across this topic. I was solving a basic classification problem on Kaggle. I wanted to make my predictions as accurate as possible to avoid opportunity costs. Also my data set was imbalanced. Hence, I was looking on various ways to deal with imbalanced data and adjusting thresholds and finally came across model calibration. So, I guess the key is to expose yourself to various problems and make your model work better.
Well explained buddy. Short and to the point
Thanks!
If you didn't put in the class_weight parameter into your LR for the imbalanced dataset, the scores would still be pretty much in a "straight" line. What created your skewed calibration curve is the class_weight parameter. You only care about the class_weight parameter if you want to use a 0.5 as a threshold to label your predictions as a 1 or 0. If you're only looking at probabilities, you should NOT touch the class_weight parameter.
Absolutely spot on! Completely agree.
Great explanation!❤️
Doubt : how do we calibrate probabilities in case of multi-class classification using OVO or the OVA SVMs?
The motivation was really well explained
Thanks 😊
Hello sir, thank you for your excellent explanation. I have inquiry regarding calibration model for transition probabilities. However I don't understand how it works and I would like to learn and know more about it. I would be really appreciate, if you can assist me on this matter. Thank you.
I think the effect of the calibration in 2.2 is clearer when looking at the mean: it's now 10% which corresponds to the 0.9 ratio you used to define the unbalanced dataset (it was 23% on the uncalibrated model). Or I'm missing something?
Anyway, thank you for this video, I wasn't aware of this calibration concept and it's quite clear now!
Yeah. You could look at the mean. But we are typically dealing with long tailed distributions (lots of samples have small values and a few samples have large values). This upwards skews the mean. Hence i talk in medians instead. But either way, you can see the effects of the "reduction" in probability values
@@CodeEmporium I understand, thanks for your reply!
I believe the strong use case would be in the medical industry where you need to "support" for the model's estimated probability say for having cancer as an example. I am not sure calibration is always required even for imbalanced dataset.
It's not required
Calibration is also required for credit risk model. In this case the actual event rate is used to calculate the unit economics and that's why it might be important that predicted and actual probabilities are close to each other.
great man, explained this very easily
Thank you!
I feel like I run into this issue all the time, but what do you even mean when you say the "true" probability in a situation like classification of something as fraud or not fraud? The "true" probability of any specific example is either 100% fraud or 100% not fraud, as something's identity as something fraudulent isn't an inherently probabalistic thing. This is in contrast to something like, say, whether the outcome a given plate appearance in baseball will be a home run or not. You could repeat the identical same plate appearance many many times and only some of them would result in a home run, most would not. That, at least, is what comes to my mind when we talk about predicting the "true" probability of unbalanced data events, that could be calibrated to. What am I missing?
1. At 0:46 you mentioned probabilities may be skewed higher as a result of balancing, any resources explaining why this happens? (this video didn't demonstrate balancing)
2. At 4:24, why does half below 47% and half above 47% make sense, why mention "that's correct"? Is there an ideal y_pred.describe() output you have in mind for any problem that comes? What would be an incorrect .describe output? (At this moment we have no knowledge of calibration plots yet, just interpreting based on .describe output)
3. Is it fit calibration on valid and plot reliability curves on test, or vice-versa? (th-cam.com/video/AunotauS5yI/w-d-xo.html did it the opposite way from your video, he used test set to train calibration layer and plotted curves on validation.)
Can you Show that in Stata? thanks
🔴nice
Great channel.
Thanks a lot
What do you mean by more represent probabilities @13:50?
When you do undersampling/oversampling, the "probabilities" from a binary classifier are inflated to represent the minority class. I.e. they don't represent true probabilities
@@CodeEmporium Aha! Thanks a lot man. really appreciate it and love your content. keep up the great job:)
are you related to marques browlee the tech guy
I am not. But that's a compliment :)
Well done, but calibration is completely useless. In Case 2, turn off class_weight='balanced' and you wont need to calibrate anything. The only time you need to calibrate a model is when you play with the class weights or up-sample the minority class, both of which are things you should NEVER do in any circumstance.