ROC Curves and Area Under the Curve (AUC) Explained
ฝัง
- เผยแพร่เมื่อ 30 ก.ย. 2024
- An ROC curve is the most commonly used way to visualize the performance of a binary classifier, and AUC is (arguably) the best way to summarize its performance in a single number. As such, gaining a deep understanding of ROC curves and AUC is beneficial for data scientists, machine learning practitioners, and medical researchers (among others).
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
RESOURCES:
- Transcript and screenshots: www.dataschool...
- Visualization: www.navan.name/...
- Research paper: people.inf.elte...
LET'S CONNECT!
- Newsletter: www.dataschool...
- Twitter: / justmarkham
- Facebook: / datascienceschool
- LinkedIn: / justmarkham
excellent explanation, the best that I have seen so far.
Thank you!
Indeed, agreed 100% with ed lee, definitely the best I explanation I have seen, much appreciated
You're very welcome!
I was about to type the same comment! Amazing explanation! Thank you for your contribution!
100% agree!!! thanks for the video
Likely the best explanation I've seen on ROC & AUC curves. Succinct yet thorough. The visualizations were extremely helpful. Nicely done.
Thank you so much for your kind and thoughtful comment! 🙏
undoubtedly one of the best explanation of ROC curve!!
Thanks very much! :)
I need to watch this a few more times to understand how it applies to my use-case, but this is a great overall explanation. Thank you for this!
You're welcome!
Thank you so much for this video. Your logical, cumulative explanation and clear visuals have made the rationale for using ROC curves and AUC far easier to understand. I'll be subscribing to your channel immediately!
Wow, thanks for your very kind comment, and for subscribing! Glad the video was helpful to you :)
A crisp and clear explanation, Thank you very much.
You're welcome!
Excellent! I am addicted to watching your vids. Thank you for the amazing work! Could you make some vids on using Tensorflow please? Cheers!
Thanks for your suggestion, and for your kind words! 👍
amazing video, thank you so much!
You're very welcome!
Awesome video!!! Covered some important points which are not covered by other videos on AUC-ROC.
Thank you!
Excellent work! Thanks very much Kevin, your video explaining ROC and AUC is the most intuitive one I have ever seen. Before watching this, it was still a little confusing for me , now I have a clear understanding of ROC and AUC.
Great to hear! :)
Thank you so much. Truly. You are so appreciated.
🙏
Great explanation! Thks for your video
You're welcome!
Usually, whenever I need goog understanding on any data science concept/statics releated topics, I search in this channel.
Thanks!
Thank you. I now understand, finally.
You’re welcome!
finally understood ROC. thank you
You're welcome!
crystal clear explanation! Thank you
You're welcome!
Excellent explanation of RoC. However I am still struggling to understand what AoC actually means. It looks like it stands for: If you randomly choose a red point, and randomly choose a blue point, then AoC is the possibility that red point is ranked ahead of blue point. Is it correct?
Exactly!
Is it a good metric for imbalance data set?
Yes, definitely!
even if some years are passed by, one of the best explanation around
Thanks very much for your kind words!
"If you can't explain it simply, you don't understand it well enough". YOU KILLED IT BRO! VERY WELL EXPLAINED!
Thanks so much! I'm so glad the video was helpful to you!
yes very good explanation indeed, I understood everything even though I'm high as a kite 😂😂😂
What the heck?! That was an awesome video, beautifully put!
Ha! Thank you :)
Besides this paper of Tom Fawcett is one of my favorite @@dataschool .
hot dawg, man, why do people in this field overcomplicate things so damn much. here is an accessible, free, easy to understand without sacrificing the innate complexity of the method message. Thank you, Data School crew!
Thanks very much for your kind words! Glad to hear the videos have been helpful to you!
Nice job. Very well explained!
Thanks!
Does plotting ROC curve take much time and computational power while calculating the FPR and TPR ?
No, I don't think so.
Just to confirm. At 7:09, the 235 and 125 used as numerators were an estimate. If not, how to you generate those values?
+Karim Nasser That's correct, those were estimates only.
what does this sentence mean "ROC and AUC curves would be identical as long as the ordering of observations by predicted probability remained the same" 11:19
if you scaled your predicted outputs to between 0&100 or 50&70 rather than 0&1 the ROC and AUC curves would behave identically as long as the order of your outputs remains the same. in fact the values of your output are irrelevant, only the order of your outputs matter for ROC and AUC.
@@harrycarey6793 Thanks Harry. And what you mean by "scaling"? Why do I need to scale?
@@zeynabmousavi1736 Scale refers to the magnitude of your values. for example a set of values between 0-1 can be scaled to be between 1-10 if the relationship between every value remains the same. here we would see the set "0, 0.3, 0.7, 1" become "0, 3, 7, 10". every value is multiplied by the same constant (10) so the relationship between all values remains the same relatively. If these values were outputs from some binary classifier, trying to predict whether or not an image contains a cat, then you would need to pick some arbitrary threshold at which to decide whether or not there is indeed a cat in each image. If you were to apply an ROC curve and measure the AUC to both these classifiers (0-1 and 0-10) they would be identical! because ROC and AUC does not care about the scale of the outputs(how big the outputs are), it only cares about the order of the predictions.
Let me know if this makes sense :)
EXCELLENT answers, Harry! Thank you!
Very good explanation!
Thank you!
Thanks for this explanation. Easy to comprehend. Best I have seen
Excellent! Thanks so much for your kind comment!
Very good and precise explanation. Thanks for creating the video.
You're very welcome!
Great explanation, thank you so much!
Good speed, clear language and nice visualization.
Subscribed.
Thank you!
is there any theoretical formula to find the ruc auc score instead of using the code >> roc_auc_score(y_test, lr_y_predict)?
YOU ARE A LIFESAVER!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Glad I could help!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! :)
Dear instructor, how are you? I am doing a diagnostic accuracy study and proposed to use ROC , would send a document related to this pleases? I need to understand this thing
I have never seen an explanation of ROC-AUC better than this...thank you so much
Thank you so much! 🙏
What an amazing way to explain these concepts
Very helpful, only that the coordinate of FPR is not clear on the figure.
Very good explanation. The best one I have seen so far.
Thanks so much for your kind words!
Great content! Thank you.
You're welcome!
A very detailed and comprehensive explanation. Thank you.
You're welcome!
The auc metric matters most in the ordering of the probabilities rather than the value.
This is the best mental model to explain ROC/AUC I seen so far, thanks a lot!
You're welcome!
What about performance of multi class classifier?
This might be of help: scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics
Thanks!
Awesome video
Thanks!
Best explanation I have seen so far .. Amazing .. pls keep going
Thank you!
Absolutely amazing and intuitive explanation. Thanks a lot
Glad you liked it!
I was searching in the net for videos and reading articles everywhere, but this is "The Best" Explanation I've found. Great work!
Thank you!
great visualisation and explanation, made everything so much easier to understand
Awesome! Glad it was helpful to you!
ROC curve the data need to in increasing or decreasing order for positive
Many thanks for this excellent video. You have a great gift for lucidly explaining complex concepts
Thank you so much! 🙌
great video! Thank you
You're very welcome!
Excellent job, Dataschool, upvoted. But how do you plot the curve _for all_ thresholds? Do you use, e.g., that the curve is concave, above the line y=x, etc., to extend from a few values to the whole curve? Also, is there a way of having an explicit formula for the ROC curve, e.g., f(x)=x^3+x-1 (made up)? I mean, this is not even a function in the strict sense since for one input(thereshold) you get two outputs.
ROC is the curve for all possible thresholds! No, there is no way to create a formula for an ROC curve.
THIS IS THE BEST EXPLANATION OF ROC-AUC ON TH-cam!
Thank you! 🙏
Schrödinger (a large company develops software for drug discovery) linked this to one of their course material's additional resources so I went and watched this. No wonder they had this video there, as the subject was really well explained on the video
That's great to hear, thanks for your kind comment!
I like your video! Thank you:)
You're welcome!
Great explanation sir. I have one question about how I plot the ROC Curve using Matlab for finding the accuracy of images. Please help
Not sure, sorry!
How can we get ROc curve in a multilabel setting
gr8 explanation.Just a quick doubt what is the count of observation vs predicted probability graph called?
That type of graph is called a histogram.
I am still not clear with the curve generation part. To generate curve, we need the value of threshold and probability will be calculated with the help of classifier. As I know, scikit-learn implementation doesn't take any such arguement, it only takes (y_score, y_true). Then how it return area under ROC?
This video will answer your question: th-cam.com/video/85dtiMz9tSo/w-d-xo.html
Instead of two distributions of "negative" and "positive", could those distributions be "null" and "observed"? This is for those cases where a binary label is unknown. Thanks!
I'd have to understand the context better in order to answer that question - feel free to explain further if you like!
Thanks for the great video. Could you please elaborate on the comment made at 11:31, namely that AUC gives the probability that a classifier will rank a randomly chosen positive observation higher than a randomly chosen negative observation. Doesn't how a classifier classifies depend on the threshold while the AUC is a way of summarizing how the classifier performs across different values of threshold?
It's hard to explain briefly, but the key is that I'm talking about ranking, and rank has nothing to do with the chosen value of the threshold. Does that help?
The part I don't understand is around the 5:56 mark. Where are you getting the 50 from @ the .8 rate? Is there a mathematical formula you used to predict that or did you just pick 50 at random? Will someone explain this a little more in-depth? I think I understand it but not sure of the reasoning behind it. Also how did 235 come about? Thank you.
There's no formula, I'm just estimating. Hope that helps!
What I don't get is where you are getting the numbers like 125, 250 etc. I don't see any labeling on the graphs and thus i'm confused :(
+odetoazam At one point, he states that each graphs has a total of 250 pixels.
+odetoazam Sorry it was confusing! Here is what I said:
We'll pretend that every blue and red pixel represents a paper for which you want to predict the admission status. This is your validation (or "hold-out") set, so you know the true admission status of each paper. The 250 red pixels are the papers that were actually admitted, and the 250 blue pixels are the papers that were not admitted.
Great explanation, and wow sound exactly like Jim Rohn, one of my favorite motivational speaker.
Thanks!
Amazing Video.
Thanks!
Could you please explain how to calculate values of 50,235 from the graphs.
Assuming there were 250 pixels under the red curve, 50 is my estimate for the number of red pixels to the right of the line when the threshold is set at 0.8, and 235 is my estimate for the number of red pixels to the right of the line when the threshold is set at 0.5. In other words, these numbers are all just estimates.
Good explanation.
A question: i have a prob with multi class classification (about 100 classes) (the shape of dataset is 1300×15)
What's the best way to model this problem? Do I use just the classics algorithm for classification like Random Forest, XGBoost, ... or does exists other algorithms for this problem?
Thanks in advance
amazing explanation the amount of information you fit into 14 minutes is magical.
Wow! Thank you so much for your kind words! :)
Is it any method to Decide Threshold or it will be only upto the decision maker.If it upto decision maker then model will differ from person to person.Kindly guide
+Elec Engg The "right" threshold depends on your priorities: Are you more interested in penalizing false positives or false negatives? So, you are correct that different people may choose different thresholds.
Can we say classifier is bad because of the smaller AUC? or it`s because the validation set is bad? How do we have a validation set of such an overlap?
Whether or not AUC is the appropriate evaluation metric depends on the objective of your model. This page might help you to decide: github.com/justmarkham/DAT8/blob/master/other/model_evaluation_comparison.md
Hope that helps!
Thank you Kevin. Great video.
You're very welcome! Glad you liked it.
thanks. very clear explanation.
You're welcome!
Thanks for this wonderful tutorial
You're very welcome!
Nice way to explain ROC. Thanks very much :)
+Andika Yudha Utomo You're very welcome!
Amazing explanation, much appreciated!
You're very welcome!
Hey, is there a relationship between the area of the histograms and area under the curves in the ROC chart?
No (if I'm understanding your question correctly).
awesome tutorial , thank you
Awesome explanation ever seen!
Thanks very much! :)
very good explanation about roc, thank you very much.
You're very welcome!
Will it be wrong if we adapt one vs one approach for multi class classification?
Sure, you can use a one vs one approach.
data school added to subscribed channels. This should be enough of a comment for this video.
Thanks for subscribing!
Very nice explanation and visualization. Thank you.
You're welcome!
sir can share the code or concept of the Visualization.......is it shiny?
I don't have the code, I'm sorry!
Excellent explanation!! Very helpful, thank you!
You are very welcome!
Definitely you should have used red color for negatives instead of positives
Nuh uh, being Positive for a disease is dangerous and therefore is red
(I know it's been 3years since you commented this
can u explain fraudulent part (+ve class being fraud)?.
Sorry, I'm not sure I understand your question?
Very nice explanation of the ROC curve !!
Thank you!
Awesome. I finally understood this.
Great to hear!
What program was used to create this visualization?
I don't know, I'm sorry!
Amazing Video, Thank you very much
+Alireza Khamesipour You're welcome!
thank you for great visual explanation.
You're welcome!
A heck of a job! Thanks. Well explained!
Thanks for your kind comment!
Thanks for the wonderful explanation. I could not have been more simple, yet correct.
You're welcome!
better than my professor's explain, awesome
Ha! Thanks :)
very nice and detailed explanation
Thanks!
Thank you kind sir. U come to aid during dark times.
You're very welcome! :)
Thanks a lot for the video. One query I have, Is it possible to plot the ROC for Continuous Datasets, also?
ROC curves can only be used for classification problems, meaning ones in which the target value is categorical. However, it doesn't matter whether the training data is made of categorical or continuous data. Hope that helps!
Data School Thanks ... this helps ... I have also come across few papers on VUC... there was one who have written about a 3D ROC Curve...
Great Presentation. What tool you are using for the presentation?
+Laeeq Ahmed I used Camtasia Recorder for the screen capture, and did all of the editing in Camtasia Studio.
What do you mean by a classifier.?
That means a model which is trying to predict a class label. This video might be helpful to you: th-cam.com/video/elojMnjn4kk/w-d-xo.html
Thanks a lot - you make my neurons spike again - :)
Ha! Great to hear :)
Thank you
It was very helpful
Great!