Hello, thanks for your video! I'm writing my thesis at the moment and confused about which performance metric to used. We're comparing binary outcomes of AI algorithm vs. radiologists in diagnosing ICH in CT scans. Looking into other studies, i've noticed the majority used AUC / accuracy as their primary end point. What's your advice on what performance metric to use?
Hi! Thank for reaching out! I believe that in that context Accuracy gives you an indication whether you have less of both "false positives" and "false negatives", but remember that it will be catastrophic with unbalanced distributions (which I believe could be the case often with diseases). In this case, I would also report the Matthews Correlation Coefficient, which is much more robust. Although it may not be common, you are protecting yourself and your peers from thinking your model is better than it is. Now about AUC, it is good to check it in order to be sure that your model is far from a random model (ROC curve far from the diagonal). Please, don't forget to watch the video I published about ROC and AUC. But... I would, for sure, also include a Precision vs Recall curve and Average Precision (AP), and there is a video for that too showing how the AP is more robust than the AUC.
Thank you very much for explaining the Matthews Correlation Coefficient. I am just a bit confused: If an actual value is negative but was predicted positive: it actually is a False Positive (because *falsely* predicted/classed as *positive* ). And if the actual value is positive but was predicted as negative than its a false negative (because *falsely* predicted/classed as *negative* ). Therefore your confusion matrix is in my opinion not correctly labelled.
@KleineSwiss You are right, the off-diagonal terms are switched. Luckily, I don't think it invalidates any of the discussions in any of the videos about performance metrics. Thank you for catching this typo!
Oi Gibran! Continue se empenhando em aprender que uma hora algo aparece! Siga o canal Téo Me Why para seguir tutoriais interessantes, principalmente de SQL: th-cam.com/channels/-Xa9J9-B4jBOoBNIHkMMKA.html
In appreciation, I would recommend your video on the subject be seen by other students studying ML course in University and Phd students. Keep it up..
Thanks Felipe, great video, greetings from Oaxaca, Mexico.
Hello, thanks for your video! I'm writing my thesis at the moment and confused about which performance metric to used. We're comparing binary outcomes of AI algorithm vs. radiologists in diagnosing ICH in CT scans. Looking into other studies, i've noticed the majority used AUC / accuracy as their primary end point. What's your advice on what performance metric to use?
Hi! Thank for reaching out! I believe that in that context Accuracy gives you an indication whether you have less of both "false positives" and "false negatives", but remember that it will be catastrophic with unbalanced distributions (which I believe could be the case often with diseases). In this case, I would also report the Matthews Correlation Coefficient, which is much more robust. Although it may not be common, you are protecting yourself and your peers from thinking your model is better than it is. Now about AUC, it is good to check it in order to be sure that your model is far from a random model (ROC curve far from the diagonal). Please, don't forget to watch the video I published about ROC and AUC. But... I would, for sure, also include a Precision vs Recall curve and Average Precision (AP), and there is a video for that too showing how the AP is more robust than the AUC.
keep up the great work!
Very good, thanks for this!
Glad you liked it!
Thank you very much for explaining the Matthews Correlation Coefficient. I am just a bit confused: If an actual value is negative but was predicted positive: it actually is a False Positive (because *falsely* predicted/classed as *positive* ). And if the actual value is positive but was predicted as negative than its a false negative (because *falsely* predicted/classed as *negative* ). Therefore your confusion matrix is in my opinion not correctly labelled.
@KleineSwiss You are right, the off-diagonal terms are switched. Luckily, I don't think it invalidates any of the discussions in any of the videos about performance metrics. Thank you for catching this typo!
Great tutorial, thanks. How did you print your the threshold vs Matthews Coef, f1, etc graph? Have you got a tutorial on this or a code example?
Lol that joke is of BD :3 .Great video though
❤
Me da um emprego?
Oi Gibran! Continue se empenhando em aprender que uma hora algo aparece! Siga o canal Téo Me Why para seguir tutoriais interessantes, principalmente de SQL: th-cam.com/channels/-Xa9J9-B4jBOoBNIHkMMKA.html