In your github repo, you use error rate to compute the relative overfitting rate R, and then calculate the weight W. However, in your slide you use accuracy score instead of error rate. I think it's more making sense if you use accuracy score to calculate the R, since the scoring function may not be accuracy score on default. What if it's f1 score instead, then the error rate won't be (1 - f1) any more. Correct me if I was wrong.
That's a good. I am just double-checking my code (github.com/rasbt/mlxtend/blob/master/mlxtend/evaluate/bootstrap_point632.py#L237) and it looks like I compute it as train_err = 1 - scoring_func(...) so it should be working with either accuracy, F1, and others where "larger is better"
@@SebastianRaschka Thanks for the quick reply! My question is, for example, we use f1 score as the scoring function, is the training error still be computed by 1 minus f1_score? It is presented in a different way as in your slide, so I’m very confused… Instead of using error rate, you use scoring functions value to calculate the R.
@@yangwang9688 Yeah, if you use the F1 score, then the "error" will still be "1-F1". That's a good point re slides. I used accuracy in the slide because the previous lectures were all based on accuracy. I think I implemented the code years before that using "error" which is why it looks a bit different. But it should be equivalent via "accuracy = 1 - error"
@@yangwang9688 Yeah. By default it is based on the accuracy. But if you want, you can override the default setting, for example, using F1. If you do that, the error becomes 1-F1 instead of 1-Acc
15:32 Hello, I don't think I will get an answer anytime soon. But I've seen in my course and on a stack website that the ACC_h,i is replaced by ACC_LS (on the whole learning set, and not on the left ones), they use Err instead of ACC but that's not important. I don't understand why you would use only a subset. Is the "official one" on LS wrong?
Hi, I still have one quick question. In the slide, your no-information rate (gamma) is computed based on 0-1 loss, and yet in your github repo you compute gamma based on scoring (such as accuracy or f1 score). I was just wondering, which one is correct?
These are great questions! Should have been more clear. Here, with 0/1 loss I mean classification error (the classification error is basically the averaged 0/1 loss). You can also compute it with the accuracy as it is 1-error (and so it's basically like 1 minus the averaged 0/1 loss)
@@SebastianRaschka So I suppose using scoring function instead of losses to compute gamma is correct? It appears to me that using scoring functions makes more sense since the NIR should be “the largest proportion of the observed classes.” Therefore, the denominator of R will then become (1 - gamma - train_error) or (train_score - gamma).
@@yangwang9688 Correct! So in the slides I am going over the original use case but in the code (on mlxtend) it's more of a general implementation where accuracy is the default (because it's the default in scikit-learn) and users can swap it with any scoring function they like (for example, precision, recall, F1, ROC AUC, etc.)
There is also one mistake I guess. When computing the gamma in the slide, it's not 1 over n cube there is 3 classes. At least in your code repo you use itertools.product() to compute all the combinations, it is still 1 over n squared if it's multi-class classification problem.
In your github repo, you use error rate to compute the relative overfitting rate R, and then calculate the weight W. However, in your slide you use accuracy score instead of error rate. I think it's more making sense if you use accuracy score to calculate the R, since the scoring function may not be accuracy score on default. What if it's f1 score instead, then the error rate won't be (1 - f1) any more. Correct me if I was wrong.
That's a good. I am just double-checking my code (github.com/rasbt/mlxtend/blob/master/mlxtend/evaluate/bootstrap_point632.py#L237) and it looks like I compute it as
train_err = 1 - scoring_func(...)
so it should be working with either accuracy, F1, and others where "larger is better"
@@SebastianRaschka Thanks for the quick reply! My question is, for example, we use f1 score as the scoring function, is the training error still be computed by 1 minus f1_score? It is presented in a different way as in your slide, so I’m very confused… Instead of using error rate, you use scoring functions value to calculate the R.
@@yangwang9688 Yeah, if you use the F1 score, then the "error" will still be "1-F1". That's a good point re slides. I used accuracy in the slide because the previous lectures were all based on accuracy. I think I implemented the code years before that using "error" which is why it looks a bit different. But it should be equivalent via "accuracy = 1 - error"
@@SebastianRaschka Just to make sure, the error can be “1-f1” and “1-acc”? And these two errors are different?
@@yangwang9688 Yeah. By default it is based on the accuracy. But if you want, you can override the default setting, for example, using F1. If you do that, the error becomes 1-F1 instead of 1-Acc
15:32 Hello, I don't think I will get an answer anytime soon. But I've seen in my course and on a stack website that the ACC_h,i is replaced by ACC_LS (on the whole learning set, and not on the left ones), they use Err instead of ACC but that's not important.
I don't understand why you would use only a subset. Is the "official one" on LS wrong?
Hi, I still have one quick question. In the slide, your no-information rate (gamma) is computed based on 0-1 loss, and yet in your github repo you compute gamma based on scoring (such as accuracy or f1 score). I was just wondering, which one is correct?
These are great questions! Should have been more clear. Here, with 0/1 loss I mean classification error (the classification error is basically the averaged 0/1 loss). You can also compute it with the accuracy as it is 1-error (and so it's basically like 1 minus the averaged 0/1 loss)
@@SebastianRaschka So I suppose using scoring function instead of losses to compute gamma is correct? It appears to me that using scoring functions makes more sense since the NIR should be “the largest proportion of the observed classes.” Therefore, the denominator of R will then become (1 - gamma - train_error) or (train_score - gamma).
@@yangwang9688 Correct! So in the slides I am going over the original use case but in the code (on mlxtend) it's more of a general implementation where accuracy is the default (because it's the default in scikit-learn) and users can swap it with any scoring function they like (for example, precision, recall, F1, ROC AUC, etc.)
5:24 can I use linear regression or some other method within this cross_val_score function?
yes, any sklearn-compatible estimator object will work.
There is also one mistake I guess. When computing the gamma in the slide, it's not 1 over n cube there is 3 classes. At least in your code repo you use itertools.product() to compute all the combinations, it is still 1 over n squared if it's multi-class classification problem.
Oh yes that's right. 1 n is for the n labels and 1 n is for the n feature vectors so it's always n^2. Not sure why I said sth different. Thanks!
@@SebastianRaschka Thanks for the clarification, super helpful video by the way 😊