I am not sure if this is a best way to deal with data imbalance and it won't work in a real case. You have used SMOTE to balance the dataset and used your test dataset from the oversampled data which is synthetic. To make sure your model is working well, you have to save part of the original imbalance dataset as your test dataset and then apply SMOTE on the rest. In this way your test dataset is a perfect representation of the original data. I am sure you f1-sccore will be very small. One of the best methods are One Class Support Vector Machine (OCSVM), Generalized One-class Discriminative Sub-spaces (GODS), One Class CNN (OCCNN) and Deep SVDD (DSVDD)
You do realize that in your pipeline, once you run the oversample step, you have 6 perfectly balanced groups with 900 samples in each group. There's no real majority class to sample from. When you then undersample from a perfectly balanced dataset, it appears to leave group intact and resamples the others. If you plot the data, it will look essentially the same as before, when you only oversampled, with some samples missing and other samples duplicated. The scores will be similar as well.
Hello Mr. Webb. Dude in the video does SMOTE on the whole dataset, which I think will cause data leakage generating overly-optimistic accuracy. However, he says that by using cross-validation, SMOTE on the whole dataset wouldn't be a problem. Do you concur?
@@thangtran145 Yes, anytime you do something to the entire dataset, you get data leakage. Anything, including SMOTE. Synthesizing data, then including some of that synthetic data in your test set will lead to overly optimistic results. The usual recommendation is to use a pipeline. SMOTE -> scale -> regression on the training set only. Leave the test set alone. Problem with unbalanced data is getting it distributed evenly. You might need to bucket the data manually. Also, you may not have enough of the minority class in your test set to tell if your model is a good predictor. You may want to score the two classes separately.
Hello thanks for the video. However I noticed that your did SMOTE before running the train test split. I am afraid that this might be causing the results to improve drastically since the the upsampled observations from the minority class might have entered the testing dataset. So basically your model learned and test on pretty much the same variable which caused the results to improve. Let me know what you think.
Hi Ammar, yes in a normal train/test split there is a risk that one of the synthetic samples generated by the original data points might be present in the test set. But since I generated synthetic data and used crossvalidation I decided to handle the imbalance on the whole set, it was faster for what I wanted to show. You can as well split the data first and then perform crossvalidation on the train set and test on the out of sample test set afterwards. Hope you found it useful.
I was confused at this point as well. I believe that altering the distribution of the test set is not a great practice, since in reality this is the distribution we expect if we deploy our model to production. The real question is how this model performs on the original test set. However, great video on SMOTE. Thank you!
@@DecisionForest So you're saying data leakage will not happen if we combine SMOTE of whole set with cross-validation? In other words, it's okay to SMOTE the entire dataset as long as we use cross-validation? Thank you
In order to truly evaluate you need to test on an IMBALANCED test set. :) you can train on a balanced train set but hold out needs to be on a true imbalanced set . Because in the real world the data you encounter will have the same imbalanced-ness and that’s what your performance metric needs to measure: how well you score on unseen imbalanced data.
Hi thanks for the content! I am confused that, instead of applying this method for the y variable, can I apply this technique for imbalanced predictors that have levels with large differences in sample size? For example, class A: 900, class B:100, class C: 2 Thanks!
I don't understand how you apply under and oversampling at the same time. One of them will balance the data, and the other one has nothing left to do...
For oversampling text data I'd suggest you use pre-trained word embeddings. For example you can generate new samples by doing permutations of similar words.
I think the pipeline just balancing the classes by oversampling automatically. and then because the class already balance, the undersampling didn't do anything
Oh yes, normally you either undersample of oversample as if we do both only the first is actually implemented. I did both sorry for the confusion, should’ve mentioned it.
@@DecisionForest it's okay I'm still learning too. If you want to do both, you must specify sampling strategy as parameter. otherwise they just autobalancing the classes.
I am not sure if this is a best way to deal with data imbalance and it won't work in a real case. You have used SMOTE to balance the dataset and used your test dataset from the oversampled data which is synthetic. To make sure your model is working well, you have to save part of the original imbalance dataset as your test dataset and then apply SMOTE on the rest. In this way your test dataset is a perfect representation of the original data. I am sure you f1-sccore will be very small. One of the best methods are One Class Support Vector Machine (OCSVM), Generalized One-class Discriminative Sub-spaces (GODS), One Class CNN (OCCNN) and Deep SVDD (DSVDD)
You do realize that in your pipeline, once you run the oversample step, you have 6 perfectly balanced groups with 900 samples in each group. There's no real majority class to sample from. When you then undersample from a perfectly balanced dataset, it appears to leave group intact and resamples the others. If you plot the data, it will look essentially the same as before, when you only oversampled, with some samples missing and other samples duplicated. The scores will be similar as well.
What do you recommend
@@brandonbosire211 Do one or the other, not both, one after the other.
Hello Mr. Webb. Dude in the video does SMOTE on the whole dataset, which I think will cause data leakage generating overly-optimistic accuracy. However, he says that by using cross-validation, SMOTE on the whole dataset wouldn't be a problem. Do you concur?
@@thangtran145 Yes, anytime you do something to the entire dataset, you get data leakage. Anything, including SMOTE. Synthesizing data, then including some of that synthetic data in your test set will lead to overly optimistic results. The usual recommendation is to use a pipeline. SMOTE -> scale -> regression on the training set only. Leave the test set alone. Problem with unbalanced data is getting it distributed evenly. You might need to bucket the data manually. Also, you may not have enough of the minority class in your test set to tell if your model is a good predictor. You may want to score the two classes separately.
Hello thanks for the video.
However I noticed that your did SMOTE before running the train test split. I am afraid that this might be causing the results to improve drastically since the the upsampled observations from the minority class might have entered the testing dataset. So basically your model learned and test on pretty much the same variable which caused the results to improve.
Let me know what you think.
Hi Ammar, yes in a normal train/test split there is a risk that one of the synthetic samples generated by the original data points might be present in the test set. But since I generated synthetic data and used crossvalidation I decided to handle the imbalance on the whole set, it was faster for what I wanted to show. You can as well split the data first and then perform crossvalidation on the train set and test on the out of sample test set afterwards. Hope you found it useful.
I was confused at this point as well. I believe that altering the distribution of the test set is not a great practice, since in reality this is the distribution we expect if we deploy our model to production. The real question is how this model performs on the original test set. However, great video on SMOTE. Thank you!
@@iliaskatsabalos spot on point. Hold out set validation is needed.
@@DecisionForest So you're saying data leakage will not happen if we combine SMOTE of whole set with cross-validation? In other words, it's okay to SMOTE the entire dataset as long as we use cross-validation? Thank you
Unfortunately, your website link and notebook link are not available here. Any suggestion?
Can you suggest any techniques to solve imbalanced image dataset??
Thank you..
In order to truly evaluate you need to test on an IMBALANCED test set. :) you can train on a balanced train set but hold out needs to be on a true imbalanced set . Because in the real world the data you encounter will have the same imbalanced-ness and that’s what your performance metric needs to measure: how well you score on unseen imbalanced data.
one of the best channels for ML!
Thank you Aneeta 🙏🏻
Hi thanks for the content!
I am confused that, instead of applying this method for the y variable, can I apply this technique for imbalanced predictors that have levels with large differences in sample size?
For example, class A: 900, class B:100, class C: 2
Thanks!
When should we use under_sampling? As I see there's a potential risk of losing information
I don't understand how you apply under and oversampling at the same time. One of them will balance the data, and the other one has nothing left to do...
Good observation, it was just to show how you apply both.
You said to deal with 'multi-class classification problems'. But what if we have imbalanced data and binary classification?
For binary it’s the same when it comes to imbalanced datasets. Just gave here the example of multiclass.
@@DecisionForest Yeah.
I thought so. Thanks!
It is not clear sir, and I have a question, what is the technique u have used in sorting the problem class imbalance?
I used SMOTE for oversampling and RandomUnderSampler for undersampling.
Excuse me, is there any way to find the original Notebook file? Can't open the one in the description. Thank you.
Just checked now, it should work.
Please how can we get the jupyter notebook code?
Great one
where do you edit your data? davinci?
As in DaVinci Resolve? I use Adobe Premiere for editing, think their products are the best for creative work.
@@DecisionForest Yeah DaVinci resolve. Adobe Premiere must be paid I guess.
@@flamboyantperson5936 Yes, it's part of Creative Cloud. If you use other apps then a subscription is worth it.
@@DecisionForest Will try. Thank you for useful information.
Can you apply SMOTE to text data
For oversampling text data I'd suggest you use pre-trained word embeddings. For example you can generate new samples by doing permutations of similar words.
Yes...Thanks
Thanks for this share.
Please Could you send me this code?
I need it .
are u sure that the undersampling method is work? the number still same 900.
It should yes. Could you detail please?
I think the pipeline just balancing the classes by oversampling automatically. and then because the class already balance, the undersampling didn't do anything
Oh yes, normally you either undersample of oversample as if we do both only the first is actually implemented. I did both sorry for the confusion, should’ve mentioned it.
@@DecisionForest it's okay I'm still learning too. If you want to do both, you must specify sampling strategy as parameter. otherwise they just autobalancing the classes.