How to deal with Imbalanced Datasets in PyTorch - Weighted Random Sampler Tutorial
ฝัง
- เผยแพร่เมื่อ 21 ส.ค. 2024
- In this video we take a look at how to solve the super common problem of having an imbalanced or skewed dataset, specifically we look at two methods namely oversampling and class weighting and how to do them both in PyTorch.
Toy dataset used in video:
www.kaggle.com...
❤️ Support the channel ❤️
/ @aladdinpersson
Paid Courses I recommend for learning (affiliate links, no extra cost for you):
⭐ Machine Learning Specialization bit.ly/3hjTBBt
⭐ Deep Learning Specialization bit.ly/3YcUkoI
📘 MLOps Specialization bit.ly/3wibaWy
📘 GAN Specialization bit.ly/3FmnZDl
📘 NLP Specialization bit.ly/3GXoQuP
✨ Free Resources that are great:
NLP: web.stanford.e...
CV: cs231n.stanford...
Deployment: fullstackdeepl...
FastAI: www.fast.ai/
💻 My Deep Learning Setup and Recording Setup:
www.amazon.com...
GitHub Repository:
github.com/ala...
✅ One-Time Donations:
Paypal: bit.ly/3buoRYH
▶️ You Can Connect with me on:
Twitter - / aladdinpersson
LinkedIn - / aladdin-persson-a95384153
Github - github.com/ala...
![อยู่คนเดียวกับเขา - ส้ม มารี (Zom Marie) [Official MV]](http://i.ytimg.com/vi/gq1cWAIQDLg/mqdefault.jpg)
A tip that I didn't mention in the video is when you're iterating through the dataset to create the sample weights is to iterate through dataset.imgs rather than just dataset. This will run much faster because we are not resizing, performing transformations and so on which we do not need to do when we are only interested in the labels of the examples.
Hi Aladdin , First of all thank you for the super good video, but I didn't figure out which part should be " iterate through dataset.imgs ", can you point it out?
You sir, deserve way more subscribers for the consistency and the diversity of the topics you choose. Keep up the good work.
Awesome video bro, this has been really helpful. I'd like to share one trick of mine: Applying more random and strong data augmentation to the examples that are limited, and less random augmentation on examples that are quiet enough, and then making sure that each batch to the model receives an equal number of examples for each unique label. The only side effect here would be that you'd have to write your own custom dataloader that does that 🥱, and to be honest, it's not easy 😂, but once you set it up then its just a matter of copying and pasting for next projects :)
Thanks again for the video.
Hey Wolf! :) That sounds interesting, I would like to check it out if you could you share an example of such a custom dataloader?
the pytorch weighted random sampler is an amazing pytorch feature. Thanks for talking about it here.
Great video！A small bug: The orders of traversal between os.walk and datasets.ImageFolder are different. In the github code, We cannot guarantee that the smaller number of samples will get a greater sampling weight.
To tackle the imbalance one can also use the Focal Loss function. The kornia library has it.
Thanks Aladdin for all your videos, they are really awesome and informative
Really appreciate it bro
There is a bug in the program that can cause you a lot of trouble. After line 22 add this line of code -> subdir.sort()
os.walk() won't traverse your folders in alphabetical order so you have to sort the generator before appending the calculated weights. The could should look something like this:
for root, subdir, files in os.walk(root_dir):
subdir.sort()
if len(files) > 0:
....
The same problem arises for me, thanks for pointing out!
Thanks Aladdin! Would you mind recommend us your learning resources? I mean most of your teaching content are pretty rare in any ML/DL book.
I'm just sharing solutions to problems I face. Googling, reading docs and github, no good learning resource there unfortunately
Hi, how can we use the WeightedRandomSampler for Object Detection task?
Great lecture.
人像加入视频时候舒服多了！！不知道看哪的时候就看脸哈哈哈。
Great lectures.
Superb!!
I implemented AugMix DataAugmentation myself to increase the minority label's samples. My question is should we stick around one data augmentation technique which is state of the art OR we should try all of the others technique?
Thanks
I prefer to use RandAugment
Randaugment is one of the best augmentation method, it will improve your model performance and I was using weight normalization like this
nSamples = [346,168,106] # class samples
normedWeights = [1 - (x / sum(nSamples)) for x in nSamples]
normedWeights = torch.FloatTensor(normedWeights).to(device)
print(normedWeights)
nn.CrossEntropyLoss(weight=normedWeights)
is your second method different than this
I prefer oversampling (and you can still use RandAugment) rather than class weighting as it seems you're doing in the example
Thank you so much for your great content! I was wondering if the loader in the video is the train_loader? do you also apply oversampling on dev_train and test dataloader?
Aladdin, you deserve more subscribers. And you need to charge more :) Just joined as a member.
Appreciate you bro
How can you use pytorch and tensorflow so well. You must have invested so many hours of practice. How do you do that? Which API you like more for ML? In Keras you handle images differently and in pytorch you handle differently. It amazing to see that you have mastery over both. Well less people can do that. Usually people like to work with only one API.
I like PyTorch more. I think you overestimate me for sure, I'm way worse than what you think and need to google everything. You just see the refined version in the video and not the horrible mistakes I make :)
@@AladdinPersson You are too humble bro! A nice lesson learned: ), good job, and keep going!
@@AladdinPersson maybe you can make a video about how you get stuck and your train of thoughts and google queries to solve it
Awesome video and this channel is so underrated in DL community. I would like to know if there will be paper implementation tutorials in the future ?
Yeah for sure. Any paper in particular you'd wanna see?
@@AladdinPersson arxiv.org/pdf/1905.05908.pdf This one if possible. Can you make some tutorials on compositional zero-shot learning and implement some papers in that field.
Great video!
Hello, How can we apply under-sampling on your case? Oversampling and undersampling can be applied together?
Thank you
Dude you are awesome. I like your pytorch tutorials. But would love if you could use google colab for next ones.
permisiion to learn sir. thank you
Thanks man
Recap at 9:00
Great video！
Thanks!
Does we have to call get data loader for once for train and once for test set or just a single time for the entire dataset?
why did you multiply the sample_weights (which is zero) by len(dataset) ?
Thank you
Another amazing tutorial! I wonder if you have a patreon page where we can support you, it is well deserved. I have a question as well. I have a large imbalanced dataset. I need to call getloader function to get train and test loaders for hyperparamter tuning. Scanning the entire trainset in each function call makes the code slower. Do you suggest a work around? Thank you!
Hello, I still don't understand how replacement = True differs from replacement = False, may you explain me it please? Thank you :)
why we can not use shuffle while using sampler?
dataset link over!
What is the editor or Ide you are using
Hi Aladdin, Your code does not work and the result is still imbalanced. What should we do?
Why does the code "not work"?
I've been trying to do this in TensorFlow with ImageDataGens for over 5 hrs now, Could you help out?
I'm not sure how you do it in Tensorflow
@@AladdinPersson That's fine, d'you have a discord btw?
Hi , If we have 3 classes, how can we do it ?
Bro..are you in linkedin!?
Yeyup, same name :)
@@AladdinPersson Request sent😌
how to do it in tensorflow? I'm struck
Can u give me solution about computational cost or memory allocation problem related to oversampling imbalance dataset. It is too hard for me to buy gpu because of high cost and it is also problem for many of us. Can anyone here give solution related to the issue?
I've been using it in almost every other projects... but never understood what's so random about it? Why random in its name? 🙂
Donno actually 🤔
Do in TENSORFLOW also bro
I don't know a good way to do it in Tensorflow... :\ Maybe someone else knows and can help out?