Great explanations! The self-labeling step reminds me of the paper "ClusterFit: Improving Generalization of Visual Representations", which shows a lot of promising results by using pseudo labels from clustering to retrain a new classifier.
This is very nice and a nice explanation of it. This works so well in this paper partly because the input dataset is nicely separable into discrete clusters. Try this with photos from the wild, not cropped to include the frog/car/object in the center of the photo. Unsupervised, it's pretty unlikely that you'll get classes you like.
I am currently bingeing your video during my work hours - but, as a PhD student in this field, I don't even feel guilty because I am learning so much. Great work, keep it up!
Cool! I was actually going to try doing this myself, exactly the same steps and all, unsupervised learning -> k-means -> self labeling. Awesome to see I wasn't so crazy after all, great explanation 😁
I had also something similar (and simpler) in mind before I watched this video: Clustering the bottleneck-encodings of images. Surely that’s made before but I haven’t found any results on the internet
Wonderful Video @Yannic. Couple of years back, during my Postdoc, I learnt Attention by hearing through your video on "Attention is All you Need" and then started my research work to build based on the intuition I got. I now get a good idea of Self Supervised Learning !!!
When I started watching the video, I was not sure if I'd be able to sit for 45 minutes to understand this paper, but thanks to your great explanation skills, I sailed through it...
Great paper review and explanation!! Thanks a ton!!! It definitely saved lot of my time in reading and understanding entire paper. Now its easy to go back and implement things
Point made in the last 30 seconds is such an important one. All hyper-param choices are based on label information making this more of a playground experiment rather than something robust
Super helpful, keep up the great work Yannic! Your ability to filter out the intuitions make you an incredible instructor. ps: what is the note-taking software you're using?
Great explanation as always, thank you! It would have been even better if you had explained the evaluation metrics ACC (clustering accuracy), NMI (normalized mutual information) and ARI (adjusted rand index).
This is too good to be true. I wouldn't be surprised if nobody is able to replicate this. But if it does work, it could open up a lot of possibilities in unventured territories in computer vision.
Many hyperparams indeed, the authors claim code + configuration files will be released soon, sounds really promising. Defining the class (dis)agreement on the embedding neighbourhood is a fine piece of representation learning 👌.
@@simonvandenhende5227 great work. what's the best way to communicate with you guys? For example, CIFAR100 ACC is significant lower than ImageNet-100 ACC, any thought why?
@@dennyw2383 You can contact me through email. CIFAR100 is evaluated using superclasses, e.g. vehicles = {bicycle, bus, motorcycle, pickup truck, train}, trees = {trees, maple, oak, palm, pine, willow}. These groups were composed based on prior human knowledge, and not on visual similarities alone. This is the main reason I see for the lower accuracy on CIFAR100. Another reason that also relates to the use of superclasses, could be the increased intra class variability.
Can you explain the basics of deep learning using the published papers for algorithms as you do now. You. Have a way in teaching that makes me do not fear from scientific papers 🙂
I have a little question about equation 2 in the paper. How is the soft assignment phi^{c} is calculated? They simply they: "The probability of sample Xi being assigned to cluster c is denoted as Φcη(Xi) but never mention how to calculte it. Am I missing this?
Yannic, could you please make a video on essential mathematics that are required for to be DL researchers? I am an CS undergrad and I always find myself not knowing enough mathematics while reading paper. Is is the case for everyone? I am amazed at your ability to go through papers with such understanding. Could you share with us how you prepared yourself that way? BS: excuse my English.
Hey, never be ashamed of your English, it's cool that you participate :) That's a good idea, but the answer will be a bit boring: linear algebra, real (multidimensional) calculus, probability / stats and numerics are most relevant
Your videos are amazing. Not only you have the technical knowledge, but you do a wonderful job explaining things. If I may suggest creating an advanced course where you show researchers/students how to implement the algorithms in these papers. I would be your first student lol :).
From the examples, I had the suspicion that it may works based on *colour and structure of the background* , combined with *confirmation bias* . The shark cluster may not care much about the sharks, bur more so about the blue water that surrounds it. The spiders may just be things in focus in front of a blurred background, caused by the small depth of field of a macro photo. It may also be based on the shape of the colour histogram, that covers more of the example clusters shown, and includes information about the structure and colours of object and background. At least in some examples it is a very strong effect, so strong that it needs confirmation bias by the authors to miss it. Maybe it is discussed in the paper, I did not check.
I would suspect that overclustering is done by shoving in a whole block of data from one class and assign the output with the highest peak the corresponding label, though can't be sure. And shouldn't the accuracy be expected to be lower with more classes, since the entropy term is maximizing the number of different clusters.
Nice videos, in particular your skepticism. How do you select the papers you will review? I find myself with a mountain of papers to read but time is never enough.
at 32:34 the accuracy of self-supervised learning followed by K-means is 35.9%. How do they decide the representing label of a cluster? Is the representing label is the majority ones in the cluster?
I have a question, what would happen if we pass images through a pretrained model such as inception and then use the obtained feature map for clustering?
6:58, a stupid question here. So if the downstream task is not a classification task, would the euclidean distance still make sense in the learned representation space? I think it does, but I am not sure. I'd really appreciate it, if anyone can shed some light here. Thanks.
43:30 ive not read the paper yet but your argument for overclustering does not apply if the authors evaluated the model on a different set that they trained on.
Should I be familiar with a information theory text book to appreciate this paper ? I'm really not sure which math text books to read to understand ML papers better.
Great Video. Really easy to understand. Thanks for that! Can i ask you something? I'm trying to compare different Clustering results of different Clustering approaches on image data. Is it possible to use internal validation indexes i.e. davies-bouldin-score? Or are there problems in terms of the euclidean space? Keep it up
I have a question: Why in representation learning, the network won't cheat and classify everything(all kinds of classes) the same? Is there a regularization that is not shown here (for example, encouraging having a diverse output)?
Hey, great videos. I have a question though. In the Representation Learning part, it seems very similar to the image reconstruction like using a variational autoencoder. Some people consider it as unsupervised learning. So what's exactly is the difference between self-supervised and unsupervised learning?
@@YannicKilcher thanks for the reply. So what, according to you, is a clear cut example that differentiates between self-supervised and unsupervised learning?
Hi, Very interesting and informative video, I have a question how do I go about detecting symbols in an engineering drawing using your technique explained here?
![[TH] 2024 PMSL SEA Finals D3 | Fall | รู้ผลทีมแชมป์ พร้อมตั๋วสู่ PMGC](http://i.ytimg.com/vi/7NxsN0Ch6vk/mqdefault.jpg)
![โฮ่ง! (SUGOI) - LYKN [ OFFICIAL MV ]](http://i.ytimg.com/vi/nFp4zjAARFs/mqdefault.jpg)
The fact you allowed us to participate in your confusion about the norm-not-norm-issue is sooo valuable. Great fan of your work, thx!
OUTLINE:
0:00-Intro & High-level Overview
2:15-Problem Statement
4:50-Why naive Clustering does not work
9:25-Representation Learning
13:40- Nearest-neighbor-based Clustering
28:00-Self-Labeling
32:10-Experiments
38:20- ImageNet Experiments
41:00-Overclusteringg
Great explanations!
The self-labeling step reminds me of the paper "ClusterFit: Improving Generalization of Visual Representations", which shows a lot of promising results by using pseudo labels from clustering to retrain a new classifier.
This is very nice and a nice explanation of it. This works so well in this paper partly because the input dataset is nicely separable into discrete clusters. Try this with photos from the wild, not cropped to include the frog/car/object in the center of the photo. Unsupervised, it's pretty unlikely that you'll get classes you like.
I am currently bingeing your video during my work hours - but, as a PhD student in this field, I don't even feel guilty because I am learning so much. Great work, keep it up!
Cool! I was actually going to try doing this myself, exactly the same steps and all, unsupervised learning -> k-means -> self labeling. Awesome to see I wasn't so crazy after all, great explanation 😁
I had also something similar (and simpler) in mind before I watched this video: Clustering the bottleneck-encodings of images. Surely that’s made before but I haven’t found any results on the internet
@Victor, Which Dataset are you aiming to work with?
Thanks for the great content
Honestly, the entire channel is an actual gold mine!
Please keep up the excellent work :)
+1
being wondering inside the paper for hours and finding this vid is a big gift for me
Wonderful Video @Yannic. Couple of years back, during my Postdoc, I learnt Attention by hearing through your video on "Attention is All you Need" and then started my research work to build based on the intuition I got. I now get a good idea of Self Supervised Learning !!!
There were some really interesting RL papers in ICLR'20. You should cover a few of them. Great explanation again, keep up this work !
Can you provide some links? I'm really interested in those papers.
When I started watching the video, I was not sure if I'd be able to sit for 45 minutes to understand this paper, but thanks to your great explanation skills, I sailed through it...
label exists.
Augmentation : I am about to end this man's career.
Great paper review and explanation!! Thanks a ton!!! It definitely saved lot of my time in reading and understanding entire paper. Now its easy to go back and implement things
Amazing method, I hope RL can use this method during self cruosity rewards. So then there will be less manual rewards for bunch of locomotion tasks
Point made in the last 30 seconds is such an important one. All hyper-param choices are based on label information making this more of a playground experiment rather than something robust
Super helpful, keep up the great work Yannic! Your ability to filter out the intuitions make you an incredible instructor.
ps: what is the note-taking software you're using?
OneNote, thanks :)
@@YannicKilcher fantastic..good job .. keep going .. I myself before reading a paper look through your videos in case you have reviewed it already
me too, very good channel@@IndranilBhattacharya_1988
Summary of Paper
1. Learn good embedding
2. Learn Classes based on embedding
3. (Extra) Use learned classes to train new NN.
K-nearest neighbours but with neural networks
Great explanation as always, thank you!
It would have been even better if you had explained the evaluation metrics ACC (clustering accuracy), NMI (normalized mutual information) and ARI (adjusted rand index).
Don't worry so much about the mistakes you do. You are doing a great job!
Thank you, for such an amazing overview
This is what I have been looking for
This is too good to be true. I wouldn't be surprised if nobody is able to replicate this. But if it does work, it could open up a lot of possibilities in unventured territories in computer vision.
Many hyperparams indeed, the authors claim code + configuration files will be released soon, sounds really promising. Defining the class (dis)agreement on the embedding neighbourhood is a fine piece of representation learning 👌.
We released the code over here :) github.com/wvangansbeke/Unsupervised-Classification
@@simonvandenhende5227 great work. what's the best way to communicate with you guys? For example, CIFAR100 ACC is significant lower than ImageNet-100 ACC, any thought why?
@@dennyw2383 You can contact me through email. CIFAR100 is evaluated using superclasses, e.g. vehicles = {bicycle, bus, motorcycle, pickup truck, train}, trees = {trees, maple, oak, palm, pine, willow}. These groups were composed based on prior human knowledge, and not on visual similarities alone. This is the main reason I see for the lower accuracy on CIFAR100. Another reason that also relates to the use of superclasses, could be the increased intra class variability.
Awesome explanation. Thanks a ton
Can you explain the basics of deep learning using the published papers for algorithms as you do now. You. Have a way in teaching that makes me do not fear from scientific papers 🙂
Nice explanation! Thank you Yannic!
I have a little question about equation 2 in the paper. How is the soft assignment phi^{c} is calculated? They simply they: "The probability of sample Xi being assigned to cluster c is denoted as Φcη(Xi) but never mention how to calculte it. Am I missing this?
It's probably a softmax after some linear layers
Amazing review, thank you so much. Keep going....
Hi I was training the model in the scan stage the total loss displayed is -ve hence to reduce we need to go from say -4 to -9 rite ? Silly question
Yannic, could you please make a video on essential mathematics that are required for to be DL researchers? I am an CS undergrad and I always find myself not knowing enough mathematics while reading paper. Is is the case for everyone?
I am amazed at your ability to go through papers with such understanding. Could you share with us how you prepared yourself that way?
BS: excuse my English.
Hey, never be ashamed of your English, it's cool that you participate :)
That's a good idea, but the answer will be a bit boring: linear algebra, real (multidimensional) calculus, probability / stats and numerics are most relevant
I have one question. If I have no labeled images, Is it possible? On update memory bank (in simclr as pretext), does it need labels??
Your videos are amazing. Not only you have the technical knowledge, but you do a wonderful job explaining things. If I may suggest creating an advanced course where you show researchers/students how to implement the algorithms in these papers. I would be your first student lol :).
God that sounds like work.... just kidding, thanks for the feedback :)
@@YannicKilcher I know there is a fine line between too much fun and work :P. This would be a graduate-level course.
How do we avoid the network projecting all images to 0 in the 1st step if we are not using a contrastive loss function?
From the examples, I had the suspicion that it may works based on *colour and structure of the background* , combined with *confirmation bias* .
The shark cluster may not care much about the sharks, bur more so about the blue water that surrounds it. The spiders may just be things in focus in front of a blurred background, caused by the small depth of field of a macro photo.
It may also be based on the shape of the colour histogram, that covers more of the example clusters shown, and includes information about the structure and colours of object and background.
At least in some examples it is a very strong effect, so strong that it needs confirmation bias by the authors to miss it. Maybe it is discussed in the paper, I did not check.
I would suspect that overclustering is done by shoving in a whole block of data from one class and assign the output with the highest peak the corresponding label, though can't be sure.
And shouldn't the accuracy be expected to be lower with more classes, since the entropy term is maximizing the number of different clusters.
This video is good 👌... If possible, can you explain the concept with coding..
+1
Thanks for the great effort 👍
Thanks for the amazing content!
The thing I love most is the sassy hand drawing lmao
at 15:06 why do they use the log in the first term of the equation?
I guess it's so they can interpret the inner product as a likelihood
Nice videos, in particular your skepticism. How do you select the papers you will review? I find myself with a mountain of papers to read but time is never enough.
Same here, I just read what seems interesting.
How well would it work to cluster the bottleneck-encoding of an autoencoder?
Good question, worth a try
Hi there, hope you are doing well. I want to know can we use only supervised learning for unlabeled image dataset?
at 32:34 the accuracy of self-supervised learning followed by K-means is 35.9%. How do they decide the representing label of a cluster? Is the representing label is the majority ones in the cluster?
That's not a flute, it's an oboe! :p
Always enjoy your videos btw :)
I have a question, what would happen if we pass images through a pretrained model such as inception and then use the obtained feature map for clustering?
Congrats for your video!!!
6:58, a stupid question here. So if the downstream task is not a classification task, would the euclidean distance still make sense in the learned representation space? I think it does, but I am not sure. I'd really appreciate it, if anyone can shed some light here. Thanks.
thanks cool videos! very informative, I always try to distill knowledge from your explanations :-)
43:30 ive not read the paper yet but your argument for overclustering does not apply if the authors evaluated the model on a different set that they trained on.
25:19 why in L2 space the inner product is always 1?
Should I be familiar with a information theory text book to appreciate this paper ? I'm really not sure which math text books to read to understand ML papers better.
Nope, this is very practical
such dope research!!
Great Video. Really easy to understand. Thanks for that!
Can i ask you something?
I'm trying to compare different Clustering results of different Clustering approaches on image data. Is it possible to use internal validation indexes i.e. davies-bouldin-score? Or are there problems in terms of the euclidean space?
Keep it up
Could you re-explain why euclidean distance would not work for raw images?
because two images can be very similar to humans, but every pixel is different
@@YannicKilcher this makes sense. Thanks!
It would be great to contact the authors to see what they think of your videos !
I have a question: Why in representation learning, the network won't cheat and classify everything(all kinds of classes) the same? Is there a regularization that is not shown here (for example, encouraging having a diverse output)?
there are a nmber of tricks, but mostly it's because of stochasticity, normalization and the inclusion of negatives
Hey, great videos. I have a question though. In the Representation Learning part, it seems very similar to the image reconstruction like using a variational autoencoder. Some people consider it as unsupervised learning. So what's exactly is the difference between self-supervised and unsupervised learning?
It's pretty much the same thing. Self-supervised tries to make it more explicit that it's "like" supervised, but with a self-invented label.
@@YannicKilcher thanks for the reply. So what, according to you, is a clear cut example that differentiates between self-supervised and unsupervised learning?
Nice overview.
great video, thanks!
Hi, Very interesting and informative video, I have a question how do I go about detecting symbols in an engineering drawing using your technique explained here?
you'd need a dataset,
@@YannicKilcher Then it will be a labled data right , can you elaborate, my email id is sarc007@gmail.com
So the first step is the most important thing rite ? Becz the later r learning from it.
yes
Woooho that is smart xD
nice dog drawing
Would you make tutorial every time.
Stop doing clicky sounds with your tongue. So annoying.
What a choot :)
0:33x.