Vision Transformers explained
ฝัง
- เผยแพร่เมื่อ 17 ส.ค. 2024
- Vision Transformer, also known as ViT, is a deep learning model that applies the Transformer architecture, originally developed for natural language processing, to computer vision tasks. It has gained attention for its ability to achieve competitive performance on image classification and other vision tasks, even without relying on convolutional neural networks (CNNs).
Transformers: • Transformers for begin...
**************************************************************************************
For queries: You can comment in comment section or you can mail me at aarohisingla1987@gmail.com
**************************************************************************************
The key idea behind the Vision Transformer is to divide an input image into smaller patches and treat them as tokens, similar to how words are treated in natural language processing. Each patch is then linearly projected and embedded with position information. These patch embeddings, along with position embeddings, are fed into a stack of Transformer encoder layers.
The Vision Transformer has shown promising results, demonstrating competitive performance on image classification tasks, object detection, and semantic segmentation.
#computervision #transformers - วิทยาศาสตร์และเทคโนโลยี
Aarohi, I am watching you for 3 years now, and each time I understand the subject. You're literally the best
Thank you so much for your incredibly kind words! It means a lot to me😊
Your explanation is one for the best I’ve heard about ViT, thank you very much
Glad it was helpful!
i agree
Hello Ma’am
Your AI and Data Science content is consistently impressive! Thanks for making complex concepts so accessible. Keep up the great work! 🚀 #ArtificialIntelligence #DataScience #ImpressiveContent 👏👍
My pleasure 😊
Thanks again for this very well explained tuto.
Glad it was helpful!
The content is amazing! Very informative, short, and to the point, which is great for beginners. Thank you for these amazing videos 😍I have only one small feedback for your future videos. The audio quality is a little bit bad and noisy. You might consider checking your microphone.
Thank you for the feedback. I will take care of noise.
Thanks for this vedio.this tutorial is very clear and explaining and we had learning to how to split the pattern
You are welcome 😊
Very nice Presentation
Thanks a lot
Your tutorials are always the best, thank you very much. I hope you would create tutorials on Segformer soon.
Thank you, I will
What's about the extra class? and i think that only the extra class is used for the classification. Please could you explain this point?
Very well explained. Thanks alot
Glad it was helpful!
Great....crystal clear the concepts greatly explained 😊
Glad it helped!
beautifully explained!
Glad it was helpful!
Hi, it is nice content btw I have one doubt.
If u divide 224*224 image into patch size = 16, that means there will be 16 grids as shown in 2:33 each patch having 14 pixels? Is my understanding correct?
You have a total of 14×14=196 patches. Each patch is 16×16 pixels in size. So, the number of grids (patches) is 196, and each patch has 16x16 pixels along each side, not 14x14.
@@CodeWithAarohi Oh okay. I got confused by looking onto image which had 16 cells. I hope that's a mistake right it should have been 14 cells right? And each cell has 16*16 pixels along its width and height
Thank You, can you explain difference between CNN and ViT side by side.
Can you please suggest how to use vision transformer for Text classification? Please
Excellent explanation
Glad it was helpful!
How dimension is reduced for each 1D vector when each pixel of 1D vector is multiplied by weights? Can u clear it?
Matrix multiplication!
Let's assume an image is split into 3x3 pixel and each pixel has 16x16 vector embedding which is flattened to 256x1 (means 256 rows and 1 column). Because we have 3x3 pixel size of image it means we have total 9 pixels. Hence if we combine the vector embedding of all the pixels (means if each pixel embedding is 256x1, then for 9 pixels it will become 256x9 i.e 256 rows and 9 columns. Now we have to pass this through linear layer. Linear layer let's say has 5 neurons. so shape for each neuron will be 256 x 1 (means 256 rows and 1 column) and for 5 neurons it will become 256x5 (menas 256 rows and 5 columns).
Now we have to do matrix Multiplication of Input with Linear layer, but wait, we cannot multiply the matrix because shape of input is 256x9 and shape of linear is 256x5. In order to multiply the matrices, the columns of Matrix A must be equal to the number of rows of Matrix B. So we will transpose the input matrix of shape 256x9 to 9x256.
Now, Let's take this as Matrix A of 9x256 and Matrix B of size 256x5. Because column of Matrix A is same as row of Matrix B, hence, dot product is possible which will result in new matrix of size 9x5.
We can see that the original matrix of patch was of size 9x256 which is reduced to 9x5.
So we will get the 3 matrices of size 9x5 each for Key, Query and Value.
Now based on attention model we can see that we have to do the matrix multiplication of Query and Key and to do so we again have to do the transpose of Matrix because both Key and Query are of same shape (Query Matrix - 9x5 , Key Matrix - 9x5). So if we transpose Key Matrix it will become 5x9 and then the matrix multiplication will be possible between these two matrices (9x5 and 5x9). The dot product output of these matrices will be a matrix of size (9x9) and this output matrix is called as Attention Filter.
Then after training we have the final updated values of this attention filter which we have to scale between 0 and 1 using softmax activation function.
This scaled attention matrix (9x9) is then multiplied with Value matrix (9x5) which will give the filtered value of Matrix (9x5).
Hence based on attention matrix we get the important feature of an image. This is the process of single attention head to extract feature.
We use multi-head attention to extract various important features of an image. Each head focuses on different combinations of features.
Thanks for making the video
My pleasure!
very good explanation. Thank you
You are welcome!
you are a genius ma Shaa Allah, thanks a lot
You are most welcome
Thanks very much the videos are awesome and genuine.
Glad you like them!
Please can you explain or give a series about the Vanilla Vision transformers from the paper to the to the programming side🙏🙏
The terms "Vanilla Vision Transformers" and "Vision Transformers" are often used interchangeably, and both refer to the same fundamental concept which is applying the Transformer architecture directly to image data for computer vision tasks.
Code with Aarohi is Best TH-cam channel for Artificial Intelligence #CodeWithAarohi
very nice video.
Many many thanks
Good one ma’am
Thanks a lot
the video was awesome . And can i know the transformer model of all the 6 encoders and 6 decoders , as I have confusion in the input architecture of decoders .
Thank you mam
Awesome explanation mam
Glad you liked it
I came to this video to learn how to do positional encoding for 2D images -- the precise math. When you come to that portion, you simply reference your intro video, re Transformers for linear text (and in which even the linear positional encoding isn't really explained).
Sorry for inconvenience. I will try to cover the math's in separate video.
@@CodeWithAarohi Thanks for responding. Your videos are terrific otherwise. Thanks for sharing your work and insights.
‘The precise math for encoding’
Bro it’s deep learning and you talk about precise math 😂
Literally those people encoded 1,2,…256 for each patch.
Thanks for this tutorials its simple and deep
You're welcome 😊
Waiting for your fusion transformer tutorial mam
Ok sure
Thank you soo much mam for this amazing video
Most welcome 😊
can you make a video on SegFormer? thanks in advance for the amazing explanation!
I will try!
Nicely Explained..!
Thank you
excellent explanation.
I wanna make a sugesstion. Maybe you should buy a microphone. There are lots of noise in background.
Thank you, I will
Kindly post a video for Deit
Noted!
Thanks a lot! it helps me :3
I'm glad!
how to know the feature importance which are generated from ViT ?
which features causes classification
While ViT doesn't inherently provide feature importance scores like some other models, you can analyze the importance of different features in the classification process by examining the attention maps generated by the model.
Attention maps in ViT represent the importance of each image patch in relation to the final prediction. Higher attention values indicate greater importance. By visualizing these attention maps, you can gain insights into which image regions contribute most to the classification decision.
@@CodeWithAarohi please make video on it madam, for one classification task , dog vs cat classification example
Thank you for making this video. Please make a python code of ViT, if possible. Thank you.
Working on it!
Please, do it on how to train ViT on semantic segmentation tasks.
Can these be apply in bank cheque processing
Yes, vision transformers (ViTs) can be applied to bank cheque processing tasks.
thank you so much Aarohi, please,could you explain SWIN transformer too with its with coding ?
Sure, I have started a playlist on transformers and will try to cover every important topic which comes under transformers
@@CodeWithAarohi thank you again, waiting for it
please create a ViT on landmark detection
I will try
@@CodeWithAarohi thank you so much you are the best
nice, can you share slide with me?
Can you provide code
In this video, I have explained Vision transformer theory. You can check next video and Code link is mentioned in description section of that video.
Why only 16×16
Original paper used this. You can try with different numbers also.
will you do vision transformers with tensorflow?
I will try.
@@CodeWithAarohi Thank you. Your method of teaching is amazing. But i am never comfortable with torch. Tensorflow is so natural for deep learning. I look forward to this .