3blue1brown has become a classic in TH-cam videos. We would describe arts as Escheresque or poetry as shakespearean but when a streamer defines a style that is compared to as a subgenre is some remarkable of an achievement by 3blue1brown
This video is a mix of actual explanation about the nature of transformers and long tangents about implementation details that are mixed together so perfectly it ensures that it's impossible follow or even know if the author of this video understands how transformers work.
The attention part gave me the most headaches I almost cried, felt so stupid. I found a better video in AssemblyAI that explains it better if that's the same case for you about the attention part.
I put 2 months into studying this. I’m a Cultural Science major and I can finally say that I understood every part of this video. I’m really proud of myself and very thankful for your excellent didactic style.
Thanks for the nice content. I am not a beginner, but unfortunately it was hard for me to follow this video. There were many concepts/terms mentioned without a brief explanation, and the pace was rather fast. If you could publish the same video, with additional examples and clarifications would be much appreciated. I understand that one would need to look up some topics and references while watching the video, but in this case it felt like I have to look up things very often. Thanks again for your effort!
you spoke for all of us, thank you for voicing our opinions. i'm a graduate data science alumni and i'm very familiar with deep learning. Watching this video, i wasn't learning anything rather i couldn't even follow-up with the video. I started litterally questioning my languistic and academic abilities. Then i realized that this guy is just summarizing the dreaded research paper "Attention is all you need" which is a bad idea if you're not going to cover explanation of terminologies and equations.
@@UpperM3 not really, to learn this concept it is necessary to at least have a basic idea of the concepts related to it. I think this is the best explaination of transformers in youTuble yet
@@UpperM3 this is more than data science, it's computer science. You need to have some understanding of comp sci concepts - parallelization, graphs, vectors, encodings, decodings. If you are "very familiar" with deep learning, you should know all this. But, yes this explanation is not geared towards beginners to deep learning/computer science. You need to understand neural networks, representations, communication paths, lengths etc. so the definitions are verbose at times. And you can always pause, and google the terms for more examples. This is technical and math jargon heavy, but yeah, it did not say it was explained for beginners. But, we can all spend some time to become more "expert". I suggest you to take notes, and fill in the gaps with Google. For e.g. " They leverage what is called self-attention, to compute updated representations for each sequence in parallel. The attention mechanism is going to allow each representation to kind of differentially consider the representation in every other position. And the communication paths will have the same length for all pairs of elements"
Thank YOU! this is the most precise and straight forward thing about Attention that I ever have. When you said the word compatibility, I finally understood why do I need to take dot product between Queries and Keys.
i am at 1/3rd of the video, stopped to come here express gratitude before getting back to it! 3B1B is a proven style, no harm in using it, later on adding to it.
Thanks for the vid. This is exactly level of the details and explanation style I needed. Many other explanations are either too vague and miss important details or too hardcore and hard to follow. This is ideal: most of the technical details are here, but still easy to follow.
Thank you for the clear and concise explanation! I understood this more than any other video I've seen yet (although I'm still learning). Looking forward to more videos like this!
These are great! I can see this channel easily becoming as popular as Yannic Kilcher’s. Thank you for all your work, your explanations give a lot of clarity without sacrificing depth!
Much needed video, if you have time I would request you to make a video series on transformers. My personal request would be on "Do Transformer Modifications Transfer Across Implementations and Applications?" and "Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth - " Again, I thank you for your work!
great explanation but when you were explaining the self-attention mechanism before multi-head attention, you forgot to mention that they way you get Q, K, V is by multiplying the input X by the corresponding weight matrix WQ, WK, or WV. When you said that "q, k, v are all learned" it made it sound like we're somehow learning q, k, v directly where as in fact we are learning the weight matrices that help generate them
Thanks for putting lot of efforts in making this video. I know how much hard work it takes to make such kind of videos. However, I would recommend as a viewer to explain with an example from the begining as it is really hard to understand with standard notations. I was really not able to follow after a while.
I appreciate the effort to put this together! 🤩 It is really beautiful! 😍 Unfortunately, the original formulation, equations, and diagram for the transformer are rather cryptic 😞
great video imagery... good job. Love seeing the flow of information, and little opinions about why things are happening. got a bit confusing when it went away from images to just pure equations, could do images and boxes with the equations?
During training you can feed your input encoder and decoder. What happens during prediction with new data? You don't know the output yet, How do you feed the decoder?
oh my brother, the rabbit hole you have sent me down.......... I'm very content . Send food from time to time please. Thank you for this video btw. I've had it on repeat for about a day. Well, ya-know, I ain't that smart. I did get CUDA installed and working in python though so that a win. Have a good day!
Around 7:20 you say Q, K, and V are all identical. What exactly do you mean? If we have separate matrices that generate these for each head (with the i indexing you mentioned), then doesn't that mean we will have h different WQ matrices, meaning we have h different Q vectors for each word (i.e. h different Q matrices for the sequence of words)?
This part could've been a little clearer. What I think can cause some confusion is that the letters Q, K and V are used in two different ways: 1) in the signature of the original Attention function (6:05) 2) in the signature of the MultiHead function (7:20) You're exactly right that we will have h different projected Q vectors for each word (same with K and V). What I'm referring to is that, for the encoder, the three arguments of the function MultiHead(Q, K, V) are all identical. The Q argument of MultiHead in layer i+1 of the encoder will be the stacked word representations output by layer i. This will also be the case for K and V, thus Q=K=V. In cross-attention, this is not the case: Q comes from the previous decoder layer while K and V are produced by the encoder. When we're doing multi-head attention, the three inputs to the original Attention function are *not* identical to each other, regardless of if we're doing self- or cross-attention. These inputs are produced by applying linear transformations to the arguments of MultiHead via the weight matrices.
hi. a small doubt: when you say the dot product of Query(Q) and Key vector(k), do you mean the "element-wise multiplication of each elements of Q & K followed by the sum of all these element-wise products such the resultant is only a single number" or simply "element-wise multiplication of each elements of Q & K, such that resultant is still a vector of the same dimension as Q&K"?
It looks like 256 and higher values of i in the positional encoding all go to 1 no matter the position value. So half the elements of the word embedding are wasted
9:51 This seems to be an error. The posicional embedding is concatenated to the word embedding. It is not a sum. Also, the posicional embedding may have an arbitrary dimension. EDIT: I was wrong
@@ariseffai Oh you are right! Thank you very much. I am supprise I didn't realize this before. I have to change my code haha. It seems weird to me because if you sum the positional to the word embedding, you are changing the information about the meaning of that particular dimension of the word embedding. I mean: Let's suppouse that the first number of the word embedding is 0.5. Let's suppose that that dimension has a meaning about the happiness of the word. That means that the word has a level of happiness of 0.5. If you add the positional embbeding you are changing that meaning of the word. You would mixing the information about happiness with some positional information. And yoy are doing this to EACH dimension of the word embedding. This seems messy to me. If you concatenate instaed of sum, this does not happen. But okay, I really don't know anything.
Hi, thanks for the video! There are several things that are still unclear to me. First I do not understand well how the architecture is dynamic with respect to the size of the input. I mean what does change structurally when we change the size of the input, are there some inner parts that should be parallely repeated? or does this architecture fix a size of max window that we hope will be larger than any sequence input? The other question is the most important one, it seems every explanation of transformer architecture I have found so far focuses one what we WANT a self attention or attention layer to do but never say a word of WHY after training those attention layers will do, by emergence, what we expect them to do. I guess it has something to do with the chosen structure of data in input and output of those layers, as well as the data flow which is forced but I do not have yet the revelation. If you could help me with those, that would be great!
Great video! Do you know if layer normalization is performed exclusively along the features axis, or if it's done both along the feature axis and the tokens (words) axis? Different sources say different things :( thank you
Authors write in the paper: "In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models". Of course, you're free to try out alternatives, but the video is not incorrect.
If this is for a university lecture, then this would be marvelous, however, most audiences are laymen, the explanation obfuscates the ease of understandig.
This is probaby a good video... ... ... ... Recorded with the --- Pensive pauses trademarked by 3 blue 1 brown. --- --- People say you're supposed to think during them --- --- But I think while the person is talking --- ___ __ And so I have all of this time ... ... .. Between the points the video makes --- --- --- Alone and with no new info. --- --- While everyone else thinks. ---- --- Teacher, is there a version of this that's already complete, where I could read it long-form without waiitng for others to think?
Someone needs to make an intuitive introduction to COMPLETE beginners, those who have no experience in AI, math, linear algebra, etc. etc. Almost all 'simple' tutorials I've come across never really give an intuitive explanation for the details, just a lot of jargon and math.
Completly agree with you. The most important is the intuition behind. Not how things are calculated. I'm not that novice in AI but i confess i understood very little here. For instance, how come we're using sin and cos to encode a position? We know the properties of those functions. So there's no point recalling them. What is the intuition behind keys, values and queries? We know how to project vectors and how to deal with linear transformations. And last, we're trying to translate a sentence. How come the input of the decoder is the target sentence we're trying to obtain? Those are the things a video should try to clarify. Otherwise, there's no value added.
Love the 3blue1brown esque style. Keep em coming
Thank you! Working on it :)
Appreciate both of you guys work in this field!
It's called manim, sir.
3blue1brown has become a classic in TH-cam videos. We would describe arts as Escheresque or poetry as shakespearean but when a streamer defines a style that is compared to as a subgenre is some remarkable of an achievement by 3blue1brown
Wish more videos were coming
I think I speak for everyone when I say.. please keep posting! This is excellent stuff!
The best help in understanding transformers while reading „Attention is all you need“ I have found.
This video is a mix of actual explanation about the nature of transformers and long tangents about implementation details that are mixed together so perfectly it ensures that it's impossible follow or even know if the author of this video understands how transformers work.
The attention part gave me the most headaches I almost cried, felt so stupid. I found a better video in AssemblyAI that explains it better if that's the same case for you about the attention part.
Your videos exhibit exceptional quality. Thank you for this outstanding contribution.
I put 2 months into studying this. I’m a Cultural Science major and I can finally say that I understood every part of this video.
I’m really proud of myself and very thankful for your excellent didactic style.
Our community is the best!!! 💪💪 Thank you very much for the amazing review!!!
Thanks for the nice content. I am not a beginner, but unfortunately it was hard for me to follow this video. There were many concepts/terms mentioned without a brief explanation, and the pace was rather fast. If you could publish the same video, with additional examples and clarifications would be much appreciated. I understand that one would need to look up some topics and references while watching the video, but in this case it felt like I have to look up things very often. Thanks again for your effort!
Absolutely. It started well then went off the rails.
you spoke for all of us, thank you for voicing our opinions. i'm a graduate data science alumni and i'm very familiar with deep learning. Watching this video, i wasn't learning anything rather i couldn't even follow-up with the video. I started litterally questioning my languistic and academic abilities. Then i realized that this guy is just summarizing the dreaded research paper "Attention is all you need" which is a bad idea if you're not going to cover explanation of terminologies and equations.
@@UpperM3 not really, to learn this concept it is necessary to at least have a basic idea of the concepts related to it. I think this is the best explaination of transformers in youTuble yet
@@UpperM3 this is more than data science, it's computer science. You need to have some understanding of comp sci concepts - parallelization, graphs, vectors, encodings, decodings. If you are "very familiar" with deep learning, you should know all this. But, yes this explanation is not geared towards beginners to deep learning/computer science. You need to understand neural networks, representations, communication paths, lengths etc. so the definitions are verbose at times. And you can always pause, and google the terms for more examples. This is technical and math jargon heavy, but yeah, it did not say it was explained for beginners. But, we can all spend some time to become more "expert". I suggest you to take notes, and fill in the gaps with Google.
For e.g. " They leverage what is called self-attention, to compute updated representations for each sequence in parallel. The attention mechanism is going to allow each representation to kind of differentially consider the representation in every other position. And the communication paths will have the same length for all pairs of elements"
@@UpperM3 there are also numerous videos that explain transformers without getting too technical -th-cam.com/video/SZorAJ4I-sA/w-d-xo.html
Thank YOU! this is the most precise and straight forward thing about Attention that I ever have. When you said the word compatibility, I finally understood why do I need to take dot product between Queries and Keys.
incredible, insightful, delightful, enlightening
i am at 1/3rd of the video, stopped to come here express gratitude before getting back to it! 3B1B is a proven style, no harm in using it, later on adding to it.
Really Attention is all we need.
Transformers are incredibly complicated but you tried to simplify it for us any way it does need another look
Thank you very much indeed
Maybe the best transformer explanation out there
Thanks for the vid. This is exactly level of the details and explanation style I needed. Many other explanations are either too vague and miss important details or too hardcore and hard to follow. This is ideal: most of the technical details are here, but still easy to follow.
best tutorial about transformer on youtube by far 🙌🏻
Thank you for the clear and concise explanation! I understood this more than any other video I've seen yet (although I'm still learning). Looking forward to more videos like this!
I watched a bunch of videos on this and I feel like after this, I actually understand it. Thank you!
Exceptionally explained! Please make more content! This stuff is worth paying for ;)
These are great! I can see this channel easily becoming as popular as Yannic Kilcher’s. Thank you for all your work, your explanations give a lot of clarity without sacrificing depth!
i thought i was smart - but then i started getting my head around all this amazing machine learning and then i was humbled. thanks for sharing
Much needed video, if you have time I would request you to make a video series on transformers. My personal request would be on "Do Transformer Modifications Transfer Across Implementations and Applications?" and "Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth - "
Again, I thank you for your work!
Very well illustrated! Thanks
I want to thank you for your effort on making these videos. I think your channel deserve more attention.
great explanation but when you were explaining the self-attention mechanism before multi-head attention, you forgot to mention that they way you get Q, K, V is by multiplying the input X by the corresponding weight matrix WQ, WK, or WV. When you said that "q, k, v are all learned" it made it sound like we're somehow learning q, k, v directly where as in fact we are learning the weight matrices that help generate them
Thanks ! Beautiful work.
Outstanding high-quality video. Thanks!
I also feel the residual connection is definitely RNN inspired.
Thanks Ari! Very useful.
Hi Ari, Seriously great stuff! Really appreciate the videos.
Definitely, TH-cam should review their suggestion algorithm. What do you think?
This channel is an incredible gem, thanks so much for your work!
Thanks for putting lot of efforts in making this video. I know how much hard work it takes to make such kind of videos. However, I would recommend as a viewer to explain with an example from the begining as it is really hard to understand with standard notations. I was really not able to follow after a while.
Amazing video.
Please make more content!
best explanation I've found! great job
Excellent explanation! Thank you!!
I really like your videos. Please keep uploading !
Thanks for a wonderful explanation.
I appreciate the effort to put this together! 🤩 It is really beautiful! 😍
Unfortunately, the original formulation, equations, and diagram for the transformer are rather cryptic 😞
Thanks Alfredo! Means a lot. I hope people can gain some clarity here.
Very well explained. Nice video!
You did such a good job explaining a not very simple topic, thanks!
Fantastic voice and description. Keep posting
I really liked your video, I finally can say I'm starting to get how it works, thank you !
Outstanding explanation!
this is amazing, well done!!
Thanks!
Thank you very much for sharing this. Wonderful work!
great video imagery... good job. Love seeing the flow of information, and little opinions about why things are happening. got a bit confusing when it went away from images to just pure equations, could do images and boxes with the equations?
Very good
Thanks for the video!
The transformer architecture diagram resembles one of Jordan Peterson's book illustrations. Except the transformer is actually coherent and useful.
Why do we sum the positional and input embeddings? Wouldnt concatenating make more sense? How would that play with dimensions?
Love the style
2:45
Let's take a moment to note: Dark Reader rules! :D
What is the meaning of fine-tuning and Pre-trained in Transformers?
Great explanation
Great explanation of the Transformer. Thank you!
During training you can feed your input encoder and decoder. What happens during prediction with new data? You don't know the output yet, How do you feed the decoder?
oh my brother, the rabbit hole you have sent me down.......... I'm very content . Send food from time to time please. Thank you for this video btw. I've had it on repeat for about a day. Well, ya-know, I ain't that smart. I did get CUDA installed and working in python though so that a win. Have a good day!
Great video! But I stiil can't understand it thoroughly..... What prerequisite knowledge do i need? Thanks!🙏
Just awesome!
Around 7:20 you say Q, K, and V are all identical. What exactly do you mean? If we have separate matrices that generate these for each head (with the i indexing you mentioned), then doesn't that mean we will have h different WQ matrices, meaning we have h different Q vectors for each word (i.e. h different Q matrices for the sequence of words)?
This part could've been a little clearer. What I think can cause some confusion is that the letters Q, K and V are used in two different ways:
1) in the signature of the original Attention function (6:05)
2) in the signature of the MultiHead function (7:20)
You're exactly right that we will have h different projected Q vectors for each word (same with K and V). What I'm referring to is that, for the encoder, the three arguments of the function MultiHead(Q, K, V) are all identical. The Q argument of MultiHead in layer i+1 of the encoder will be the stacked word representations output by layer i. This will also be the case for K and V, thus Q=K=V. In cross-attention, this is not the case: Q comes from the previous decoder layer while K and V are produced by the encoder.
When we're doing multi-head attention, the three inputs to the original Attention function are *not* identical to each other, regardless of if we're doing self- or cross-attention. These inputs are produced by applying linear transformations to the arguments of MultiHead via the weight matrices.
15:40 altought attention networks are expensive, we use them as they can be trained in parallel right ?
Please correct me if I am wrong
hi. a small doubt: when you say the dot product of Query(Q) and Key vector(k), do you mean the "element-wise multiplication of each elements of Q & K followed by the sum of all these element-wise products such the resultant is only a single number" or simply "element-wise multiplication of each elements of Q & K, such that resultant is still a vector of the same dimension as Q&K"?
I have question about this fragment 06:58. Im not sure but I suppose that R dimensionality schould be d_k*D not D*d_k
Here when you use "sequence", you mean a sentence or the whole text ?
great Video! But I start to lost from queries key values lol
8:35 I don't understand what is _pos_ and _i_ here. Here pos is the position in the sequence (0 for the first word, ...) but what is i ?
So the transformers came back to their home planet actually :)
I can't thank you enough. :)
how decoder match the size of output of the encoder with his input in every step to could make the multiplication of the matrixes
Me Brain be melted in a good way .
It looks like 256 and higher values of i in the positional encoding all go to 1 no matter the position value. So half the elements of the word embedding are wasted
great content!
9:51
This seems to be an error.
The posicional embedding is concatenated to the word embedding.
It is not a sum.
Also, the posicional embedding may have an arbitrary dimension.
EDIT: I was wrong
From the paper: "The positional encodings have the same dimension d_model as the embeddings, so that the two can be summed."
i.imgur.com/pJU5O9n.png
@@ariseffai Oh you are right! Thank you very much.
I am supprise I didn't realize this before. I have to change my code haha.
It seems weird to me because if you sum the positional to the word embedding, you are changing the information about the meaning of that particular dimension of the word embedding. I mean: Let's suppouse that the first number of the word embedding is 0.5. Let's suppose that that dimension has a meaning about the happiness of the word. That means that the word has a level of happiness of 0.5. If you add the positional embbeding you are changing that meaning of the word. You would mixing the information about happiness with some positional information. And yoy are doing this to EACH dimension of the word embedding. This seems messy to me. If you concatenate instaed of sum, this does not happen.
But okay, I really don't know anything.
Good! Liked & Subscribed
Good stuff!
Hi, thanks for the video! There are several things that are still unclear to me. First I do not understand well how the architecture is dynamic with respect to the size of the input. I mean what does change structurally when we change the size of the input, are there some inner parts that should be parallely repeated? or does this architecture fix a size of max window that we hope will be larger than any sequence input?
The other question is the most important one, it seems every explanation of transformer architecture I have found so far focuses one what we WANT a self attention or attention layer to do but never say a word of WHY after training those attention layers will do, by emergence, what we expect them to do. I guess it has something to do with the chosen structure of data in input and output of those layers, as well as the data flow which is forced but I do not have yet the revelation.
If you could help me with those, that would be great!
Great video! Do you know if layer normalization is performed exclusively along the features axis, or if it's done both along the feature axis and the tokens (words) axis? Different sources say different things :( thank you
Do you have a Twitter? I always share excellent videos I find there, and would like to give you credit?
@ari_seff :)
What animation software did you use?
This one used a combination of matplotlib, keynote, and FCP. I've also used manim in a couple videos.
Dude I was looking for a channel like yours for weeks !
Badass
13:06 is incorrect. The queries and keys come from encoder and the values come from previous attention block.
Authors write in the paper: "In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models". Of course, you're free to try out alternatives, but the video is not incorrect.
Did everybody get that? 😅
I’m going to guess that they are robots in disguise?
If this is for a university lecture, then this would be marvelous, however, most audiences are laymen, the explanation obfuscates the ease of understandig.
algorithm, more!
It is not clear from the visualization what you are talking about. A lot of the words are not defined and the diagrams are missing a lot of stuff.
Almost none of the terms and jargon were explained. People who need this video won't understand it, people who don't won't click.
We turn off the music when we talk
This is probaby a good video...
...
...
...
Recorded with the
---
Pensive pauses trademarked by 3 blue 1 brown.
---
---
People say you're supposed to think during them
---
---
But I think while the person is talking
---
___
__
And so I have all of this time
...
...
..
Between the points the video makes
---
---
---
Alone and with no new info.
---
---
While everyone else thinks.
----
---
Teacher, is there a version of this that's already complete, where I could read it long-form without waiitng for others to think?
Someone needs to make an intuitive introduction to COMPLETE beginners, those who have no experience in AI, math, linear algebra, etc. etc. Almost all 'simple' tutorials I've come across never really give an intuitive explanation for the details, just a lot of jargon and math.
Completly agree with you. The most important is the intuition behind. Not how things are calculated.
I'm not that novice in AI but i confess i understood very little here. For instance, how come we're using sin and cos to encode a position?
We know the properties of those functions. So there's no point recalling them.
What is the intuition behind keys, values and queries?
We know how to project vectors and how to deal with linear transformations.
And last, we're trying to translate a sentence. How come the input of the decoder is the target sentence we're trying to obtain?
Those are the things a video should try to clarify. Otherwise, there's no value added.
I like the animation style and the way you talk, but it was still really hard to follow. I did not understand much even though I study CS in a master
This is introduction to almost everything, I can't catch up or understand everything. Maybe this is for someone who understand them all already
why does this video have back ground music?
what a disaster this transformers are. they did not know what they were doing ha ha ha
Too much jargon without explanation mate
I lost my attention half way to
Whyyyyyy the annoying piano!!!!!