this is great but would've loved if you could have taken a sample sentence as an input and show us how it transforms as it moves through the different parts of the transformer. Perhaps an idea for the next video!
I must say you’ve given the best explanation on transformers that’ve saved me lots of time studying the original paper. Please produce more vids like this, I would recommend the BERT family and the GPT family as well 👏👍
Sorry to be offtopic but does anyone know of a tool to log back into an instagram account?? I stupidly forgot the password. I appreciate any tips you can offer me.
@Matias Santino I really appreciate your reply. I found the site through google and Im trying it out atm. Takes a while so I will reply here later with my results.
12:56 encoder has hiddens state of key-value pairs, and in the decoder, the previous output is compressed into a query. The next output is produced by mapping this query and the set of keys and values.
@@mrexnx Correct me if I'm wrong but the only reason you put the mask so it doesn't attend to "future" words in the sentence is cause of the nature of the Ebglish language..since English is written left tor ought unlike other languages. Otherwise you shouldn't have thst mask because you would need to attend to words on right or maybe left also?
There is actually a small mistake at 12:56: The Ecnoders Output are the VALUES and keys! for the Decoders second self attention. So it is: Value and Key from Encoder with Query from Decoder are combined. from the "Self Attention is all you need" paper: "In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder."
Ik that's true but still it doesn't make sense, What's the use of the masked attentions score that we just computed in that Multi-Head attention layer since it's not carried forward
I asked chatgpt (GPT4) for the same and look at the response: "Let's break down the process: Encoders in Transformers: They take the input sequence and process it through multiple layers of self-attention and feed-forward networks. The final output from the last encoder is used as the "Keys" (K) and "Values" (V) for the subsequent decoder. Decoders in Transformers: Decoders also have multiple layers. In the first self-attention mechanism of the decoder, the target sequence (for example, the partially generated translation) attends to itself. This is analogous to the self-attention in the encoder but has a slight difference. In the decoder's self-attention, future tokens are masked, so they can't be attended to. This ensures that the prediction for a particular word doesn’t depend on future words in the sequence. In the subsequent multi-head attention mechanism, the "Queries" (Q) come from the decoder's previous layer, and they attend to the "Keys" (K) and "Values" (V) from the encoder's output. This essentially means that the decoder is using the information from the encoder to help generate the next token in the sequence. So, your statement is correct: "Value and Key from Encoder with Query from Decoder are combined." In the Transformer's decoder, for every step in its layers, the Queries (Q) from the decoder attend to the Keys (K) and Values (V) from the encoder output."
indeed it is like this as the dot product of the keys and querries construct the relation between the input and the already generated output if noy K and Q where from the encoder it wouldn't capture the relation between the input and already generated output
I have been struggling with this architecture for an eternity now and this is the first time I really understood what's going on in this graphic. Thank you so much for this nice and clear explanation!
Correction: The sine and cosine functions for the positional embedding are applied to the input embedding dimension, not the time steps! oof! For the readers check out the written version of an illustrated guide to Transformers here towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0
My first read about Micheal Phi was "Stop Installing Tensorflow using pip for performance sake!" in TowardDataScience blog (as I recall you was "Micheal Nguyen" at that time). My first impression was like "oh this guy was good at explanation". Then I read his several blogs, and now here I am. I never knew that you have a channel. You are one of the best educator I've ever known. Thanks so much.
This was not just the best transformer explanation, but the best explanation in general that I've ever seen. You know what to abstract for a fluid and clear explanation. Congratulations. You should do more videos like this.
Woah! Exquisite, It's a 15 min video but I spent over an hour taking notes and understanding. You have done a great job, keep it up. Thank you so much! Such explanations are rare. ;)
Now i have two great heroes that explain complex concept using mindblowin visualization, first is 3b1b for complex math topics, then Michael Phi for complex machine learning architecture! Just wow ... salute sir! thank you so much!
Wow , this was great, I have watched a no of videos on the transformer models, and they have all contributed to my understanding, but this puts everything together so neatly. Amazing, please keep making more such videos.
Thanks for your explanation, very clean and well built in every argument about transformers. I was so lucky to get this video randomly on TH-cam. Good job!
I used multiple sources to learn about the transformer architecture. Regarding the decoder part, you really helped me understanding what was the input and how the different operations are performed ! Thanks a lot :)
Thank you for this amazing explanation. It really helped after an insufficient explanation from my DL lecture. The prof did not even mention that the final part is a classification over the vocabulary for an nlp task!
Wow, by far the best explanation. Teaching really is an art where even experts with their blind spots can be very bad at explaining (even if they know it well themselves). Well done. It's really too bad your previous video was a year ago. I really hope you make more videos explaining other deep learning papers such as Faster RCNN etc. Thanks for posting this.
Thank you so much for the step by step explanation. This is a good starting point for ML dummies like me to get a grasp on the transformer model architecture.
Good explanation. What boggles my mind is that this architecture can not only produce reasonable sentences, but there can be some logic going on behind the sequence of sentences as we've seen in chatGPT. It is mind-boggling that there must be some amount of deeper conceptualization represented in the embeddings too! amazing
That's most clear and breif explanation about the key idea of 'transformer', especially on how transformer works and how self-attention self attended why self attention is self attention.
Brooo you are seriously my god😭😭🙏🙏...thanks a lot for this video...no one... literally no one could teach me transformer and your video just got drilled into my mind...please make other videos like this for bert gpt xlnet xlm etc etc... I'm really thankful to you
Quick Q : Say at 5:58, on the left side of the slide, QKV are inputs to linear layers, but on the right it looks like positional encodings are passed to the linear layers and Q,K,V are outputs of the layers. I guess the left is just a typo and right side is correct representation. Can you please help confirm. By the way this is the best explanation i have found, you are just amazing. Thank you so much brother.
Brilliant explanation with visually intuitive animations ! I rarely comment or subscribe to anything but this time I instantly do both after watching the video. And how coincidental it is that this was uploaded on my birthday. Hope to see more videos from you.
12:55 A small mistake: K and V should be encoder stack's output, and Q is the first Multi-headed Attention sublayer's output. Still, this guide is really awesome! Thanks for your effort bro!
Favourite video on the topic! I'm reasonably knowledgeable on ML, but the other 5-10 videos I've tried so far all resulted in increased confusion. This is clear.Nice one 👍🏿
Thanks a lot for the intuitive introduction. But at 6:29, shall we transpose the matrix of Q, K here? In the video, multipling a 3x4 matrix and a 4x3 matrix leads to a 3x3 score matrix instead of 4x4 one
Wow Michael, this is a superb explanation of the transformer architecture. You even went into detail about the meaning of the Q,K,V vectors and masking concepts which were hard for me to grasp. I bounced around through 3-4 videos about the transformer arch, and for each one I claimed it was the best explanation on the topic. But your video takes the cake and explains it in half the time as the others. Thank you for sharing! Also, great job on the visuals which are on par with 3blue1brown's animations.
While I still have some questions, this is a pretty good explanation, I mean I actually have an idea of how this works! Gonna watch it like 2 more times.
This is the best explanation of a transfomer or any architecture I have ever watched. Highly impressive how such a complex topic is dismantled into visually appealing and understandable explanations.
Man thanks for this video, reading a paper for newbie is super difficult, but such explanations like you've posted for key, value and query as well as reasoning for masking is very, very helpful. I subscribed to your channel and am looking forward for new stuff.
Wow. This is some really good explanation! I don't have much NLP background except RNN/LSTM and things before DL (N-gram), but wanted to know more about Attention mechanism for robotics application. (my field) Most other explanation either skimmed over the mathematics, or used NLP specific nomenclature/concepts that made it hard to understand for non-NLP people. This was some good stuff! Much appreciated and Keep up the good work!
🎯 Key Takeaways for quick navigation: 00:00 🌐 Transformers are revolutionary in natural language processing, breaking NLP records and finding applications in various fields. 02:43 🤖 The core concept of Transformers is the attention mechanism, which allows the model to focus on relevant words. 07:16 🔍 Multi-headed attention in Transformers lets the model associate each word with others, providing a rich context for generating text. 10:06 🧠 Transformers use an encoder-decoder architecture, with the decoder being auto-regressive, generating text word by word. 12:48 🔄 Stacking encoder and decoder layers in Transformers enhances their ability to learn and process different attention representations. Made with HARPA AI
This illustrated explanation is just so well done :OOOOO I'm a novice at Deep neuronal networks and just by looking at the video, I just understood everything ! Completely recommended to understand Transformers :) Good work :D
In "encoder-decoder attention" layers, the "queries "come from the previous decoder layer, and the" keys "and "values" come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]. -- Attention is all you need
A really good and nice video that helped me understand transformer quicker, thanks ! After reading a little more, I think there is small mistake about the roles of the encoder outputs and the decoder first Multi-Headed attention outputs. The encoder output is supposed to be the Key and the Value and the decoder first Multi-Headed attention output the Query (you say that the encoder output is Q and K and the decoder first Multi-Headed attention is V). A small error that got me to think a little bit too long on how the dimensions could match at the end of the Multi-Headed attention ^^ Thanks again for the video, it was clear as crystal !
感谢！ Thank you so much, I've looked up all the resources on the Internet but still messed up with the mechanism. It's really a clear and detailed explanation.
this is great but would've loved if you could have taken a sample sentence as an input and show us how it transforms as it moves through the different parts of the transformer.
Perhaps an idea for the next video!
@The A.I. Hacker - Michael Phi please do !
The video actually led me to expect this example as well! It would be highly beneficial.
I must say you’ve given the best explanation on transformers that’ve saved me lots of time studying the original paper. Please produce more vids like this, I would recommend the BERT family and the GPT family as well 👏👍
I agree. I can't seem to find a good explanation on the BERT model
Sorry to be offtopic but does anyone know of a tool to log back into an instagram account??
I stupidly forgot the password. I appreciate any tips you can offer me.
@Matias Santino I really appreciate your reply. I found the site through google and Im trying it out atm.
Takes a while so I will reply here later with my results.
@Matias Santino It worked and I actually got access to my account again. I'm so happy:D
Thank you so much you saved my account!
@Ronnie Adam you are welcome :)
Man this guy made a video on Transformers 4 years ago!!
You are awesome!!
Perfect explanation!!!!!
Thanks a lot for the video!
A complex process- I need to listen to this multiple times to fully understand this.
Same here
12:56 encoder has hiddens state of key-value pairs, and in the decoder, the previous output is compressed into a query. The next output is produced by mapping this query and the set of keys and values.
this is critical! I was pretty confused on this for awhile until I realized he swapped the Query and Values on accident.
ah someone else realised it. this comment should be pinned ^^
you are correct
@@mrexnx Correct me if I'm wrong but the only reason you put the mask so it doesn't attend to "future" words in the sentence is cause of the nature of the Ebglish language..since English is written left tor ought unlike other languages. Otherwise you shouldn't have thst mask because you would need to attend to words on right or maybe left also?
@@leif1075 I also thought something like that, that means, in case of Arabic, this direction of masking should not work !
There is actually a small mistake at 12:56: The Ecnoders Output are the VALUES and keys! for the Decoders second self attention.
So it is: Value and Key from Encoder with Query from Decoder are combined.
from the "Self Attention is all you need" paper: "In "encoder-decoder attention" layers, the queries come from the previous decoder layer,
and the memory keys and values come from the output of the encoder."
Ik that's true but still it doesn't make sense, What's the use of the masked attentions score that we just computed in that Multi-Head attention layer since it's not carried forward
yes you are right
I asked chatgpt (GPT4) for the same and look at the response:
"Let's break down the process:
Encoders in Transformers:
They take the input sequence and process it through multiple layers of self-attention and feed-forward networks.
The final output from the last encoder is used as the "Keys" (K) and "Values" (V) for the subsequent decoder.
Decoders in Transformers:
Decoders also have multiple layers.
In the first self-attention mechanism of the decoder, the target sequence (for example, the partially generated translation) attends to itself. This is analogous to the self-attention in the encoder but has a slight difference. In the decoder's self-attention, future tokens are masked, so they can't be attended to. This ensures that the prediction for a particular word doesn’t depend on future words in the sequence.
In the subsequent multi-head attention mechanism, the "Queries" (Q) come from the decoder's previous layer, and they attend to the "Keys" (K) and "Values" (V) from the encoder's output.
This essentially means that the decoder is using the information from the encoder to help generate the next token in the sequence.
So, your statement is correct: "Value and Key from Encoder with Query from Decoder are combined." In the Transformer's decoder, for every step in its layers, the Queries (Q) from the decoder attend to the Keys (K) and Values (V) from the encoder output."
Unless it’s a decoder only transformer
indeed it is like this as the dot product of the keys and querries construct the relation between the input and the already generated output if noy K and Q where from the encoder it wouldn't capture the relation between the input and already generated output
This is THE BEST transformer video I have encountered.
I have been struggling with this architecture for an eternity now and this is the first time I really understood what's going on in this graphic. Thank you so much for this nice and clear explanation!
What about the architecture made you struggle if I may ask?
Correction: The sine and cosine functions for the positional embedding are applied to the input embedding dimension, not the time steps! oof!
For the readers check out the written version of an illustrated guide to Transformers here towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0
thanx man . i was gonna ask have you written a papper for it like you did for LSTMs
Also the positional embedding in the illustrated guide is incorrect. Its alternating between even and odd
Your video is the best explanation of Transformers I have ever seen
How do we actually split the queries and keys into multiple values for multiheaded attention?
haha thanks micheal, I was like wtf eveything i learned is wrong at 4:18 until i scrolled down
This video marks an end to my search for one place explanation of Transformers. Thanks a lot for putting this up! :)
This tutorial is absolute brilliant, I have to see it again and read the illustrated guide, there are so many infos!! Thank you!!!
Strongly agree!
This 15 minute video is the best explanation of transformers I have found.
Honestly this is the best explanation I've ever seen on transformers and attention
My first read about Micheal Phi was "Stop Installing Tensorflow using pip for performance sake!" in TowardDataScience blog (as I recall you was "Micheal Nguyen" at that time). My first impression was like "oh this guy was good at explanation". Then I read his several blogs, and now here I am. I never knew that you have a channel. You are one of the best educator I've ever known. Thanks so much.
This is by far the best explanation I’ve ever seen on Transformer Networks. Very very well done
This is the best explaination on transformers anywhere on the web
This seems to be one of the best videos on Transformers
This was not just the best transformer explanation, but the best explanation in general that I've ever seen. You know what to abstract for a fluid and clear explanation. Congratulations. You should do more videos like this.
check our video on transformer as well and please provide feedback on our ai4u channel
This is incredible! You explained transformers so well!
Best explanation I've seen so far, thanks so much! 😃
This is the best explanation of transformers models, please keep going on this channel. There are lots of models still need to explain!
The animation of the matrices actually helped a lot! Quality explanation video!
One word excellent , none of the explanation will match this in TH-cam ....stunned
Woah! Exquisite, It's a 15 min video but I spent over an hour taking notes and understanding. You have done a great job, keep it up. Thank you so much! Such explanations are rare. ;)
Now i have two great heroes that explain complex concept using mindblowin visualization, first is 3b1b for complex math topics, then Michael Phi for complex machine learning architecture! Just wow ... salute sir! thank you so much!
If you only knew how relevant this would be 2 years later. Thank you!
In 7:27, In the right the attention wieghts is a 4*4matrix while value matrix is 3*4, a 4*3 matrix for value will be more appropriate
I didn't see any other video explaining this concept this beautiful and clear. Also visual animations helped so much
Thank you 🙏🏼
This is literally the best explanation of Transformers I have ever seen!
I watched many videos, but I'm 100% sure that this video is only the best among those in explaining the transformers
Wow , this was great, I have watched a no of videos on the transformer models, and they have all contributed to my understanding, but this puts everything together so neatly. Amazing, please keep making more such videos.
amazing explanation, honestly this is the first time i understand how Transformers work.
Half of it went through my head. Just beautiful. I'll watch it many more times.. That's how I know the content is gooood.
Your videos are the BEST, I understood RNNs, LSTMs, GRUs and Transformers in less than an hour.
Thankyou.
Thanks for your explanation, very clean and well built in every argument about transformers. I was so lucky to get this video randomly on TH-cam. Good job!
One of the most clearest explanation of Transformers I have ever seen
Best video on transformers on TH-cam, thank you so much
That's absolute genius ! You made it so easy to understand such a mighty concept.
Thank you. Appreciate the kind words 🙏
This transformer tutorial is more than meets the eye
lmao
I used multiple sources to learn about the transformer architecture. Regarding the decoder part, you really helped me understanding what was the input and how the different operations are performed ! Thanks a lot :)
Best video ever on transformers, trust me I tried others, just positional encoding is missing, but rest is gold.
Thank you.
Thank you for this amazing explanation. It really helped after an insufficient explanation from my DL lecture. The prof did not even mention that the final part is a classification over the vocabulary for an nlp task!
This is the best video I have seen about transformers. Very articulate and concise. Great job
This is by far the best explanation that I have seen.
Wow, by far the best explanation. Teaching really is an art where even experts with their blind spots can be very bad at explaining (even if they know it well themselves). Well done. It's really too bad your previous video was a year ago. I really hope you make more videos explaining other deep learning papers such as Faster RCNN etc. Thanks for posting this.
Yes it was through this video that I finally got my head around Transformers. Now I can read through more resources and the Paper itself
Your explanation of q k and v is the thing that finally did it for me, I get it!
I watched a second time not for better understand the video, but to appreciate it. It is very well done and pretty clear. Thank you.
The best video ever for Transformer.
Thank you so much for the step by step explanation. This is a good starting point for ML dummies like me to get a grasp on the transformer model architecture.
Good explanation. What boggles my mind is that this architecture can not only produce reasonable sentences, but there can be some logic going on behind the sequence of sentences as we've seen in chatGPT. It is mind-boggling that there must be some amount of deeper conceptualization represented in the embeddings too! amazing
No, and it's not even understandable how you got to such conclusion.
Best Transformers Explanation I have seen thank you very much, Liked the video and Subscribed !! Keep it up :))
That's most clear and breif explanation about the key idea of 'transformer', especially on how transformer works and how self-attention self attended why self attention is self attention.
Brooo you are seriously my god😭😭🙏🙏...thanks a lot for this video...no one... literally no one could teach me transformer and your video just got drilled into my mind...please make other videos like this for bert gpt xlnet xlm etc etc... I'm really thankful to you
I am not technically skilled on ML and I understood on a high level how this work. I feel so grateful for this video 🙏
That's a very good explanation imo! Thanks for taking the time to produce such a gem.
This is one of the best introductory videos I've seen on this subject. Thank you!
Best illustration on transformer.
Subscribed for many more to come.
This explanation is INCREDIBLE!!!
Quick Q : Say at 5:58, on the left side of the slide, QKV are inputs to linear layers, but on the right it looks like positional encodings are passed to the linear layers and Q,K,V are outputs of the layers. I guess the left is just a typo and right side is correct representation. Can you please help confirm. By the way this is the best explanation i have found, you are just amazing. Thank you so much brother.
You are correct. That was a typo. Positional encodings matrix are passed to each of the three linear layers to produce Q, K and V respecetively.
You did an amazing job explaining the workflow … looked for more similar stuff… please continue … I hope you will be back to help people like me
hands down the best explation for transformer models !
Brilliant explanation with visually intuitive animations ! I rarely comment or subscribe to anything but this time I instantly do both after watching the video. And how coincidental it is that this was uploaded on my birthday. Hope to see more videos from you.
Wow, Wow, Wow... My transformer has no more words to describe.!!!
Hahah thanks!
12:55 A small mistake: K and V should be encoder stack's output, and Q is the first Multi-headed Attention sublayer's output. Still, this guide is really awesome! Thanks for your effort bro!
This is the best explanation on transformers. Thank you so much for the video.
Thanks, I particularly liked that you went into as much detail for the decoder as for the encoder.
Best examplanation I have seen so far. Great job! You destroyed almost all my questions.
Happy to help!
The best explanation so far. Loved the animated illustration.
Favourite video on the topic! I'm reasonably knowledgeable on ML, but the other 5-10 videos I've tried so far all resulted in increased confusion. This is clear.Nice one 👍🏿
Thanks a lot for the intuitive introduction. But at 6:29, shall we transpose the matrix of Q, K here? In the video, multipling a 3x4 matrix and a 4x3 matrix leads to a 3x3 score matrix instead of 4x4 one
Thanks for the effort you put into making the animation on the slide.
This video clears all the doubts about transformers model. please make some more detailed videos on other topics waiting for your reply pls.
Amazing! thank you so much - great quality of the video and content
Wow Michael, this is a superb explanation of the transformer architecture. You even went into detail about the meaning of the Q,K,V vectors and masking concepts which were hard for me to grasp. I bounced around through 3-4 videos about the transformer arch, and for each one I claimed it was the best explanation on the topic. But your video takes the cake and explains it in half the time as the others. Thank you for sharing! Also, great job on the visuals which are on par with 3blue1brown's animations.
Thank you 😃
You literally solved ALL my doubts about transformers
Wow..one of the best videos I have watched on transformers...so simple to grasp. Please make more videos.
Bro you need to post more nowadays, we need you!!
While I still have some questions, this is a pretty good explanation, I mean I actually have an idea of how this works! Gonna watch it like 2 more times.
This is the best explanation of a transfomer or any architecture I have ever watched. Highly impressive how such a complex topic is dismantled into visually appealing and understandable explanations.
Best explanation 😊
you are a legend, took me a week to understand this!
@The A.I. Hacker amazing video. I love the visual approach.
The information density in this video is absolutely insane! Great work!
One of the best explanation of attention mechanism
Awesome explanation - you have great explanation skills. Keep it up !
Thanks, will do!
whoa bro, hats off what a nice explanation and one of the best explanation video for transformers.
This was beautiful.
This was the best explanation out there. You Sir, are a person of highest quality.
Man thanks for this video, reading a paper for newbie is super difficult, but such explanations like you've posted for key, value and query as well as reasoning for masking is very, very helpful. I subscribed to your channel and am looking forward for new stuff.
You have such a sweet and pleasant voice. Thank you, mate, for the good explanation. 😊
Wow. This is some really good explanation! I don't have much NLP background except RNN/LSTM and things before DL (N-gram), but wanted to know more about Attention mechanism for robotics application. (my field) Most other explanation either skimmed over the mathematics, or used NLP specific nomenclature/concepts that made it hard to understand for non-NLP people.
This was some good stuff! Much appreciated and Keep up the good work!
🎯 Key Takeaways for quick navigation:
00:00 🌐 Transformers are revolutionary in natural language processing, breaking NLP records and finding applications in various fields.
02:43 🤖 The core concept of Transformers is the attention mechanism, which allows the model to focus on relevant words.
07:16 🔍 Multi-headed attention in Transformers lets the model associate each word with others, providing a rich context for generating text.
10:06 🧠 Transformers use an encoder-decoder architecture, with the decoder being auto-regressive, generating text word by word.
12:48 🔄 Stacking encoder and decoder layers in Transformers enhances their ability to learn and process different attention representations.
Made with HARPA AI
This is incredible. I've been watching videos and reading papers about transformer and attention for days, this is the best material so far.
Incredible video! I hope you are doing well and find the time to make more, especially with the recent popularity explosion of AI.
Great deep dive into transformers. Helped me understand this architecture.
The explanation about Transformer architecture is clear, and the animation in presentation is really good, it catches my attention :)
This illustrated explanation is just so well done :OOOOO
I'm a novice at Deep neuronal networks and just by looking at the video, I just understood everything !
Completely recommended to understand Transformers :)
Good work :D
In "encoder-decoder attention" layers, the "queries "come from the previous decoder layer,
and the" keys "and "values" come from the output of the encoder. This allows every
position in the decoder to attend over all positions in the input sequence. This mimics the
typical encoder-decoder attention mechanisms in sequence-to-sequence models such as
[38, 2, 9]. -- Attention is all you need
best ever explanation i came across, you made it very useful as well as fun to watch, thanks
A really good and nice video that helped me understand transformer quicker, thanks !
After reading a little more, I think there is small mistake about the roles of the encoder outputs and the decoder first Multi-Headed attention outputs.
The encoder output is supposed to be the Key and the Value and the decoder first Multi-Headed attention output the Query (you say that the encoder output is Q and K and the decoder first Multi-Headed attention is V). A small error that got me to think a little bit too long on how the dimensions could match at the end of the Multi-Headed attention ^^
Thanks again for the video, it was clear as crystal !
I agree. This is confirmed in section 3.2.3 and bullet point 1 in (Vaswani, A. Attention Is All You Need. (2017)).
感谢！ Thank you so much, I've looked up all the resources on the Internet but still messed up with the mechanism. It's really a clear and detailed explanation.