K is a matrix representing the T previously seen words and V is the matrix representing the full dictionary of words of the target language, right? But what are K and V exactly? What values do these matrices hold? Are they learned?
Where experience and 'thinking outside the box' can beat education in some cases. He should be getting an 'honorary' bachelor degree, if he hasn't already.
Excuse me? that's fake news! Even his linked in profile says Duke 1998 (yes it's the same noam shazeer from this exact same paper)... "Noam Shazeer is an Engineer at Google. He graduated from Duke in 1998 with a double major in Mathematics and Computer Science"
I think he meant that Noam doesn't have a PhD. Noam does have a bachelors degree, but he started working at Google pretty soon after graduating (literally decades ago) and has contributed to many important Google technologies in his time there. Noam was a Google old-timer back when I started working there in 2005.
"He didn't put a trophy into the suitcase because it was too small." is an ambiguous statement. "it" could refer to either the trophy or the suitcase. It seems like the answer is mainly decided on probability from past experience, rather than the intended (ambiguous) meaning, similar to a survey or experiment with too small a sample size. It is also possible that he didn't want to put a too small a trophy into the suitcase in case it ended up being jostled about too much, and became damaged; although that is a less likely, but still a possible explanation and would need a thought process to come to that conclusion, or some further context, to clarify the intended meaning. People on the Autistic spectrum (HFA / Asperger's) have that same problem when phrasing thoughts (ambiguous meaning), and are often misunderstood because of it. When a statement has two (or more) possible meanings, then it's probably unfair to judge the performance of a system in 'getting the answer right' as there isn't a definite correct answer to begin with, just a more likely one. A word for word translation, with grammatical correction applied would probably achieve a better result in a case like this. Google translate seems to somewhat agree. Original: He didn't put a trophy into the suitcase because it was too small Google translate: Er hat keine Trophäe in den Koffer gesteckt, weil er zu klein war. Back to english: He did not put a trophy in his suitcase because he was too small. Word for word translation (incorrect, but probably still understandable if you speak German): er nicht stellen ein Trophäe in das koffer da es was auch klein. Google translate of word to word to english (much better but still wrong - where did the 'also' come from?):he does not put a trophy in the suitcase as it is also small.
In VIT, it is clearly stated that a "small dataset" like imagenet doesnt show promising results but a larger dataset like the jft gives amazing results, so this maybe a start, but it is far from perfection. Btw, I am not contradicting your statement. 😁. and also JFT is not an open source dataset(yet)
Strange thing, he mention "attention" term before explaining what it is. What was EXACT meaning of this Query Key Value magic ??? I suspect speakers just copy thoughts of another people mechanically, not understaning real meaning of operations !
...comes from Google - Check. ...TensorFlow T-shirt - Check. Most viewers therefore rate this lecture highly - Check. This is very hand-wavy throughout with relatively no rigor shown. There are many lectures/presentations online which actually explain the nuts and bolts and wider use cases of Attention mechanisms. Maybe the title of this video should be something else, like "Our group's success with one use case (language translation) of Attention." Frankly, the drive-by treatment of the technical details of language translation case was almost terrible and should have probably been omitted.
Most I gather from this talk is that "attention" is a pretty terrible term. Something like "fuzzy lookup" or "matching" or "mapping" would have been much more descriptive, but oh well, which researcher needs to think about terminology before unleashing it on the world.
I also couldn't understand.. The architecture diagram is not very intuitive, and I was expecting some elaborations.. However, the explanation was not clear...
His communication skill are like a runner who is tripping over his shoelaces. Unfortunately, it seems to be quite a common ailment of even "brilliant" coders (or shall I say, scientists) that they can't explain their ideas to others clearly using natural language. It's like they have no model of someone else's knowledge and take so many things for granted that their attempts at "explanation" just sound like gobbledygook to those who expect to be taught something. That's why we have technical writers, teachers, popular science books etc.
@Yancy Stevens Yes, to communicate you have to model in your head whoever you are communicating to, what they know, don't know, and foremost what they want to know. Otherwise it's just a fail, no matter how much knowledge you have.
![SAFEPLANET - ดาวเคราะห์ [ THE PLANET ]](http://i.ytimg.com/vi/4X1XL3I_k3c/mqdefault.jpg)
![[#2024MAMA] BIGBANG (빅뱅) - 뱅뱅뱅 (BANG BANG BANG) + FANTASTIC BABY | Mnet 241123 방송](http://i.ytimg.com/vi/EQsYfiPR9uM/mqdefault.jpg)
![The moment we stopped understanding AI [AlexNet]](/img/n.gif)
Starting @ 15:45, in well under 2 minutes, attention explained! Only a true master can do it. Love.
K is a matrix representing the T previously seen words and V is the matrix representing the full dictionary of words of the target language, right? But what are K and V exactly? What values do these matrices hold? Are they learned?
The passion that he transmits is priceless
I love how excited he is.
This is the best explanation of attention I have seen so far! And I have been looking :)
Bcoz, he is one of the co-author of the revolutionary paper which introduced it
Great lecture. Best explanation of attention in just a few words.
Great presentation, he's having fun explaining the bits....great camera work- it was fun watching a moving cam than a boring still view.
I just wish he hadn't stood right in front of what he was trying to show people, but I love his passion for explaining what he's talking about.
very interesting of him to call deep learning a trade :)
Thanks for sharing! It'd be great if the video could pay more attention to the slides though.
Thank you for your comment, Franck! You can download the slides here: picampus-school.com/open-day-2017-presentations-download/
@@pischool6210 perfect, thanks!
Great talk, Łukasz.
Nice explain about position and multihead ...
"there is this guy, he never got his bachelor but he wrote most of these papers" - appreciation
Where experience and 'thinking outside the box' can beat education in some cases. He should be getting an 'honorary' bachelor degree, if he hasn't already.
Excuse me? that's fake news! Even his linked in profile says Duke 1998 (yes it's the same noam shazeer from this exact same paper)... "Noam Shazeer is an Engineer at Google. He graduated from Duke in 1998 with a double major in Mathematics and Computer Science"
I think he meant that Noam doesn't have a PhD. Noam does have a bachelors degree, but he started working at Google pretty soon after graduating (literally decades ago) and has contributed to many important Google technologies in his time there. Noam was a Google old-timer back when I started working there in 2005.
Great content in this video. Would love if you had the multi-headed devil covered! Though, great video to get the overall view..
Thank you !
very nice lecture ...enjoyed it lot.
Great talk. Something very useful.
They should invent a device that can always tell the time of day when the user wants.
nothing beats GPT 2 TL;DR summarization trick
Nice explanation!!!!
where is the library he talks about to get the details of training the DL "right"?
31:55
@@kvsnoufal is there one for pytorch?
ChatGPT gave me link to this video)
"He didn't put a trophy into the suitcase because it was too small." is an ambiguous statement. "it" could refer to either the trophy or the suitcase. It seems like the answer is mainly decided on probability from past experience, rather than the intended (ambiguous) meaning, similar to a survey or experiment with too small a sample size. It is also possible that he didn't want to put a too small a trophy into the suitcase in case it ended up being jostled about too much, and became damaged; although that is a less likely, but still a possible explanation and would need a thought process to come to that conclusion, or some further context, to clarify the intended meaning. People on the Autistic spectrum (HFA / Asperger's) have that same problem when phrasing thoughts (ambiguous meaning), and are often misunderstood because of it. When a statement has two (or more) possible meanings, then it's probably unfair to judge the performance of a system in 'getting the answer right' as there isn't a definite correct answer to begin with, just a more likely one.
A word for word translation, with grammatical correction applied would probably achieve a better result in a case like this. Google translate seems to somewhat agree.
Original: He didn't put a trophy into the suitcase because it was too small
Google translate: Er hat keine Trophäe in den Koffer gesteckt, weil er zu klein war.
Back to english: He did not put a trophy in his suitcase because he was too small.
Word for word translation (incorrect, but probably still understandable if you speak German): er nicht stellen ein Trophäe in das koffer da es was auch klein.
Google translate of word to word to english (much better but still wrong - where did the 'also' come from?):he does not put a trophy in the suitcase as it is also small.
Nice lecture. I enjoyed it
does embedding gets trained or key or query or value gets trained i am confused. please help
math majors/ graduate math students skip to 15:36
Here I get it.
Thank u
Best expaination!!
Thanks buddy
Is there a way to get the slides?
You can download them here: picampus-school.com/open-day-2017-presentations-download/
Pi School I have just checked and it seems the slides linked to the presentation is "tensorflow workshop", can you please double-check?
Dear Vladimir, have a look here: drive.google.com/file/d/0B8BcJC1Y8XqobGNBYVpteDdFOWc/view
great explanation !
@27:14 multitasking
What model is that at the beginning? Can I somehow get the machine produced texts which where shown at the beginning of the presentation?
"
Can I somehow get the machine produced texts which where shown at the beginning of the presentation?
47:55 "We tried it on images it didn't work so well". 2020, Visual Transformer: am I a joke to you?
In VIT, it is clearly stated that a "small dataset" like imagenet doesnt show promising results but a larger dataset like the jft gives amazing results, so this maybe a start, but it is far from perfection. Btw, I am not contradicting your statement. 😁. and also JFT is not an open source dataset(yet)
@@souhamghosh8714 True Google folks ^^
“Hi, I am from google, you know what i got, TPUs..more than you can imagine”😂
Strange thing, he mention "attention" term before explaining what it is. What was EXACT meaning of this Query Key Value magic ??? I suspect speakers just copy thoughts of another people mechanically, not understaning real meaning of operations !
Are the talks he is referring to (as the previous talks) available on TH-cam?
Hello! Sure. You can find all the Masterclasses from our Open Day here 👉th-cam.com/play/PLU3hjga27ZUiuL8V0CVlidBK27CDxWf-F.html
Is there any way i could get the ppts for reference?
Hi, sure! You can download it here: picampus-school.com/open-day-2017-presentations-download/
@@pischool6210 thankyou so much!!!!
Transformers learned translation without language dependency O_o
the video image is too poor, you need to fix it more
Slides?
drive.google.com/file/d/0B8BcJC1Y8XqobGNBYVpteDdFOWc/view
Tarea 6
...comes from Google - Check. ...TensorFlow T-shirt - Check. Most viewers therefore rate this lecture highly - Check.
This is very hand-wavy throughout with relatively no rigor shown. There are many lectures/presentations online which actually explain the nuts and bolts and wider use cases of Attention mechanisms. Maybe the title of this video should be something else, like "Our group's success with one use case (language translation) of Attention." Frankly, the drive-by treatment of the technical details of language translation case was almost terrible and should have probably been omitted.
which lectures do you like that explain attention mechanisms and their wider use?
is this guy one of the father's of modern AI? is he a primary reason for chatGPT?
yes
good video but his and the camera placements are subobtimal
Most I gather from this talk is that "attention" is a pretty terrible term. Something like "fuzzy lookup" or "matching" or "mapping" would have been much more descriptive, but oh well, which researcher needs to think about terminology before unleashing it on the world.
can't believe this guy was one of the authors of Transformer. He just can not explain what he was doing!
I thought it was a really insightful talk. I'm preparing a talk about Transformer myself, and this talk was super inspiring :)
I also couldn't understand.. The architecture diagram is not very intuitive, and I was expecting some elaborations.. However, the explanation was not clear...
If you don’t already know something about attention then it’s impossible to follow the presentation. Everything is explained very poorly...
His communication skill are like a runner who is tripping over his shoelaces.
Unfortunately, it seems to be quite a common ailment of even "brilliant" coders (or shall I say, scientists) that they can't explain their ideas to others clearly using natural language. It's like they have no model of someone else's knowledge and take so many things for granted that their attempts at "explanation" just sound like gobbledygook to those who expect to be taught something. That's why we have technical writers, teachers, popular science books etc.
@Yancy Stevens Yes, to communicate you have to model in your head whoever you are communicating to, what they know, don't know, and foremost what they want to know. Otherwise it's just a fail, no matter how much knowledge you have.