I am the first graduate of my family. It will be near impossible to listen MIT lecture unless there is a initiative like this. Now education/knowledge is open-sourced
They explained the topic in a very easy manner even a guy with no background in ML can understand it. I wish they also had the tutorial for the Practical Sessions.
While stuck at home over the summer, I decided to code up the infrastructure for a bunch of different kinds of AI. I started with perceptron layers, then added recurrent perceptron layers, then LSTMs (that was particularly hard - I had to intensely study at least a dozen research papers to piece it together before all the unstated pieces gelled) , then convolutional nets, then a transformer (encoder/decoder) infrastructure. What I discovered in this process, is that the amount of "intro" level information, as well as "how to AI in python" sort of stuff, completely drowns out the nuts and bolts. In the end, you're stuck reading research papers, which are targeted at an audience that is already a subject matter expert. I suppose universities are supposed to fill that gap, but honestly, this stuff is just not that hard once you decode the language of the field. It's just differential calculus and a bit of optimization theory. Good 3rd year engineering students have the math background, and combine that with some coding skills and you're golden. The other thing I learned in this process, is that convolutional nets and encoder/decoders are just amazing. Even though I wrote every line of code, and understand how and why they work, and train them myself, it feels like magic to watch them work.
everything about this course is phenomenal. It's so good that sometimes I get distracted by thinking about how amazing the course is. The language and the pace used are perfect. The slides are perfect; there's just the right amount of information on the slides so that I don't get overwhelmed by having too much to read while listening, (something I struggle with in my regular classes). It's just so fascinating. Teachers all around the world should take notes. Thank you so much for sharing this with us.
@@DarkLordAli95 Sure, the slides have been published since last year on the 2020 course site: introtodeeplearning.com/2020/. The most recent course iteration contains the 2021 slides (which are also published, but slightly different from this talk).
I would like to say thank you, Alexander and Ava for making it happen! I would make my personal note for Convulational neural network and RNN in 2am and learning from MIT, thank you
Amazing delivery! I have never thought that I would be able to understand any lecture from MIT 100% because of the lack of pre-requisites knowledge. Looking forward to see more videos on GNN, GCN etc..
Thanks! You may also want to check out next week's lecture which will also be on Deep Sequence Modeling but contain a lot of cool new material on Transformers and Attention. The link will be here but it is not published yet: th-cam.com/video/QvkQ1B3FBqA/w-d-xo.html
This one is nice. Nicely fit with my class's slides. The best part for me is that it clearly reveals the concept of timestep computational graph, which I have not seen in previous introduction videos.
Please, correct me if I'm wrong. At 36:18, "uninterupted gradient flow" is essentially due to the fact that operations along the route c0,c1,c2,c3,... has no weights to be updated at all. From what I can see, there are kinds 3 operations along that route: [1] point-wise multiplication (type C x C -> C) [2] point-wise addition (type C x C -> C) [3] copying (type C -> C x C) where C is whatever data type c0,c1,c2,c3 is.
I struggled with this when I came across and ended up resorting to delving into the maths to grasp why LSTMs solve the vanishing/exploding gradient problem. "Uninterrupted gradient flow" does not explain sufficiently. I found this article to be of use (if you can wade through the maths it really explains why LSTMs are better than RNNs for gradient problems) medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577
it's a single RNN cell with more information (and basically the addition of a cell state and also the gates). One unit in the layer, not quite a single neuron.
Hi Ava, Thank you for sharing the knowledge. I have couple of questions: 1. First of all how does back propagation in RNN leads to vanishing gradient but not in case of deep neural network since Mr Amini did not bring it up in the introductory lecture. 2. Secondly you mentioned that having weights as identity matrix will solve the problem of vanishing gradients. So how is that possible. What is the math behind it ? 3. Thirdly, I am confused as to how you are showing the matrix as n*m matrix in the first place. Since the example you took had a sentence which seem to be like a n*1 matrix (where n is the number of words, isn't it?)then how can the weights be a n*m matrix(focusing on the dimensions here). If what is shown is correct then how will you multiple the weight matrix with the example that you have described. 4. And lastly, I could not correlate how relu results in derivative greater than 1. I mean when you use the relu function, then combination of product of weights *x is forced to be a constant value 1 isn't it. So no matter what the value of the dot product of weights and x (and dot product of the state and another weight matrix )is going to be as long as it is greater than zero it will result in a y value of 1, if I understand correctly. However, the value of the function is constant no matter what, then wouldn't that yield the derivative of 0 since the derivative of constant is 0. Maybe I have misunderstood something but you further clarifying this would help certainly. Thanks in Advance!
This series of MIT lectures were very effective and productive, especially in this lockdown and WFH situation. I learned a lot. Thank You !!! Alexander Amini Sir to make it possible.
Its vague in the diagrams... in reality the input to each gate is calculated as: weight * ht-1 + weight * xt + bias, where each gate has their own weight and bias and ht-1 is the previous 'hidden state' and xt is the new input.
@@mattrowlands5751 I thought about it and most probably they represent flow of information with the tanh box representing tanh(Whh*ht-1 + Wxh*xt) i.e the entire function/expression has been abstracted to that box.
How does the vanishing gradients problem is solved by the LSTM? If the LSTM includes sigmoid operations, doesn't that contribute to make gradients smaller?
Absolutely excellent explanation! To further motivate and inspire your great work, where the ball goes next is the holy grail of all models of everything: Quantum Mechanics. You can easily infer the quantum "genome" sequence of all reality with this technology. Again, great presentation and content
Hi all I was just confused about something. The point of RNNs is to preserve the order in a sequence. Let us say our sequence is "I took my cat fora walk". If we use a really large fixed window and make it into a fixed vector to put into a Vanilla feedforward network is the order preserved? I guess what I am really asking is can a feedforward network preserve order?
Great Lecture Series! I just tried to download the slides from your site, but they are unavailable; the site just states that slides and videos are upcoming for 2021 lectures. Do you know where to find the 2020 lecture slides?
I didn't get the "Sharing parameters across the sequence" part! 9:35 what kind of parameters we are talking about here? how they can actually be "shared" across the sequence??
Sharing parameter means the position of a parameter ( in this example: a word) should not be fixed. The parameter (word) can be anywhere in sequence and also present multiple times.
I am getting an error while playing the songs ('C:\Users ame\PycharmProjects\tensors\venv\lib\site-packages\mitdeeplearning\bin\abc2wav' is not recognized as an internal or external command, operable program or batch file.)
If a neural net generates music, after training using a copyrighted music, does the rights of generated music still belong to the copyrighted music creator?
A question related to lab session 2: Why does the last code block show that there are no songs in the text when clearly I generated it using the generate_text function in the previous block?
Can I get master's theses or phd regarding this topic[Time-Series Deep-Learning Classifier for Human Activity Recognition Based On Smartphone Built-in Sensors ]
Hi, I havent finished the video (currently in 16:24), however, I was wondering how do you input a string or text into the RNN. Is it as a string or do you change it to numbers with a tokenize? If you do not use a tokenizer and if you input as a number what method do you use to change strings to numbers?
In RNN you will not enter complete string you will enter each word as a first input , sec world as second input in the form of Vector( you need to convet the word into vector ,use wordtovec library for that).
Thanks Ava....for sharing us informative lectures....Learning a lot from your videos...Can I ask any questions from you people either in slack or somewhere else?
Thank you for sharing knowledge and making it available for all. I am feeling difficulty in understanding the lecture videos, I had never before learnt anything about ML or DL but I really enjoyed the first video. Is there any perquisite information required for this series of lecture?
I am 30 now and I wish to have these sort of contents on my school/college days. Still trying to learn as much as possible. Thank you for your contribution to democratize world class education
if u cant go to MIT, make MIT come to you. Thanks to everybody who made it possible.
I am the first graduate of my family. It will be near impossible to listen MIT lecture unless there is a initiative like this.
Now education/knowledge is open-sourced
They explained the topic in a very easy manner even a guy with no background in ML can understand it.
I wish they also had the tutorial for the Practical Sessions.
github.com/aamini/introtodeeplearning
@@nitroyetevn Thanks you so much!
@@kpr7717 No problem!
unbelievable series!! I'm learning so much from these lectures compared to other youtube vids
CRAZY GOOD quality for each minute of the lecture.
Great stuff!
While stuck at home over the summer, I decided to code up the infrastructure for a bunch of different kinds of AI. I started with perceptron layers, then added recurrent perceptron layers, then LSTMs (that was particularly hard - I had to intensely study at least a dozen research papers to piece it together before all the unstated pieces gelled) , then convolutional nets, then a transformer (encoder/decoder) infrastructure. What I discovered in this process, is that the amount of "intro" level information, as well as "how to AI in python" sort of stuff, completely drowns out the nuts and bolts. In the end, you're stuck reading research papers, which are targeted at an audience that is already a subject matter expert. I suppose universities are supposed to fill that gap, but honestly, this stuff is just not that hard once you decode the language of the field. It's just differential calculus and a bit of optimization theory. Good 3rd year engineering students have the math background, and combine that with some coding skills and you're golden. The other thing I learned in this process, is that convolutional nets and encoder/decoders are just amazing. Even though I wrote every line of code, and understand how and why they work, and train them myself, it feels like magic to watch them work.
Top tier comment.
Excellent presentation! Typo alert in slide shown at around the 18:19 timestamp: The loss corresponding to y_t should be L_t, not L_3.
everything about this course is phenomenal. It's so good that sometimes I get distracted by thinking about how amazing the course is. The language and the pace used are perfect. The slides are perfect; there's just the right amount of information on the slides so that I don't get overwhelmed by having too much to read while listening, (something I struggle with in my regular classes).
It's just so fascinating. Teachers all around the world should take notes.
Thank you so much for sharing this with us.
Thank you!!
@@AAmini Thanks for the reply Alexander. Could you please let me know if you're going to upload these slides anytime soon?
@@DarkLordAli95 Sure, the slides have been published since last year on the 2020 course site: introtodeeplearning.com/2020/. The most recent course iteration contains the 2021 slides (which are also published, but slightly different from this talk).
I can't believe such useful information is available to us for free. Thanks!
I would like to say thank you, Alexander and Ava for making it happen! I would make my personal note for Convulational neural network and RNN in 2am and learning from MIT, thank you
Thanks Alexander and Ava for this free but complete content about NN. Education at its best.
Not from IIT NIT IIIT but I'm learning from MIT thanks for providing such great learning experience.
i loved ava way of explanation such hard topics and break it up in easy pieces that can be understood
Amazing delivery! I have never thought that I would be able to understand any lecture from MIT 100% because of the lack of pre-requisites knowledge. Looking forward to see more videos on GNN, GCN etc..
Thanks! You may also want to check out next week's lecture which will also be on Deep Sequence Modeling but contain a lot of cool new material on Transformers and Attention. The link will be here but it is not published yet: th-cam.com/video/QvkQ1B3FBqA/w-d-xo.html
@@AAmini Thank you very much. Looking forward to it. ....
This one is nice. Nicely fit with my class's slides. The best part for me is that it clearly reveals the concept of timestep computational graph, which I have not seen in previous introduction videos.
This Lectures have been super cool, has clarified a lot of things for me, can't wait for the 2021 Series.
This is nice, leaving behind the mathematics behind, how the vanishing gradient will be overcome by LSTM is nice explanation for RNN.
Please, correct me if I'm wrong. At 36:18, "uninterupted gradient flow" is essentially due to the fact that operations along the route c0,c1,c2,c3,... has no weights to be updated at all. From what I can see, there are kinds 3 operations along that route:
[1] point-wise multiplication (type C x C -> C)
[2] point-wise addition (type C x C -> C)
[3] copying (type C -> C x C)
where C is whatever data type c0,c1,c2,c3 is.
I struggled with this when I came across and ended up resorting to delving into the maths to grasp why LSTMs solve the vanishing/exploding gradient problem. "Uninterrupted gradient flow" does not explain sufficiently.
I found this article to be of use (if you can wade through the maths it really explains why LSTMs are better than RNNs for gradient problems)
medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577
Being a bootcamp I had never thought the materials could be so meaty and well made. I love the videos. Great job!
I don't think she meant "bootcamp" in the literal sense.
Thanks for sharing these lectures as open source. Looking forward to the rest of the boot camp videos and will attempt the lab exercise :)
great material
great course. I didn’t think that while sitting in Almaty I could watch MIT courses
I love this lecture. I am learning so much with this series. Thank you very much for sharing! :)
Hi everyone, At 31:00, Does the block in the middle of the slice represent a single neuron or a layer?
it's a single RNN cell with more information (and basically the addition of a cell state and also the gates). One unit in the layer, not quite a single neuron.
Hi Ava,
Thank you for sharing the knowledge. I have couple of questions:
1. First of all how does back propagation in RNN leads to vanishing gradient but not in case of deep neural network since Mr Amini did not bring it up in the introductory lecture.
2. Secondly you mentioned that having weights as identity matrix will solve the problem of vanishing gradients. So how is that possible. What is the math behind it ?
3. Thirdly, I am confused as to how you are showing the matrix as n*m matrix in the first place. Since the example you took had a sentence which seem to be like a n*1 matrix (where n is the number of words, isn't it?)then how can the weights be a n*m matrix(focusing on the dimensions here). If what is shown is correct then how will you multiple the weight matrix with the example that you have described.
4. And lastly, I could not correlate how relu results in derivative greater than 1. I mean when you use the relu function, then combination of product of weights *x is forced to be a constant value 1 isn't it. So no matter what the value of the dot product of weights and x (and dot product of the state and another weight matrix )is going to be as long as it is greater than zero it will result in a y value of 1, if I understand correctly. However, the value of the function is constant no matter what, then wouldn't that yield the derivative of 0 since the derivative of constant is 0. Maybe I have misunderstood something but you further clarifying this would help certainly.
Thanks in Advance!
This series of MIT lectures were very effective and productive, especially in this lockdown and WFH situation. I learned a lot. Thank You !!! Alexander Amini Sir to make it possible.
Despite not being very good in maths, I am understanding all these lectures.
❤️Love from India❤️
Outstanding educational sharing...I love you MIT...no place like you in this world...or the next
In the LSTM diagram, what operation does the intersection of the wires perform?
Its vague in the diagrams... in reality the input to each gate is calculated as: weight * ht-1 + weight * xt + bias, where each gate has their own weight and bias and ht-1 is the previous 'hidden state' and xt is the new input.
@@mattrowlands5751 I thought about it and most probably they represent flow of information with the tanh box representing
tanh(Whh*ht-1 + Wxh*xt) i.e the entire function/expression has been abstracted to that box.
@@mattrowlands5751 Thanks btw
SARTHAK SHUKLA Yep that is correct. I personally think these diagrams are very misleading and confusing. Best of luck to you my friend.
How does the vanishing gradients problem is solved by the LSTM? If the LSTM includes sigmoid operations, doesn't that contribute to make gradients smaller?
The best one on the TH-cam
Thanks! You should also check out the new 2022 version, it's even better!!!
very nice course. lots of new stuff. looking forward to new releases
at 34:38 what is the use of RHS tanh block in LSTM??please ans if you know.thanks
Great course. Thanks for sharing. Is RNN module equivalent to a simple neuron? I am little confused with terms cell and module
I am having a doubt!
What will be the initial step in the first iteration? Means what is ht-1 in first iteration?
Really amazing video! A ton of info in 45 minutes but in an easy to understand manner!
Absolutely excellent explanation! To further motivate and inspire your great work, where the ball goes next is the holy grail of all models of everything: Quantum Mechanics. You can easily infer the quantum "genome" sequence of all reality with this technology. Again, great presentation and content
Excellent! This has been super useful to get my head around ML terminologies and potentially use them on my ongoing project. Thank you very very much!
Thanks MIT for providing access to such an amazing series of lectures!
great lecture
Hi all I was just confused about something. The point of RNNs is to preserve the order in a sequence. Let us say our sequence is "I took my cat fora walk". If we use a really large fixed window and make it into a fixed vector to put into a Vanilla feedforward network is the order preserved? I guess what I am really asking is can a feedforward network preserve order?
Great Lecture Series! I just tried to download the slides from your site, but they are unavailable; the site just states that slides and videos are upcoming for 2021 lectures. Do you know where to find the 2020 lecture slides?
they're still not there :(
Wonderful lectures, I love them. The things are so much simplified and made easy to learn
Thanx for provide High quality content..... Always wanted to go MIT
This was an incredibly well-organized lecture on recurrent neural networks. Thank you so much for the video!
This is a good lecture and a great instructor :)
I have been eagerly awaiting this video
Thanks from Poland for this great and valuable content :)
I didn't get the "Sharing parameters across the sequence" part! 9:35
what kind of parameters we are talking about here?
how they can actually be "shared" across the sequence??
Sharing parameter means the position of a parameter ( in this example: a word) should not be fixed.
The parameter (word) can be anywhere in sequence and also present multiple times.
I am getting an error while playing the songs ('C:\Users
ame\PycharmProjects\tensors\venv\lib\site-packages\mitdeeplearning\bin\abc2wav' is not recognized as an internal or external command,
operable program or batch file.)
36:20 does this suggest that all loss information is propagated through the internal state of the cell?
Thank you so much for making top class education accessible, especially such an important topic!
great lecture . very informative
If a neural net generates music, after training using a copyrighted music, does the rights of generated music still belong to the copyrighted music creator?
No
A question related to lab session 2:
Why does the last code block show that there are no songs in the text when clearly I generated it using the generate_text function in the previous block?
Any suggested readings
I know I can't go to MIT as I have tons of questions and they only seem to have a couple
How do you choose between different non-linearity functions? In this example here, we used a tanh but could we have used a sigmoid instead?
great courses. Thank you for sharing. you rock! Proud of you in Iran.
Very Good .. Great Lecture .. Thanks
Can I get master's theses or phd regarding this topic[Time-Series Deep-Learning Classifier for Human Activity Recognition Based On Smartphone Built-in Sensors ]
Could somebody explain in detail about shared parameters that she talks about ? Maybe with another example ?
Well done MIT, always one step ahead
I have a question, does feed forward have the vanishing gradient problem, if so how to fix it ?
When we talk about Ht can we say that we are giving some sort of feedback to the perceptron?
Hi, I havent finished the video (currently in 16:24), however, I was wondering how do you input a string or text into the RNN. Is it as a string or do you change it to numbers with a tokenize? If you do not use a tokenizer and if you input as a number what method do you use to change strings to numbers?
In RNN you will not enter complete string you will enter each word as a first input , sec world as second input in the form of Vector( you need to convet the word into vector ,use wordtovec library for that).
What can we do if we r stuck in TODO portion of lab .
what if give solution to this problem, will i get a job?
Thanks for sharing this great lecture!!
wow you explain it amazingly clear and understandable. Thanks MIT for sharing
These lectures are fabulous, but this one is top tier. Excellent. Thank you!
excellent!
for us watching online, is there other material we can use to go along with these lectures?
Can we also have some coding lectures associated with the topics you are discussing here?
I will be glad if u can include it.
if you check their website you can find some coding problems
I would like to attend or get my hands on the lab material. :)
There is lab material in their website's section of 2020.
sir can we get solutions of the lab todo problems
They are all available on the course github repo. Please check the website for more details.
This was really helpful!
Are there any hands-on projects for Applications of RNN in music generation?
Thanks Ava....for sharing us informative lectures....Learning a lot from your videos...Can I ask any questions from you people either in slack or somewhere else?
Thank you for sharing knowledge and making it available for all. I am feeling difficulty in understanding the lecture videos, I had never before learnt anything about ML or DL but I really enjoyed the first video.
Is there any perquisite information required for this series of lecture?
I'd say basic linear algebra, calculus and programming
The lab parts are really useful thanks alot
Amazing coure! Thank you!
Anyone knows where are the projects?
so how is c_t different from h_t? How is this "cell state" different from h_t? And why are both of those used?
How exactly is counts calculated???
I am learning new things!
Pre requisite to learn from this course?
Thank you very much, for giving knowledge from another dimension, great course. :)
amazing explained in only 45 minutes. Why don't they teach like this at my university?
Why I feel like RNN is quite similar to Kalman Filter...
Where to get the data for lab?
I am 30 now and I wish to have these sort of contents on my school/college days. Still trying to learn as much as possible. Thank you for your contribution to democratize world class education
THE SLIDES ARE NOT AVAILABLE IN THAT LINK.
Great lecture Ava. Is there any chance I can consult with you on a project I’m working on? You can choose the hourly tutor rate?
Where can I get lab tutorials?
How to get the relevant lab content?
Visit course website
Thanks very much well understanding
Amazing series, thank you! Would love to see more from MIT! (:
Great lecture from MIT!!
Thank you @ava soleimany
thankyou for this series of lectures