You are probably the one TH-camr among several blog poster/TH-camr/Researchers who clearly understands what CapsuleNet is and how dynamic routing works. Great work on the slides.
Brilliant. Just brilliant. One of the best videos not only about capsule nets, but also about deep learning. The way you explain this architecture is amazing. Really, really good job.
Well actually the routing weights are *not* learned: the raw routing weights actually get reset to 0 for every new image, both during training and for new predictions. The routing by agreement algorithm is a *routing* algorithm, *not* a learning algorithm. It decides where to send the signal to, based on the amount of agreement it measures, and that's it. The learned parameters are the convolutional layers' weights, and the W_ij matrices, and those are learned using regular backpropagation. I hope this helps.
Great question! Imagine if, by chance, all the images of the digit 5 in the training set had a particular set of 3 pixels that were black, and these 3 pixels were not all black for other digits. During training, the CapsNet would certainly detect this pattern and use it for classification. Without a decoder, all the CapsNet would need to classify the digit 5 would be the value of these 3 pixels. It would perform great on the training set, but it would probably not work well on new instances (assuming the particular pattern was a result of chance). In other words, the CapsNet would overfit the training data. By adding the decoder on top of the digit layer, we are encouraging the CapsNet to propagate all the information required to reconstruct the digit all the way from the inputs up to the digit layer. This forces the CapsNet to encode the whole image up to the digit layer, not just the few pixels that it thinks it needs, and thus it is more likely to make its final classification decision based on these higher level representations of the input image, rather than on just a few pixel values. Another way to explain this is to say that we are imposing a constraint on the CapsNet, thus we are limiting its degrees of freedom during training, and this is the essence of regularization. By reducing the degrees of freedom of a learning algorithm, we reduce the chances that it will overfit the training data (but if we go too far, it may underfit the data). Hope this makes sense.
Thank you so much for subtitling the video. For the hearing impaired such as me, unsubtitled videos are hard to follow and for some of us, basically useless. Not only is the video subtitled, the explanation is amazingly straightforward and intuitive. I definitely would not have managed to understand this paper without you.
In all my questions, I totally forgot to congratulate you on the job done. Great explanation man! Seriously, this video is the most insightful one around the web.👍
Please keep making more videos. Having read the paper a few times, I was still confused in parts. Your lucid explanation cleared up most of them. Hats off!
i went and bought your book after watching this video.. you explain so clearly here that i had no doubts the book would be as clear and awesome. Thank you.
This is one of the most pedagogic videos in deep learning. Explaining the CapsNets in a very simple and efficient language is indeed an art. Thank you @Aurélien Géron
A clear explanation without being too technical, I find it a good intro to capsule networks before jumping into the article and/or the codes... thank you for taking your time and preparing this video!
You wrote this book? You are technically my mentor. This book gave me the exact kickstart from theoretical aspects to the hands-on programming and It was so easy and fun to comprehand that I wish I'd started my journey with it directly in the first place! Amazing work, book and video.
Thanks for your very kind words Saransh, you are very encouraging! :) My best reward for this work is when I meet people who tell me what cool things they built using Machine Learning. It's always different and exciting. Glad I could help!
You're a fantastic communicator! I think anybody who watches your videos or reads your content appreciates the work you put in. I hope you continue making great content :)
All, just want to add that the book is brilliant as well. Very easy read and as the title says, hands-on (if you choose to). Highly recommend that people buy it! Aurelien, keep the videos coming,you are a natural.
This has to be one of the best explained tutorials on any ML subject I've ever watched! Had to pause, revisit certain points in the video, but understood CapsNets perfectly. Hope to discover other tutorials by you!
I have to say that i was quite shocked when i heard "its my first youtube video", the quality of this video is as if you have been making videos for past 15 years. Please do make more videos
By far the best explanation of capsule networks I have seen. kudos for simplifying it so well. Please keep making more videos. Will be buying your book; your ability to simplify concepts and explain meaningfully is more than amply demonstrated by just this one video.
Aurélien Géron this is amazing. By all standards I think you are supremely gifted. And your approach as well is quite unique i.e. taking papers and making them understandable to everyone. Keep going man!
Thanks Selase, I appreciate it! There are many other channels that summarize papers these days, such as "Two minute papers", Siraj Raval's channel, "CrashCourse" and so on, and there are a lot of great channels to learn Machine Learning, whether basic or advanced, such as on Hugo Larochelle's channel, Standford's channel, MIT OpenCourseWare, not to mention of course Andrew Ng's Coursera classes, Geoffrey Hinton's classes and many more. I'd love to stand on the shoulders of these giants, but I have yet to reach the top of there toes. :)
I absolutely love your book. This video sure helps a lot. Please do make more videos and release more books on ML and DL, and I will be sure to be the first one to support them.
Yes, please make more videos! I'm currently learning and working through your book. The way you explain things is very helpful for me, videos on new papers and techniques would be very much appreciated. Good work! :)
This is one of the, if not THE best video explanations of a concept/technique i’ve watched.... and i am constantly watching such videos. I look forward to more of your content :)
Thanks a lot Daniel, such encouraging feedback definitely motivates me to create more videos. I'm currently recording a second video to dive into the implementation of capsule networks. Then I have a couple ideas for more videos, but I hope I'll have time to work on them.
Awesome, sounds good :) If you do as good a job as you did with this one, I imagine even videos introducing the well established techniques (basics of CNNs, RNNs etc ) would gain a lot of traction too....
Thank you for the video Aurélien! Please, make more of them. Your explanations are very clear and easy to follow. Actually your book, Hands-on ML, was the first book about Deep Learning that didn't make me want to kill myself while reading it xD
Hi Aurélien, fantastic video! Your book has been extremely valuable in moving from theoretical understanding and basic implementations of ML/DL algorithms to real-world application and pipelines. Looking forward to more videos in future (subscribed).
Awesome explanation. Would be nice if we have a video explanation about Matrix Capsules with EM routing paper. I'm currently in second re-read but still having confused about some concepts presented in the paper.
Excellent video, thanks a lot for the clear explanation and the examples, it's a great complement to the paper. I have subscribed and I am looking forward for other paper reviews!
Thanks Horace, any particular topic you would like more information on? This is my very first video, but I plan on making more, so I'll be looking for good topics to cover.
I think there's a severe lack of material past the beginner's level that help people learn more than just "how do I apply this in code". I think blogs/publications like www.inference.vc/ or distill.pub/ are on this level. I think the best thing to do is choose topics that you feel like you have the most insight/intuition towards. However, here's a short list of topics I think would be cool to have a video on: A3C, policy gradients (well, RL in general), generative adversarial networks, maybe some of the theoretical work on generalization/local optima (eg: why neural nets don't get stuck in bad local minima, why they generalize, etc.), variational autoencoders. Great work, and I'm sure you've already seen Hinton's comment on your video! :)
Nice suggestions, thanks. I'm a big fan of distill.pub, which I discovered through Christopher Olah's fantastic blog. I'm not sure I can reach that level (I'm just a practitioner, not a researcher), but I can always try. :)
Why is there a need of a convolutional layer for the first layer of the CapsNet? What is essentially done is that the output feature map of the convolutional layer is converted into small vectors by reshaping and then routed to the capsules in the next layer. Why can't we consider the input image as a raw pixels' feature map and route that information directly to a layer of capsules (the convolutional capsules)?
Interesting question. If I understand correctly, there would be one primary capsule per pixel, and each primary capsule would simply output a vector equal to the squashed pixel value (a 3D vector for an RGB image). That might actually work, I encourage you to try! I'm guessing the authors chose to start with two convolutional layers for performance reasons. Another possible reason why they didn't choose this option might be that the routing by agreement algorithm pushes capsules to route their output to just one (or very few) capsules in the next layer. Suppose there's a black pixel in the image: it might be a part of just about any possible object, but routing by agreement will push this pixel's capsule to route its output to just one (or a few) higher-level capsules. There's a risk that the correct one won't be in the list. It seems to me that it might be a good idea to narrow down the options before routing by agreement kicks in. That's what the two convolutional layers do. That said, I'm just guessing, here: you could try your idea and publish a paper if it works well! :)
Well, firstly thanks for such a quick reply. No, I am not saying on the pixel level. We can create patches of the image that the convolutional capsule takes as an input. So, it is more like a small window of fixed receptive field let's say (9 x 9) . We can take all those pixels and create the high dimensional vector for that window. My intuition behind doing this is quite similar to the content based attention used in Seq-2-Seq models. It's like we are assigning only a small part of the image a capsule (well technically, to multiple capsules with different co-efficients.). As you said, let's say a part of the image that contains an eye (for the facial dataset), it could be contributing to multiple capsules in the next layers. But, we don't have to worry about it. The capsules that have higher scalar product with this eye only, will receive the eye patch. I loved concept of the paper, there was this only thing that bugged me a bit. The architecture is not homogeneous. first layer is convolutional and subsequent layers are capsules. Anyways, this is still an idea that I have. Will definitely work on it. Thank you for your encouragement. Btw, what's your github? I would love to follow you. mine -> github.com/akanimax
You're very welcome. Oh, right, I see what you want to do. To go from the 9x9 patch to the high-dimensional vector for that patch, would you simply flatten the patch to get an 81D vector, then squash that vector? If so, then the patch's brightness would determine the output vector's length, which represents the estimated probability in a capsule network. So the primary capsules could never be confident about the presence of a dark feature. This seems problematic. To avoid this, we would probably have to apply a transformation matrix to the 9x9 patch before we flatten and squash it, so that the brightness information ends up being a part of the orientation of the output vector (it's a pose parameter), rather than defining its length. A convolutional layer would be a convenient way to do this.
Robin Richtsfeld Thanks for the explanation. Now it is clear to me why the lower layer needs to be convolutional. Still I'd like to see what happens if we directly start with the primary caps layer instead of the convolutional layer.
Aurélien Géron "So the primary capsule could never be confident about the presence of a dark feature". I don't know, but I am wondering: doesn't it make the architecture more like humans? Even we can't be sure of dark features. Btw, thank you for your explanation. I now understand why we need the convolutional layer at the beginning. If I understand correctly, the only difference between the primary caps layer and the first convolutional layer's architectures is that the output of the primary caps layer is dynamically routed to the digitCaps layer while the output of the convolutional layer is fed as it is (with a stride of 2). Just a last question: why isn't the output of convolutional layer not routed to the primary caps layer? Wouldn't that make the convolutional layer another primary caps layer? I have tried to find an explanation of this a lot, but couldn't. From where I am, there are no mentors. It is my sincere request if you could please address this last question. I promise this is the last one :). Thanks!
This explanation is the best one that I have found! Thank you so much for making this video. Definitely looking forward to the next one about implementation of capsule networks. Do you think maybe you could also do one on the mathematical intuition behind the part where the weights and coupling coefficients interact to compute the prediction vector and then ultimately the activity vector?
Hi Sidrah, thanks for your kind words. I actually uploaded a video about the TensorFlow implementation of CapsNets just a few hours ago: th-cam.com/video/2Kawrd5szHE/w-d-xo.html I like the idea of a video about the math intuitions behind ML! In the meantime, you may want to take a look at my Jupyter notebook about Linear Algebra: nbviewer.jupyter.org/github/ageron/handson-ml/blob/master/math_linear_algebra.ipynb It explains how a matrix multiplication can perform all sorts of linear transformations, such as a rotation, resize, skew, and so on. Hope this helps. :)
I have a question. Pardon me if this is a stupid question, but in case my dad won't be able to answer it I will post what I asked him. This is what I sent my dad; perhaps you'd be better at explaining it: I have questions. When he is talking about the primary capsules to create the house and boat capsules, does he say that he uses both the rectangle capsule then the triangle capsule as the primary capsules in order to create the potential position of the other shape? What use is this when both structure the same shape? When it comes to the crowding, do you need to have trained the network to recognize a house capsule and a boat capsule to look for those first so that it becomes the more logical explanation of the image (as opposed to the inverted house with random rectangle and triangle), or will the system be able to learn what a boat capsule and house capsule is with no prior training and be able to separate the two figures from that shape?
Thank you such lucid explanation. Nowadays, it seems that capsule networks gain less attention than 2 years ago. Is this true or just my false perception?
Hello, the margin loss equation at 15:55 is T_k max(0, m_p - ||v||^2) + lambda (1 - T_k) max(0, ||v||^2 - m_m) but the "square" is outside the max functions on the paper (v2, arxiv.org/abs/1710.09829) which one is correct?
Hi, good catch, this is indeed an error in my video, the square should be outside the max. My apologies. I'll add an annotation at that point in the video. Thank you very much for spotting this error.
Oh nooooo! TH-cam dropped video annotations , I can't add an error message. I guess the only option is to add a warning in the video description. I'll soon upload another video to go into the details of the implementation, and I'll make it clear at that point. Thanks again for catching the error.
Hi Aurélien, this video is amazingly clear. About being less sensitive to pose estimation and scale... Humans/Toddlers do learn with an integrated camera viewpoint estimator (an integrated accelerometer called the internal hear... which leads to pitch and roll) a depth map (stereovision with 2 eyes which can lead to quickly determining the size) ... and they're able to walk/interact around the object.... Don't you think these priors could help to boost what's necessary to push capsnets to the next level ? For instance when i'm looking at the boat from the sea, i know it's far away (disparity ...) and i see it moving... and the horizon is not tilted because my head is standing vertically. It is much different from a boat drawing printed on a book that could be tilted because the book is tilted... (i know it's a planar representation using disparity again + not moving). So we can let computers try to learn by themselves that extra complexity of not knowing how the pictures were taken but maybe helping them could lead to better performances.
Geoff Hinton on this video: "This is an amazingly good video. I wish I could explain capsules that well."
If anybody else was curious where he posted it: www.reddit.com/r/MachineLearning/comments/7ew7ba/d_capsule_networks_capsnets_tutorial/dq8yc9p/
Do you know how to get the slides in the video？if you kwon ,please give me the link, thank you!
Hi Xiaofei, I put the link to the slides at the end of the video description. Enjoy!
Yes Yes Yes!
haha true
You are probably the one TH-camr among several blog poster/TH-camr/Researchers who clearly understands what CapsuleNet is and how dynamic routing works. Great work on the slides.
Brilliant. Just brilliant. One of the best videos not only about capsule nets, but also about deep learning. The way you explain this architecture is amazing. Really, really good job.
Wow, thanks a lot, you are very encouraging. :)
so only C_ij are learned using routing algo...other parameters backprop using total loss?
Well actually the routing weights are *not* learned: the raw routing weights actually get reset to 0 for every new image, both during training and for new predictions. The routing by agreement algorithm is a *routing* algorithm, *not* a learning algorithm. It decides where to send the signal to, based on the amount of agreement it measures, and that's it. The learned parameters are the convolutional layers' weights, and the W_ij matrices, and those are learned using regular backpropagation. I hope this helps.
Thank ..last question...why the decoder helps to avoid overfitting and to generalize?
Great question! Imagine if, by chance, all the images of the digit 5 in the training set had a particular set of 3 pixels that were black, and these 3 pixels were not all black for other digits. During training, the CapsNet would certainly detect this pattern and use it for classification. Without a decoder, all the CapsNet would need to classify the digit 5 would be the value of these 3 pixels. It would perform great on the training set, but it would probably not work well on new instances (assuming the particular pattern was a result of chance). In other words, the CapsNet would overfit the training data.
By adding the decoder on top of the digit layer, we are encouraging the CapsNet to propagate all the information required to reconstruct the digit all the way from the inputs up to the digit layer. This forces the CapsNet to encode the whole image up to the digit layer, not just the few pixels that it thinks it needs, and thus it is more likely to make its final classification decision based on these higher level representations of the input image, rather than on just a few pixel values.
Another way to explain this is to say that we are imposing a constraint on the CapsNet, thus we are limiting its degrees of freedom during training, and this is the essence of regularization. By reducing the degrees of freedom of a learning algorithm, we reduce the chances that it will overfit the training data (but if we go too far, it may underfit the data).
Hope this makes sense.
Thank you so much for subtitling the video. For the hearing impaired such as me, unsubtitled videos are hard to follow and for some of us, basically useless. Not only is the video subtitled, the explanation is amazingly straightforward and intuitive. I definitely would not have managed to understand this paper without you.
hstrainer I'm glad it was useful to you! I also added subtitles to my other videos, in case you're interested. :)
In all my questions, I totally forgot to congratulate you on the job done.
Great explanation man! Seriously, this video is the most insightful one around the web.👍
I was expecting "just another capsule tutorial" but I was wrong. Good job
Spent an entire day trying to understand hinton's paper. Then i came upon this video, which explained the architecture amazingly clearly. Great job!
What an exceptional explanation. Thank you.
You're very welcome, Joseph, I'm glad you liked the video. :)
Please keep making more videos. Having read the paper a few times, I was still confused in parts. Your lucid explanation cleared up most of them. Hats off!
i went and bought your book after watching this video.. you explain so clearly here that i had no doubts the book would be as clear and awesome. Thank you.
Thanks, I hope you find it useful! :)
This is one of the most pedagogic videos in deep learning. Explaining the CapsNets in a very simple and efficient language is indeed an art. Thank you @Aurélien Géron
A clear explanation without being too technical, I find it a good intro to capsule networks before jumping into the article and/or the codes... thank you for taking your time and preparing this video!
You wrote this book? You are technically my mentor. This book gave me the exact kickstart from theoretical aspects to the hands-on programming and It was so easy and fun to comprehand that I wish I'd started my journey with it directly in the first place! Amazing work, book and video.
Thanks for your very kind words Saransh, you are very encouraging! :) My best reward for this work is when I meet people who tell me what cool things they built using Machine Learning. It's always different and exciting. Glad I could help!
If, as you said this was your first video on TH-cam, then there is a great "learning from you" future ahead of us. Brilliant, please keep going!
You're a fantastic communicator! I think anybody who watches your videos or reads your content appreciates the work you put in. I hope you continue making great content :)
All, just want to add that the book is brilliant as well. Very easy read and as the title says, hands-on (if you choose to). Highly recommend that people buy it!
Aurelien, keep the videos coming,you are a natural.
Wow! This is just marvelous! It's like adding particle filters to neural networks. Brilliant idea!
Well done, as someone with only a minimal background in mathematics and A.I., this was very understandable, you truly have a gift!
Thanks Nick, I appreciate it!
This has to be one of the best explained tutorials on any ML subject I've ever watched! Had to pause, revisit certain points in the video, but understood CapsNets perfectly. Hope to discover other tutorials by you!
Despite not knowing much about machine learning, I'm actually able to follow much of what this guy is saying. He's a very good teacher!
I have to say that i was quite shocked when i heard "its my first youtube video", the quality of this video is as if you have been making videos for past 15 years. Please do make more videos
By far the best explanation of capsule networks I have seen. kudos for simplifying it so well. Please keep making more videos. Will be buying your book; your ability to simplify concepts and explain meaningfully is more than amply demonstrated by just this one video.
Aurélien Géron this is amazing. By all standards I think you are supremely gifted. And your approach as well is quite unique i.e. taking papers and making them understandable to everyone. Keep going man!
Thanks Selase, I appreciate it! There are many other channels that summarize papers these days, such as "Two minute papers", Siraj Raval's channel, "CrashCourse" and so on, and there are a lot of great channels to learn Machine Learning, whether basic or advanced, such as on Hugo Larochelle's channel, Standford's channel, MIT OpenCourseWare, not to mention of course Andrew Ng's Coursera classes, Geoffrey Hinton's classes and many more. I'd love to stand on the shoulders of these giants, but I have yet to reach the top of there toes. :)
Well, you make a good point and I do understand. Siraj Raval's is a great channel as well. Thank you very much.
Hinton about this video: « I wish I could explain capsules that well ». Kudos!
(From his last Reddit comment)
Finally, I found an intuitive explanation of the capsule idea. Thank you for doing a great job of distilling the idea.
I absolutely love your book. This video sure helps a lot. Please do make more videos and release more books on ML and DL, and I will be sure to be the first one to support them.
Yes, please make more videos! I'm currently learning and working through your book. The way you explain things is very helpful for me, videos on new papers and techniques would be very much appreciated. Good work! :)
This is one of the, if not THE best video explanations of a concept/technique i’ve watched.... and i am constantly watching such videos.
I look forward to more of your content :)
Thanks a lot Daniel, such encouraging feedback definitely motivates me to create more videos. I'm currently recording a second video to dive into the implementation of capsule networks. Then I have a couple ideas for more videos, but I hope I'll have time to work on them.
Awesome, sounds good :) If you do as good a job as you did with this one, I imagine even videos introducing the well established techniques (basics of CNNs, RNNs etc ) would gain a lot of traction too....
A very clear presentation of CapsNets, Bravo Aurélien.
I'm still pretty new to this but your explanation made a lot of sense. Please keep making these videos! They're excellent.
Thanks Thomas, these comments are certainly motivating me to make more! :)
Great explanation, clean slides and fantastic English ! Thanks !
Great intro ! Sound, slow enough to follow and good examples to convey the idea ! Thx
there couldn't be any better explanation than this, although their paper was not that much explicit still, you have explained it so clearly
best description of capsNet even hinton could not do this :)
wow! I think I ve found the best video explaining CapsNet
Excellent. I was pumped when I saw your name on this video, as I am a huge fan of your Hands-on ML book! Thank you!
Sweet video, I bought your book a few months ago and highly look forward to reading it.
Great idea to use a simple example to describe the features of capsule networks. Great job!
Thank you for the video Aurélien! Please, make more of them. Your explanations are very clear and easy to follow. Actually your book, Hands-on ML, was the first book about Deep Learning that didn't make me want to kill myself while reading it xD
Enlightning explanation of Hinton's capsule! Thank you for this and your nice book!
Your explanation and illustration are unbelievably great!
Thank you very much! :))
Great, Aurélien. You should do more videos like this one!
By far the best explanation video I've seen, this is just brilliant!
Hi Aurélien, fantastic video! Your book has been extremely valuable in moving from theoretical understanding and basic implementations of ML/DL algorithms to real-world application and pipelines. Looking forward to more videos in future (subscribed).
This was so great. I'd love to see something like this for transformers!
Aurélien Géron (Author)
- You are awesome and the best teacher in the world :)
Thanks so much! :)
Very clear explanation! Would love to watch your other videos 👍
Awesome video! Waiting for more.
This is super clear and extremely useful! Good work!
Merci Aurélien!
Superbe vidéo, bien présentée, et droit au but :) Thanks for taking the effort to produce this, and making it time-efficient to watch.
oh was craving for this type of explanation
Awesome explanation. Would be nice if we have a video explanation about Matrix Capsules with EM routing paper. I'm currently in second re-read but still having confused about some concepts presented in the paper.
excellent explanation. Much clearer than the paper and I've read that a few times :)
Excellent explanation and excellent source code. Thank you!
Thanks for the very clear explanation : it made a difficult architecture seem much simpler.
Excellent! Very clear delivery.
Fantastic video on capsule networks. This is the best video I have seen. Nice explanation
Great explanation! And off course, my preferred book on Machine Learning. Thanks a lot for sharing all this information!
This is by far an amazing explanation I have ever seen. Please do videos on other major papers out there. I subscribed to you right way!
Excellent video, thanks a lot for the clear explanation and the examples, it's a great complement to the paper. I have subscribed and I am looking forward for other paper reviews!
Awesome explanation, thank you!
Looking forward to the implementation video.
Excellent video. Hard to find good ml explanations on TH-cam among all the hype.
Thanks Horace, any particular topic you would like more information on? This is my very first video, but I plan on making more, so I'll be looking for good topics to cover.
I think there's a severe lack of material past the beginner's level that help people learn more than just "how do I apply this in code". I think blogs/publications like www.inference.vc/ or distill.pub/ are on this level.
I think the best thing to do is choose topics that you feel like you have the most insight/intuition towards. However, here's a short list of topics I think would be cool to have a video on: A3C, policy gradients (well, RL in general), generative adversarial networks, maybe some of the theoretical work on generalization/local optima (eg: why neural nets don't get stuck in bad local minima, why they generalize, etc.), variational autoencoders.
Great work, and I'm sure you've already seen Hinton's comment on your video! :)
Nice suggestions, thanks. I'm a big fan of distill.pub, which I discovered through Christopher Olah's fantastic blog. I'm not sure I can reach that level (I'm just a practitioner, not a researcher), but I can always try. :)
This is so awesome! Please keep making videos.
Thank you, the explanation was clear, and with the code, I can understand the Capsule Networks better.
Amazing explanation. Please do make more of these videos!
Incredible work, Mr Géron. Beautiful explanation!
Wow, this was an amazingly clear explanation of CapsNets. Thanks for all of your hard work!
this is the most wonderful capsnet video tutorial I've ever seen!!
thanks a lot
This was a great explanation! Please make more videos on different ml topics
Super solid. I got here via metafilter. I am subscribed now. Thanks!
Why is there a need of a convolutional layer for the first layer of the CapsNet? What is essentially done is that the output feature map of the convolutional layer is converted into small vectors by reshaping and then routed to the capsules in the next layer. Why can't we consider the input image as a raw pixels' feature map and route that information directly to a layer of capsules (the convolutional capsules)?
Interesting question. If I understand correctly, there would be one primary capsule per pixel, and each primary capsule would simply output a vector equal to the squashed pixel value (a 3D vector for an RGB image). That might actually work, I encourage you to try! I'm guessing the authors chose to start with two convolutional layers for performance reasons. Another possible reason why they didn't choose this option might be that the routing by agreement algorithm pushes capsules to route their output to just one (or very few) capsules in the next layer. Suppose there's a black pixel in the image: it might be a part of just about any possible object, but routing by agreement will push this pixel's capsule to route its output to just one (or a few) higher-level capsules. There's a risk that the correct one won't be in the list. It seems to me that it might be a good idea to narrow down the options before routing by agreement kicks in. That's what the two convolutional layers do. That said, I'm just guessing, here: you could try your idea and publish a paper if it works well! :)
Well, firstly thanks for such a quick reply. No, I am not saying on the pixel level. We can create patches of the image that the convolutional capsule takes as an input. So, it is more like a small window of fixed receptive field let's say (9 x 9) . We can take all those pixels and create the high dimensional vector for that window.
My intuition behind doing this is quite similar to the content based attention used in Seq-2-Seq models. It's like we are assigning only a small part of the image a capsule (well technically, to multiple capsules with different co-efficients.).
As you said, let's say a part of the image that contains an eye (for the facial dataset), it could be contributing to multiple capsules in the next layers. But, we don't have to worry about it. The capsules that have higher scalar product with this eye only, will receive the eye patch.
I loved concept of the paper, there was this only thing that bugged me a bit. The architecture is not homogeneous. first layer is convolutional and subsequent layers are capsules.
Anyways, this is still an idea that I have. Will definitely work on it. Thank you for your encouragement. Btw, what's your github? I would love to follow you.
mine -> github.com/akanimax
You're very welcome. Oh, right, I see what you want to do. To go from the 9x9 patch to the high-dimensional vector for that patch, would you simply flatten the patch to get an 81D vector, then squash that vector? If so, then the patch's brightness would determine the output vector's length, which represents the estimated probability in a capsule network. So the primary capsules could never be confident about the presence of a dark feature. This seems problematic. To avoid this, we would probably have to apply a transformation matrix to the 9x9 patch before we flatten and squash it, so that the brightness information ends up being a part of the orientation of the output vector (it's a pose parameter), rather than defining its length. A convolutional layer would be a convenient way to do this.
Robin Richtsfeld Thanks for the explanation. Now it is clear to me why the lower layer needs to be convolutional. Still I'd like to see what happens if we directly start with the primary caps layer instead of the convolutional layer.
Aurélien Géron "So the primary capsule could never be confident about the presence of a dark feature". I don't know, but I am wondering: doesn't it make the architecture more like humans? Even we can't be sure of dark features.
Btw, thank you for your explanation. I now understand why we need the convolutional layer at the beginning. If I understand correctly, the only difference between the primary caps layer and the first convolutional layer's architectures is that the output of the primary caps layer is dynamically routed to the digitCaps layer while the output of the convolutional layer is fed as it is (with a stride of 2).
Just a last question: why isn't the output of convolutional layer not routed to the primary caps layer? Wouldn't that make the convolutional layer another primary caps layer? I have tried to find an explanation of this a lot, but couldn't. From where I am, there are no mentors. It is my sincere request if you could please address this last question. I promise this is the last one :).
Thanks!
That's the most useful video I found on this topic.
this is a wonderful clear presentation, well done.
Amazing explanation. brilliant work.
This explanation is the best one that I have found! Thank you so much for making this video. Definitely looking forward to the next one about implementation of capsule networks. Do you think maybe you could also do one on the mathematical intuition behind the part where the weights and coupling coefficients interact to compute the prediction vector and then ultimately the activity vector?
Hi Sidrah, thanks for your kind words. I actually uploaded a video about the TensorFlow implementation of CapsNets just a few hours ago: th-cam.com/video/2Kawrd5szHE/w-d-xo.html
I like the idea of a video about the math intuitions behind ML! In the meantime, you may want to take a look at my Jupyter notebook about Linear Algebra: nbviewer.jupyter.org/github/ageron/handson-ml/blob/master/math_linear_algebra.ipynb
It explains how a matrix multiplication can perform all sorts of linear transformations, such as a rotation, resize, skew, and so on. Hope this helps. :)
Thank you... These resources are helping immensely!
In a word, it was great!
Clear and simple explanation. Thanks!
Thank you so much for these excellent videos!
And I think your book is very good companion to Coursera Andrew Ng courses.
From the excellent book to the great videos... Thank you!
Wow.... what a brilliant explanation!!!! yes, please, more videos from you
Still slightly confused on some aspects, but got most of it! Great Explaination! :D
Thank you for this video. It's been helpful for my capstone
I have a question. Pardon me if this is a stupid question, but in case my dad won't be able to answer it I will post what I asked him. This is what I sent my dad; perhaps you'd be better at explaining it:
I have questions. When he is talking about the primary capsules to create the house and boat capsules, does he say that he uses both the rectangle capsule then the triangle capsule as the primary capsules in order to create the potential position of the other shape? What use is this when both structure the same shape? When it comes to the crowding, do you need to have trained the network to recognize a house capsule and a boat capsule to look for those first so that it becomes the more logical explanation of the image (as opposed to the inverted house with random rectangle and triangle), or will the system be able to learn what a boat capsule and house capsule is with no prior training and be able to separate the two figures from that shape?
This was an excellent explanation! Thanks for the video and the implementation links!
Thank you very much: this was very clear, and very well explained.
What a brilliant man!
Thank you such lucid explanation. Nowadays, it seems that capsule networks gain less attention than 2 years ago. Is this true or just my false perception?
Hello,
the margin loss equation at 15:55 is T_k max(0, m_p - ||v||^2) + lambda (1 - T_k) max(0, ||v||^2 - m_m)
but the "square" is outside the max functions on the paper (v2, arxiv.org/abs/1710.09829)
which one is correct?
Hi, good catch, this is indeed an error in my video, the square should be outside the max. My apologies. I'll add an annotation at that point in the video. Thank you very much for spotting this error.
Oh nooooo! TH-cam dropped video annotations , I can't add an error message. I guess the only option is to add a warning in the video description. I'll soon upload another video to go into the details of the implementation, and I'll make it clear at that point. Thanks again for catching the error.
It still a good tutorial! Thank you!
Capsule networks
Dynamic routing between capsules
Computer graphics - instantiation parameter - rendering
Capsule predicts
Boat - rectangle triangle
Capsule network - image classifier
Awesome video!! Each and every second is so informative!! Thanks a lot.
This is really a fantastic tutorial video and it helps me A LOT. Thank you for your sharing!
Thanks for the brilliant explanation
Hi Aurélien, this video is amazingly clear. About being less sensitive to pose estimation and scale... Humans/Toddlers do learn with an integrated camera viewpoint estimator (an integrated accelerometer called the internal hear... which leads to pitch and roll) a depth map (stereovision with 2 eyes which can lead to quickly determining the size) ... and they're able to walk/interact around the object.... Don't you think these priors could help to boost what's necessary to push capsnets to the next level ? For instance when i'm looking at the boat from the sea, i know it's far away (disparity ...) and i see it moving... and the horizon is not tilted because my head is standing vertically. It is much different from a boat drawing printed on a book that could be tilted because the book is tilted... (i know it's a planar representation using disparity again + not moving). So we can let computers try to learn by themselves that extra complexity of not knowing how the pictures were taken but maybe helping them could lead to better performances.
For a 1st video, it's an awesome one ! Bien joué :)
i just came across this thing and holy shit this is absoutely captivating,!!brilliant
Excellent video!
Informative video actually and helped me a lot to know about capsule network. Thanks a lot man😊
Thanks for the video. Good luck with your channel!
Thank you very much for this excellent video!