Regarding position specific computations: We always could concat position-encoding feature maps, e.g. computed kernels would not only be content but also real position dependent. The video contains a great 10min CNN intro/recap! Some notes for completeness: - There are {1,2,3..N}-D convolutions, 4D weights are the 2D conv-case (most popular, due to image use-case). - The center-output mappings only stay at the same position with proper padding, otherwise there the output-feature maps will be (w-kw+1)x(h-kh+1) (e.g. 4x4 input with 3x3 kernel -> 2x2 output without padding).
I don't know if standard convolutional operations on tensor frameworks support per location kernels which might be a barrier for practitioners in the short term. That said, I really like the idea.
Love your channel and way you explain Please do tell something related to research. How to do and what should need to do regarding deep learning in computer vision.
At 24:00 you mention that this "involution kernel" is also spatial agnostic, since it will generate the same one for two different pixel if they have the same channel components. Do you think it would be worthwhile to add a positional encoding to the channels to make each "involution kernel" truly position specific?
That's interesting, but is there really a need to/do we want that? It will enforce the idea that pixels in the top left corner of an image are semantically different from pixels somewhere else in the image, when in reality that is not true since the location of a pixel is more an artifact of whoever took the photograph than actual semantic meaning. Won't this make it lose translation invariance?
Maybe, we just create a bunch of kernels manually, then apply every kernel to every channel, after several layers, we get a channel tree network, a network without training needed, we marked every activate cell as a connection to the target, more connections more credits to the target.
Only if it turns out to be useful, but these nets are not intended to prune useless, random "features." Ever wondered why BERT transformers have 8 heads? They throw enough compute/storage at any problem and hope some useful feature would float to the top.
Too bad that the emphasis is on the number of parameters or flops, these are known useless proxy measures of the things that really matter: generalization and computation time. The later point is a huge disappointment (at least for those still believing in the relevance of flops), as RedNet are actually *slower* (Table 2 of the paper) than ResNet. Ooops, the x-axis of those comparison graphs are now irrelevant: how to RedNet then *really* compares to old ResNet, not to mention newer variants?
this could be explained because pytorch call a very well optimized nvidia CuDNN implementation for the convolution operation used in Resnet whereas their new operation is written in pure pytorch. Using computation time is a bad idea for theorical papers as results would be even more sensitive to hardware lottery
Hello Yannic! First of all, thank you for the work you do to present the papers to us. I think it's obvious that you're getting faster and faster at producing the stuff. But now it's like this: you might start working soon. And maybe these videos are also a great advertisement for you to get exciting positions. Nevertheless, I hope that you will continue to find the time to fill your channel with life in the future. Take care Jürgen
@@frazuppi4897 what makes the code bad ? I’ve quickly scheme through it and it doesn’t look that bad to me. Also I guess the original author was ironic ? He would have preferred more math in the paper ? Trying to learn here :) thanks !
15:10 The spacial-agnosticity of CNNs are counteracted by the fact that the features that these kernels extract are propagated further down the layers. Eventually a DNN at the end or a global max pooling does some non-spacial-agnostic stuff. . Nice idea, it would be interesting to see more of this meta-generated neural networks.
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
OUTLINE:
0:00 - Intro & Overview
3:00 - Principles of Convolution
10:50 - Towards spatial-specific computations
17:00 - The Involution Operator
20:00 - Comparison to Self-Attention
25:15 - Experimental Results
30:30 - Comments & Conclusion
i love your channel yannic! keep up your good work!
These guys are presenting at CVPR tomorrow at 6am!
Regarding position specific computations: We always could concat position-encoding feature maps, e.g. computed kernels would not only be content but also real position dependent.
The video contains a great 10min CNN intro/recap! Some notes for completeness:
- There are {1,2,3..N}-D convolutions, 4D weights are the 2D conv-case (most popular, due to image use-case).
- The center-output mappings only stay at the same position with proper padding, otherwise there the output-feature maps will be (w-kw+1)x(h-kh+1) (e.g. 4x4 input with 3x3 kernel -> 2x2 output without padding).
14:35 - "Whatever, we don't do slidey slidey anymore"
I got my convolution revision pretty well. Such a lovely one
The idea here seems a straightforward combination of fast weight memory networks and locally connected layers
I don't know if standard convolutional operations on tensor frameworks support per location kernels which might be a barrier for practitioners in the short term. That said, I really like the idea.
Love your channel and way you explain
Please do tell something related to research. How to do and what should need to do regarding deep learning in computer vision.
Where do new channels come from and how does information from different channels get fused together?
About your comment at 24:00, if the pixel also contained spatial information (e.g. RGBXY), wouldn't this then be spatial-specific?
At 24:00 you mention that this "involution kernel" is also spatial agnostic, since it will generate the same one for two different pixel if they have the same channel components. Do you think it would be worthwhile to add a positional encoding to the channels to make each "involution kernel" truly position specific?
That's interesting, but is there really a need to/do we want that? It will enforce the idea that pixels in the top left corner of an image are semantically different from pixels somewhere else in the image, when in reality that is not true since the location of a pixel is more an artifact of whoever took the photograph than actual semantic meaning. Won't this make it lose translation invariance?
2:32 - 2:59 This reminded me of Schmidhuber 😀
Maybe, we just create a bunch of kernels manually, then apply every kernel to every channel, after several layers, we get a channel tree network, a network without training needed, we marked every activate cell as a connection to the target, more connections more credits to the target.
Where to learn about transformers & Self Attention?
thanks to their pseudo-code, I got that kernels are not direct learnable weights (in contrast to normal convolution convention )
What do you think about Kervolutions?
I heard there are multiple instances in AWS, whose name is yannic_xxxx.
Doesn't channel = feature... roughly ?
Only if it turns out to be useful, but these nets are not intended to prune useless, random "features." Ever wondered why BERT transformers have 8 heads? They throw enough compute/storage at any problem and hope some useful feature would float to the top.
on a per-layer basis, yes more or less. channel is the technical name for the dimension, while feature is a more conceptual thing
isn't the first part just depthwise convolution?
Too bad that the emphasis is on the number of parameters or flops, these are known useless proxy measures of the things that really matter: generalization and computation time. The later point is a huge disappointment (at least for those still believing in the relevance of flops), as RedNet are actually *slower* (Table 2 of the paper) than ResNet. Ooops, the x-axis of those comparison graphs are now irrelevant: how to RedNet then *really* compares to old ResNet, not to mention newer variants?
this could be explained because pytorch call a very well optimized nvidia CuDNN implementation for the convolution operation used in Resnet whereas their new operation is written in pure pytorch. Using computation time is a bad idea for theorical papers as results would be even more sensitive to hardware lottery
what do you use to highlight on the pdf?
first comment!
I like turtles
Free Hong Kong!!!
內卷
Hello Yannic! First of all, thank you for the work you do to present the papers to us. I think it's obvious that you're getting faster and faster at producing the stuff.
But now it's like this: you might start working soon. And maybe these videos are also a great advertisement for you to get exciting positions. Nevertheless, I hope that you will continue to find the time to fill your channel with life in the future.
Take care Jürgen
that is some convoluted writing!
yet another good paper with pytorch pseudo code instead of math heavy latex
@@frazuppi4897 what makes the code bad ? I’ve quickly scheme through it and it doesn’t look that bad to me. Also I guess the original author was ironic ? He would have preferred more math in the paper ? Trying to learn here :) thanks !
15:10 The spacial-agnosticity of CNNs are counteracted by the fact that the features that these kernels extract are propagated further down the layers. Eventually a DNN at the end or a global max pooling does some non-spacial-agnostic stuff.
.
Nice idea, it would be interesting to see more of this meta-generated neural networks.
Keep rocking the papers! You will be to 100k subscribers before you know it.
Amazing channel, man! Thank you very much!
These researchers are going too far!!
second
Amazing