The basic reason we don't use (even number) x (even number) layers, is because those layers don't have a "center". Having a "center" pixel (as in a 3 x 3 configuration) is very useful for max and average pooling - it's much more convenient for us.
I didn't understand it. End of the day for even sized filter you could consider any pixel to be the center pixel right? it will end up giving similar values though not same. also max pooling and average pooling works on a output feature map so how is it related?
This is honestly the best video related to machine learning in general I have seen, amazing work. Most people just pull architectures out of thin air or make a clumsy disclaimer to experiment with numbers. This video shows 3d visual representations of popular CNN architectures, and really helps you build all cnns in general.
Perhaps 2x2 kernel is a common trick for learnable stride-2 downsample kernel and upsample deconvolution kernel. It is a more likely about computation efficiency instead of network performance, because such kernels are almost equivalent to downsample/upsample followed by a 3x3 kernel. In this regard, 2x2 combo with stride-2 down/upsample operations do not shrink the resultant feature map size by 2 as 3x3 kernel does, possibly beneficial to image generation tasks. In GAN, 2x2 or 4x4 kernels are commonly found in discriminators which emphasize non-overlapping kernels to avoid grid artifacts.
I think odd sized filters are mainly used since we often use a stride of 1. Each pixel (except for the edges) will then be filtered based on the surrounding pixels (defined by the kernel size). If the kernel size is even the pixel that the kernel represents would be the average pixel of the 4 middle pixels. It introduces a sort of shift of 0.5 pixel. I think it might be fine mathematically speaking, but it feels odd or wrong. Also if you worked with Gaussian filters (which I assume many CNN researchers has) you are literaly forced to use odd sized filters there.
I have some hobbyist signal processing experience of a few decades, and these new methods seem so amateurish compared to what we had in the past. FFT, FHT, DCT, MDCT, FIR filters, IIR filters, FIR design based on frequency response, edge adapted filters (so no need for smaller outputs), filter banks, biorthogonal filter banks, window functions, wavelets, wavelet transforms, laplacian pyramids, curvelets, counterlets, non-separable wavelets, multiresolution analysis, compressive sensing, sparse reconstruction, SIFT, SURF, BRISK, FREAK, yadda yadda. Yes we even had even length filters, and different filters for analysis than for synthesis.
There are Equivalents in AI Model Development and Inference for those though. Many of these signal processing techniques have analogs or are directly applicable in AI and machine learning: - **FFT, FHT, DCT, and MDCT:** Used in feature extraction and preprocessing steps for machine learning models, especially in audio and image processing. - **FIR and IIR Filters:** Used in preprocessing steps to filter and clean data before feeding it into models. - **Wavelets and Wavelet Transforms:** Applied for feature extraction and data compression, useful in handling time-series data. - **Compressive Sensing and Sparse Reconstruction:** Important in developing models that can work with limited data and in reducing the dimensionality of data. - **SIFT, SURF, BRISK, and FREAK:** Feature detection and description techniques that are foundational in computer vision tasks like object recognition and image matching. In AI, techniques like convolutional neural networks (CNNs) often use concepts from signal processing (like filtering and convolutions) to process data in a way that mimics these traditional methods. Signal processing principles help in designing more efficient algorithms and models, improving performance in tasks such as image recognition, speech processing, and time-series analysis.
Yes, I imagine that many AI students that only see 2D animations are surprised to learn the 2D convolutions actually work with 3D tensors (or 4D if it's batched). That was one of my main motivations for creating these animations :)
really helpful for understanding the concept, correct me if i'm wrong, so for the first conv2d layer, it will always contains 1 feature for black and white image, and 3 features for rgb image. And after that the number of features increases depending the number of filters used in the convolution.
Hi Animated AI, thanks for your great video. I have below question: 4:45 indicated the color of filters (i.e. red, yellow, green, blue) represent the "Features". A filter (e.g. the red one) itself in 3-dimension (Height, Width, Feature) also include "Feature". Thus, the "Feature" appear twice. Please could you advise why we need "Feature" twice?
I don’t see a reason for 1x1. All you achieve is loosing information, while also creating N-features, each scaled by a certain factor. This can also be achieved within a normal layer (the scaling i mean). There is rly no point. Obviously outside of Depthwise-Pointwise combo. Pls correct me if I’m missing smt.
Wait so why do we need larger filters in first layer ? To extract more features from only the 3 channels ? And what is better, more chained filters with lower channel count, or lesser amount of chained filters with more channels ?
The filters in the first layer don't need to be larger. There's just no performance benefit to splitting them into a chain of smaller filters. And the reason for that is that the number of features increases dramatically from the input (typically 3 channels for RGB) to something like 16 or 32. The performance benefit of splitting a large filter into smaller filters assumes the number of features stays the same from input to output. > And what is better, more chained filters with lower channel count, or lesser amount of chained filters with more channels? This really depends on the data, how long the chain is, and how many filters you have. It's an ongoing area of research where researchers have found great results in both cases.
at 4:23, you said that the exception to the rule "3x3 filters are more efficient than larger filters" is the first layer, since the input only has 3 channels. i still haven't got this part. i thought when comparing the number of weights needed for each kind of filters, only the size of the filters matter, not the number of channels in the input
I've been trying to avoid equations in the videos, but the formula for the total number of weights needed is (filter width * filter height * filter count * input feature count). You can see this represented visually in my filter count video. Assuming that the filter count is the same as the input feature count, it's more efficient to break large (5x5, 7x7, ...) filters into multiple 3x3 filters. A concrete example where all inputs and outputs have F feature dimensions: One 7x7: (7 * 7 * F * F) = 49 * F^2 Three 3x3s: (3 * 3 * F * F) + (3 * 3 * F * F) + (3 * 3 * F * F) = 27 * F^2 But for the first layer, the filter count is usually much higher than the input feature count of 3. It's more efficient to perform this dramatic increase in feature count using one large filter than with multiple smaller one. A concrete example where the first layer has 16 filters: One 7x7: (7 * 7 * 3 * 16) = 2352 Three 3x3s: (3 * 3 * 3 * 16) + (3 * 3 * 16 * 16) + (3 * 3 * 16 * 16) = 5040
The basic reason we don't use (even number) x (even number) layers, is because those layers don't have a "center". Having a "center" pixel (as in a 3 x 3 configuration) is very useful for max and average pooling - it's much more convenient for us.
I didn't understand it.
End of the day for even sized filter you could consider any pixel to be the center pixel right?
it will end up giving similar values though not same.
also max pooling and average pooling works on a output feature map so how is it related?
This is honestly the best video related to machine learning in general I have seen, amazing work. Most people just pull architectures out of thin air or make a clumsy disclaimer to experiment with numbers. This video shows 3d visual representations of popular CNN architectures, and really helps you build all cnns in general.
Thanks for the effort of maxing this excellent visualization! This creates a very good intuition for how convolutions work and why 3x3 is dominant.
There was no reason that I should have this very question and there had to be a great video telling me the exact reason why on the internet. Bless!
Such an amazing video. Your going to hit 50k soon! Keep this up!!!
Perhaps 2x2 kernel is a common trick for learnable stride-2 downsample kernel and upsample deconvolution kernel. It is a more likely about computation efficiency instead of network performance, because such kernels are almost equivalent to downsample/upsample followed by a 3x3 kernel. In this regard, 2x2 combo with stride-2 down/upsample operations do not shrink the resultant feature map size by 2 as 3x3 kernel does, possibly beneficial to image generation tasks. In GAN, 2x2 or 4x4 kernels are commonly found in discriminators which emphasize non-overlapping kernels to avoid grid artifacts.
I think odd sized filters are mainly used since we often use a stride of 1. Each pixel (except for the edges) will then be filtered based on the surrounding pixels (defined by the kernel size). If the kernel size is even the pixel that the kernel represents would be the average pixel of the 4 middle pixels. It introduces a sort of shift of 0.5 pixel. I think it might be fine mathematically speaking, but it feels odd or wrong. Also if you worked with Gaussian filters (which I assume many CNN researchers has) you are literaly forced to use odd sized filters there.
I have some hobbyist signal processing experience of a few decades, and these new methods seem so amateurish compared to what we had in the past. FFT, FHT, DCT, MDCT, FIR filters, IIR filters, FIR design based on frequency response, edge adapted filters (so no need for smaller outputs), filter banks, biorthogonal filter banks, window functions, wavelets, wavelet transforms, laplacian pyramids, curvelets, counterlets, non-separable wavelets, multiresolution analysis, compressive sensing, sparse reconstruction, SIFT, SURF, BRISK, FREAK, yadda yadda. Yes we even had even length filters, and different filters for analysis than for synthesis.
There are Equivalents in AI Model Development and Inference for those though.
Many of these signal processing techniques have analogs or are directly applicable in AI and machine learning:
- **FFT, FHT, DCT, and MDCT:** Used in feature extraction and preprocessing steps for machine learning models, especially in audio and image processing.
- **FIR and IIR Filters:** Used in preprocessing steps to filter and clean data before feeding it into models.
- **Wavelets and Wavelet Transforms:** Applied for feature extraction and data compression, useful in handling time-series data.
- **Compressive Sensing and Sparse Reconstruction:** Important in developing models that can work with limited data and in reducing the dimensionality of data.
- **SIFT, SURF, BRISK, and FREAK:** Feature detection and description techniques that are foundational in computer vision tasks like object recognition and image matching.
In AI, techniques like convolutional neural networks (CNNs) often use concepts from signal processing (like filtering and convolutions) to process data in a way that mimics these traditional methods. Signal processing principles help in designing more efficient algorithms and models, improving performance in tasks such as image recognition, speech processing, and time-series analysis.
Well explained.
great work in animation and research
friking love your videos! Keep up with your awesome work! :D
In the Unet, GAN architecture when it is required to generate a feature map half of its actual size a 4x4 kernel size is used.
Good Work. amazing explanation
this is awesome and is inspiring me to learn blender!
Wow really beatiful animations , great job! However I got kinda confused since I always saw convolution in 2d haha
Yes, I imagine that many AI students that only see 2D animations are surprised to learn the 2D convolutions actually work with 3D tensors (or 4D if it's batched). That was one of my main motivations for creating these animations :)
About 2*2 filter, a paper maybe helpful
Super cool, thank you!
good explanation. Looking forward to more.
this is amazing.
"we dont talk about the goose goblin" - MadTV
really helpful for understanding the concept, correct me if i'm wrong, so for the first conv2d layer, it will always contains 1 feature for black and white image, and 3 features for rgb image. And after that the number of features increases depending the number of filters used in the convolution.
Yes, that's correct
great work, your video do help me a lot👍
Is there a good reason for why filter sizes of even numbers aren't used at all, except that the padding will be uneven if using "same"?
Great Video Keep going like this 😊
Hi Animated AI, thanks for your great video. I have below question:
4:45 indicated the color of filters (i.e. red, yellow, green, blue) represent the "Features". A filter (e.g. the red one) itself in 3-dimension (Height, Width, Feature) also include "Feature". Thus, the "Feature" appear twice. Please could you advise why we need "Feature" twice?
The even size of the kernel does not allow symmetrical coverage of the area around the pixel
Nice video I like it!
I don’t see a reason for 1x1. All you achieve is loosing information, while also creating N-features, each scaled by a certain factor. This can also be achieved within a normal layer (the scaling i mean). There is rly no point.
Obviously outside of Depthwise-Pointwise combo.
Pls correct me if I’m missing smt.
Wait so why do we need larger filters in first layer ? To extract more features from only the 3 channels ?
And what is better, more chained filters with lower channel count, or lesser amount of chained filters with more channels ?
The filters in the first layer don't need to be larger. There's just no performance benefit to splitting them into a chain of smaller filters. And the reason for that is that the number of features increases dramatically from the input (typically 3 channels for RGB) to something like 16 or 32. The performance benefit of splitting a large filter into smaller filters assumes the number of features stays the same from input to output.
> And what is better, more chained filters with lower channel count, or lesser amount of chained filters with more channels?
This really depends on the data, how long the chain is, and how many filters you have. It's an ongoing area of research where researchers have found great results in both cases.
Wait, I didn’t get why for the first layer 5x5 or 7x7 works better.
Check out my reply to this comment for an explanation: th-cam.com/video/V9ZYDCnItr0/w-d-xo.html&lc=UgweJZ_Bri8emvyNAMF4AaABAg.9hOBaZTROlX9hQHXzxdOZM
at 4:23, you said that the exception to the rule "3x3 filters are more efficient than larger filters" is the first layer, since the input only has 3 channels. i still haven't got this part. i thought when comparing the number of weights needed for each kind of filters, only the size of the filters matter, not the number of channels in the input
I've been trying to avoid equations in the videos, but the formula for the total number of weights needed is (filter width * filter height * filter count * input feature count). You can see this represented visually in my filter count video.
Assuming that the filter count is the same as the input feature count, it's more efficient to break large (5x5, 7x7, ...) filters into multiple 3x3 filters. A concrete example where all inputs and outputs have F feature dimensions:
One 7x7: (7 * 7 * F * F) = 49 * F^2
Three 3x3s: (3 * 3 * F * F) + (3 * 3 * F * F) + (3 * 3 * F * F) = 27 * F^2
But for the first layer, the filter count is usually much higher than the input feature count of 3. It's more efficient to perform this dramatic increase in feature count using one large filter than with multiple smaller one. A concrete example where the first layer has 16 filters:
One 7x7: (7 * 7 * 3 * 16) = 2352
Three 3x3s: (3 * 3 * 3 * 16) + (3 * 3 * 16 * 16) + (3 * 3 * 16 * 16) = 5040
Thanks for taking the time to explain this!@@animatedai
How about of kernel 1x7, 7x1 in inception modul C. Please help me
wanna bet e by e would be somehow mathematically optimal?
Nİce