ERRATA (from Lucas Beyer): "I believe you missed the main concern with "batch cheating". It's for losses that act on the full batch, as opposed to on each sample individually. For example, triplet in FaceNet or n-pairs in CLIP. BN allows for "shortcut" solution to loss. See also BatchReNorm paper."
This is exactly what I was going to say. They got rid of batchnorm, however, they are still normalising the gradients using their respective weights. Also, this seems very similar to OpenAI's PPO algorithm. They also clip the gradients during the training process.
@@pepe_reeze9320 I think it refers to loss functions that arent just a sum over a function of individual examples, but a general function of all the examples. For example, a contrastive loss is a loss that depends on a pair of examples, but not as a sum over the loss on each example. So a contrastive loss may be D(f(X1),f(X2)), where D is some measure of discrepancy of the output of the net for example X1 and example X2. Batch norm makes the output depend on all the examples, so that f(X1) will actually depend somewhat on X2 (and viceversa), and that can cause the network to cheat, because it's task is to produce an output similar to that for X2, without knowledge of X2
I think the concern Yannic pointed for additive losses is technically also valid, but seems much less likely to cause problems. And the even subtler concern for the gradient clipping seems to me even less likely to cause problems
@@GuillermoValleCosmos Thanks, you confirmed my expectation! I'll rephrase in my own words: BN injects information of batch statistics into the network and thereby gives each sample of the batch information about other samples. Each sample “knows” mean and variance of their batch. This dependency is problematic in contrastive learning because the model could cheat by exploiting information about other samples instead of learning reasonable operations to obtain discriminative features based on each sample alone. In my opinion, this problem is less relevant for non-contrastive losses as the samples are contrasted from an absolute “anchor”, i.e. their respective class label, and not relative from each other.
I applaud the inclusion of a negative results appendix, and hopefully in the future it will become a standard or even required section (by conference/journal).
My take for why they did the architecture search on top of AGC was that to be fair with architectures optimised for BatchNorm. BatchNorm was out in 2015 and since then all the architectures either by grad students or NAS has been designed or searched with BN as one of its core ingredient. This is especially true for the EfficientNet they are comparing against. Now they have designed AGC trick that fixes all the problems of BN but do it in a different mechanism, just plug that in the place of BN layer for architectures optimised with BN wouldn't be very fair for it right? So they basically developed a BN replacement and squash a few years of NAS research into one, and did it with a more realisitc metric (actual training speed vs theoretical FLOp), which I think is pretty good.
EffNet-B7 and NFNet-F1 achieved the same Top-1 84.7 on ImageNet and have similar FLOPs (37B vs 35B). NFNet-F1 was 8.7x faster on training, but how about inference time? as we know we could always merge BN into weights to speed up those normalized networks. Does anyone have any idea?
Excellent walk through! Great points on the gradient clipping, it should definitely be done before averaging. I wonder if replacing the clipping by the median gradient would have a similar effect...
it isn't a clipping, its a rescaling (it's dividing by magnitude/lambda) ; if you have a batch of 4k, with 1 bad data, and you rescale, you're going to down-weight all the other items in the batch with the problematic sample. Consider the limit of all data in 1 batch, with a bad sample (correctly detected) -- all you would do is take small steps in the bad direction repeatedly.
I would go as far as to say: The batch normalisation shows the inferiority of the model to accommodate for the complexity coming from the variance of the data. It basically accommodates currently for the fact that research has not yet figured out why Batch Normalization is necessary at all for a model given training data. Of course the same argument can be held against the data being unfit for the network. But here we have no other choice but to accept the data as is. Same with preprocessing like transforming audio time signals to frequency domain first.
and you still see some people tell you, go for tensorflow because, well, you know , pytorch is just for hardcore research. machine learning, is still in "building process", and many knowledge, skills, considered now as solide, will be dismissed tomorow. Time for research and improvement. Not certainties.
I agree pytorch is better, I started deep learning with keras and tensorflow, I was struggling with understanding with semantic of the code , but since I started pytorch it allow me to understand every bit of the code , and taking advantage of my programming skills and there is Kind of freedom in using it
Does the gradient clipping occur also (repeatedly) for the gradients that get passed back further into the network, or only at the point where they accumulate at each specific parameter? For the latter case I'd probably try transforming them by a scaled asinh() or erf() function instead of clipping.
Could you do a video on transformers for time series data? Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting or Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
You might be the first cited TH-cam channel in a machine learning research paper, but not the first TH-cam channel cited in a paper. Thunderfoot's channel was cited in the Nature paper "Coulomb explosion during the early stages of the reaction of alkali metals with water" in 2015.
Yes, but only because the gradient is a linear operator. Just like when you want to find the minimas/maximas of a polynomial you can calculate the derivative for every term separately.
Could you share the information what software do you use to annotate pdfs? I couldn't find a tool that gives you a blank page to draw on alongside every pdf page. And I'd like to use it in my classes. I tried to find this information on your channel, but I gave up as there are too many videos where this information could be hidden.
Don't they also change the image resolution for the training set(smaller images while training, and a larger image while inference), there are a lot of the little things that went into achieving this kind of performance gain.
Similar to "get rid of batch norm" - have you had a look into the HSIC bottleneck paper, trying to get rid of backprop? arxiv.org/pdf/1908.01580v3.pdf Coming from the information theory direction myself, i find this approach fascinating. Kind of strange this paper hasn't got more attention.
A pytorch implementation at github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py, because not everyone knows Jax. BTW its a shame that Google DeepMind doesn't use TensorFlow for that kind of projects.
Like if you did a random projection just before the input to a net in a lot of ways that normalizes the statistics of the input. Well literally normalizes as in makes Gaussian. A fixed randomly chosen pattern of sign flips before the fast Hadamard transform is a quick random projection. Repeat for better quality. You may still choose to normalize the vector length as well. You may find you no longer need bias terms!!!!!!!👿👽👻
ERRATA (from Lucas Beyer): "I believe you missed the main concern with "batch cheating". It's for losses that act on the full batch, as opposed to on each sample individually.
For example, triplet in FaceNet or n-pairs in CLIP. BN allows for "shortcut" solution to loss. See also BatchReNorm paper."
This is exactly what I was going to say. They got rid of batchnorm, however, they are still normalising the gradients using their respective weights. Also, this seems very similar to OpenAI's PPO algorithm. They also clip the gradients during the training process.
Could you elaborate on how batch cheating occurs for "losses that act on the full batch"? What do you mean by "full batch"?
@@pepe_reeze9320 I think it refers to loss functions that arent just a sum over a function of individual examples, but a general function of all the examples.
For example, a contrastive loss is a loss that depends on a pair of examples, but not as a sum over the loss on each example.
So a contrastive loss may be D(f(X1),f(X2)), where D is some measure of discrepancy of the output of the net for example X1 and example X2. Batch norm makes the output depend on all the examples, so that f(X1) will actually depend somewhat on X2 (and viceversa), and that can cause the network to cheat, because it's task is to produce an output similar to that for X2, without knowledge of X2
I think the concern Yannic pointed for additive losses is technically also valid, but seems much less likely to cause problems. And the even subtler concern for the gradient clipping seems to me even less likely to cause problems
@@GuillermoValleCosmos Thanks, you confirmed my expectation! I'll rephrase in my own words:
BN injects information of batch statistics into the network and thereby gives each sample of the batch information about other samples. Each sample “knows” mean and variance of their batch. This dependency is problematic in contrastive learning because the model could cheat by exploiting information about other samples instead of learning reasonable operations to obtain discriminative features based on each sample alone.
In my opinion, this problem is less relevant for non-contrastive losses as the samples are contrasted from an absolute “anchor”, i.e. their respective class label, and not relative from each other.
I applaud the inclusion of a negative results appendix, and hopefully in the future it will become a standard or even required section (by conference/journal).
Agree!
I liked that you compared this to Speed-running.
Glitch-less == no reward hacking?? Feature or cheating? ;)
My take for why they did the architecture search on top of AGC was that to be fair with architectures optimised for BatchNorm. BatchNorm was out in 2015 and since then all the architectures either by grad students or NAS has been designed or searched with BN as one of its core ingredient. This is especially true for the EfficientNet they are comparing against.
Now they have designed AGC trick that fixes all the problems of BN but do it in a different mechanism, just plug that in the place of BN layer for architectures optimised with BN wouldn't be very fair for it right? So they basically developed a BN replacement and squash a few years of NAS research into one, and did it with a more realisitc metric (actual training speed vs theoretical FLOp), which I think is pretty good.
Then the main problem that remains is that we do not know whether the improvements are due to better NAS alone or combination of both.
DeepMind: new SOTA model
Google: you must have used tensorflow right?
DeepMind: tensor what?
Google: Uh okay, at least tell me you used TensorBoard
DeepMind: Yeah su- *cough wandb *cough -re
The BEST deep learning channel on youtube. Congrats!
Dude telling how to beat a SOTA paper on youtube, EPIC; we would love to cite the channel
EffNet-B7 and NFNet-F1 achieved the same Top-1 84.7 on ImageNet and have similar FLOPs (37B vs 35B). NFNet-F1 was 8.7x faster on training, but how about inference time? as we know we could always merge BN into weights to speed up those normalized networks. Does anyone have any idea?
love your work, as always.
you sometimes make me think about things deeper than I originally did, or pull a new perspective on certain concepts
Excellent walk through! Great points on the gradient clipping, it should definitely be done before averaging. I wonder if replacing the clipping by the median gradient would have a similar effect...
Was waiting for this! Thank you!!
Thanks for these videos man, you help a lot with your explanations and your way of navigating the paper!
I wonder how it would compare to per training example clipping. Perhaps the compute overhead would be too much.
I wonder how to adapt this nfn for semantic segmentation?
Great for undergraduates, thank you !
You really are a fantastic teacher. Thank you, again!
it isn't a clipping, its a rescaling (it's dividing by magnitude/lambda) ; if you have a batch of 4k, with 1 bad data, and you rescale, you're going to down-weight all the other items in the batch with the problematic sample. Consider the limit of all data in 1 batch, with a bad sample (correctly detected) -- all you would do is take small steps in the bad direction repeatedly.
"Don't come at me math people" - Yannic 2021
I would go as far as to say: The batch normalisation shows the inferiority of the model to accommodate for the complexity coming from the variance of the data. It basically accommodates currently for the fact that research has not yet figured out why Batch Normalization is necessary at all for a model given training data.
Of course the same argument can be held against the data being unfit for the network. But here we have no other choice but to accept the data as is.
Same with preprocessing like transforming audio time signals to frequency domain first.
and you still see some people tell you, go for tensorflow because, well, you know , pytorch is just for hardcore research. machine learning, is still in "building process", and many knowledge, skills, considered now as solide, will be dismissed tomorow. Time for research and improvement. Not certainties.
I agree pytorch is better, I started deep learning with keras and tensorflow, I was struggling with understanding with semantic of the code , but since I started pytorch it allow me to understand every bit of the code , and taking advantage of my programming skills and there is Kind of freedom in using it
Thanks for this. I love your explanations.
Yes, great paper! I always distrusted this strange hack-norm!!1
Okay, but still not solved :(
Thank you! Great explanation. Keep up the good work :)
Does the gradient clipping occur also (repeatedly) for the gradients that get passed back further into the network, or only at the point where they accumulate at each specific parameter? For the latter case I'd probably try transforming them by a scaled asinh() or erf() function instead of clipping.
also thought about that
I feel like every second day there is new SOTA in everything
Can't help it, these companies do speed running to get the big money asap
Could you do a video on transformers for time series data?
Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting
or
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
You might be the first cited TH-cam channel in a machine learning research paper, but not the first TH-cam channel cited in a paper. Thunderfoot's channel was cited in the Nature paper "Coulomb explosion during the early stages of the reaction of alkali metals with water" in 2015.
The sum is finite, so you can swap it round. You’re good. -Math person
Yes, but only because the gradient is a linear operator. Just like when you want to find the minimas/maximas of a polynomial you can calculate the derivative for every term separately.
Could you share the information what software do you use to annotate pdfs? I couldn't find a tool that gives you a blank page to draw on alongside every pdf page. And I'd like to use it in my classes.
I tried to find this information on your channel, but I gave up as there are too many videos where this information could be hidden.
There actually is a tutorial he made about how he reads papers and which apps he uses. AFAIK he uses OneNote for paper annotation
@@1998sini Thank you :-) I found it. It helps me a lot.
@@piotr.ziolo. No problem man, glad I could help :)
This isn't state of the art though no? Efficient Net L2+meta psuedo labeling has highest top-1 or am I missing something?
Thanks for the awesome video, as usual. Now the touche: How's your thesis going? :D
I'm procrastinating :D
@@YannicKilcher we are rooting for you, your videos have literally helped me advance dramatically, really cant say thank you enough
I love your work!
cant wait to see u as the first cited YT channel XD
Just wondering was their result actually verified or they simply put that on their papers....
Don't they also change the image resolution for the training set(smaller images while training, and a larger image while inference), there are a lot of the little things that went into achieving this kind of performance gain.
LayerNorm still needs a lot of memory, but does’t it at least partially resolve the issues that BatchNorm has. Funny it wasn’t mentioned.
Similar to "get rid of batch norm" - have you had a look into the HSIC bottleneck paper, trying to get rid of backprop?
arxiv.org/pdf/1908.01580v3.pdf
Coming from the information theory direction myself, i find this approach fascinating. Kind of strange this paper hasn't got more attention.
A pytorch implementation at github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py, because not everyone knows Jax. BTW its a shame that Google DeepMind doesn't use TensorFlow for that kind of projects.
thanks...lmao, they use partial tf since they know tf is not fast when it comes to deep research
Like if you did a random projection just before the input to a net in a lot of ways that normalizes the statistics of the input. Well literally normalizes as in makes Gaussian. A fixed randomly chosen pattern of sign flips before the fast Hadamard transform is a quick random projection. Repeat for better quality. You may still choose to normalize the vector length as well. You may find you no longer need bias terms!!!!!!!👿👽👻
At this rate someone "Yoloing" the AI will create the singularity...
Creates a network, that is ready to make inference before training)
I feel you're being too critical, complaining about dropout really? That's totally independent of the BN stuff.
7:36 Spot the teenage mutant ninja turtle.
If you are here Bcuz you don't want to dive into the original paper...be blessed !
I love you
First!
"What's the property of dropout"
Efficienter NET XD
so fast
BS matters
It is a fake high-performance model