I completely agree ... Because the number of updates happening depend on batch size and even the size of the update. So if the learning rate is scaled according to batch size linearly the model can perform very well even with much smaller batches.
Really like the videos. However, I want to warn against the general statement that a batch size of one is not recommended. It really depends on the problem/data. So don't simply dismiss stochastic gradient descent, try it!
Hi! Maybe you can help me with this one: if I want to test an already pre-trained image classifier, how do I proceed regarding the amount of images used? The set containing test images has 100k images, I guess it wouldn't make any sense to load them all at once, so how do I proceed? Thanks!
could you just load some at a time. Like the first 50, then while that is processing you make a request to get the next 50 while discarding the first 50. Then just repeat? assuming you can get them via a request.
It does! Your batch has to fit in memory, or it won't work. When you are working with images, for example, you'll quickly find that your batch size can't be too large if you want to fit it in the GPU's memory.
What do you think of (artificially) adding noise to the learning rate. I feel like it used to be more popular to do that, but almost never see it these days.
This is the kind of thing that I hate about deep learning. A single parameter in the optimization method can completely change the results. Batches should be small but not too small. How small? That's for heuristics but will change on different data sets.
I read somewhere about the ability to fit batches in a GPU... can't remember where exactly. That being said, I've seen experiments that show that it really doesn't matter much (if at all.)
It's better for memory usage. GPU, CPU, hard drives, SSD and other in the current 2-bit logic uses memory blocks with sizes of power 2. 2^5 = 32, 2^6=64, 2^7=128 etc. You always want maximum usage of memory. For example you have array with floats, each float will take 32 bits. So, at least it divisible by 32.
the question is whether if you use a batch and reach the global minimum is your model functionally equivalent to one that didn't batch? Are the weights identical... no they aren't . if your model is generative you don't have equivalence with batch/non batch.
If you scale the batch size by the learning rate (i.e. lr=(batch_size/32.)*0.01) then the stochastic gradient descent looks sort of okay here.
Interesting :)
I completely agree ... Because the number of updates happening depend on batch size and even the size of the update. So if the learning rate is scaled according to batch size linearly the model can perform very well even with much smaller batches.
For me batch size and grad accum make the most difference along with lr scaling
Really like the videos. However, I want to warn against the general statement that a batch size of one is not recommended. It really depends on the problem/data. So don't simply dismiss stochastic gradient descent, try it!
I think that’s fair. I’ve never used it in any of the problems I’ve worked on, but you are right.
Hi! Maybe you can help me with this one: if I want to test an already pre-trained image classifier, how do I proceed regarding the amount of images used? The set containing test images has 100k images, I guess it wouldn't make any sense to load them all at once, so how do I proceed? Thanks!
could you just load some at a time. Like the first 50, then while that is processing you make a request to get the next 50 while discarding the first 50. Then just repeat? assuming you can get them via a request.
so, what is the optimal batch size?
It depends. Start with 32 and experiment from there.
@@underfitted Does the amount of main memory Ram or GPU ram make a difference? (great videos!)
It does! Your batch has to fit in memory, or it won't work. When you are working with images, for example, you'll quickly find that your batch size can't be too large if you want to fit it in the GPU's memory.
What do you think of (artificially) adding noise to the learning rate. I feel like it used to be more popular to do that, but almost never see it these days.
Yeah… never seen that honestly. I’ve used schedules to decrease the learning rate over time, but never read about adding noise to it.
This is the kind of thing that I hate about deep learning. A single parameter in the optimization method can completely change the results. Batches should be small but not too small. How small? That's for heuristics but will change on different data sets.
Good work, hope your channel gets more attention
Thanks, John! It takes time and work but I’ll make it happen.
Hey love this video! Was losing touch of the basics !
Glad it was helpful!
If you generate a dummy dataset and set a static learning rate, then smaller batch sizes work better? wtf?
i didnt see a helpful video like this one in the entire internet, thank you ♥
Glad it was helpful!
Amazing!!
Did u know why the batch size os always 32, 64, 128?
I read somewhere about the ability to fit batches in a GPU... can't remember where exactly. That being said, I've seen experiments that show that it really doesn't matter much (if at all.)
It's better for memory usage. GPU, CPU, hard drives, SSD and other in the current 2-bit logic uses memory blocks with sizes of power 2. 2^5 = 32, 2^6=64, 2^7=128 etc. You always want maximum usage of memory. For example you have array with floats, each float will take 32 bits. So, at least it divisible by 32.
Great video man 👏
Your videos are amazing. Thank you so much for this great knowledge and beautiful videos.
Glad you like them!
I love your presentation style! Very energetic :)
Thanks
Good content. Try improving ur way of teaching. Learning should in relaxed tone
Thanks! This was an old video. I’ve tried to improve in the latest few.
A new cat video. Cute.
the question is whether if you use a batch and reach the global minimum is your model functionally equivalent to one that didn't batch? Are the weights identical... no they aren't . if your model is generative you don't have equivalence with batch/non batch.