Tips Tricks 16 - How much memory to train a DL model on large images
ฝัง
- เผยแพร่เมื่อ 9 ก.พ. 2025
- Rough calculation to estimate the required memory (esp. GPU) to train a deep learning model.
Code generated in the video can be downloaded from here:
github.com/bns...
Great explanation. I used to manage the image and batch size on an ad-hoc basis resulting in OOM errors. This tool now gives me a better way to anticipate memory requirements.
Actual memory requirements depend on a few other factors but this calculation gives a rough estimate.
Great explanation!! Incredible to see how big a single image can get once it's gone through the whole network.
Exactly!
This is correct if you want to keep all output data in memory. In many cases, the output data is temporary (used once) and no need to keep it. In such case, you can create a buffer in to swap data in & out which save significant memory footprint....
Great video as always! I learn more from you, than from most of my teachers at the graduate school 😅
Not sure if I am happy to hear that or sad but I am glad you find these tutorials to be informative.
this is an important video, when you decide to buy a GPU, you will usually have the challenge to select which of them.
Thanks Dear Sreeni
You are absolutely right, this is the basic calculation you need to do while shopping for a GPU. In fact, I recommend signing up for colab pro for a month to see if 16GB works for you. If not, try Google cloud AI where you can subscribe to higher powered GPUs. Once you confirm that a specific GPU works for your application, do the math in terms of purchasing a GPU vs. subscribing via Google (or AWS or something else).
@@DigitalSreeni thanks for replying Mr.Sreeni,,you have been making very important course,I will have been appreciating you.
Best wishes,
Behnoud from 🇮🇷
Such useful information in a relatively short video! Amazing!
Glad you think so!
@@DigitalSreeni btw I assume this would not work for models that contain transfer-learning. For example a model with VGG-16 as backbone and then some FC layers. When we call model.layers() we don't have the option to expand the nested layers. Please correct me if I'm wrong
Best Channel for Information on DL and AI.
Thank you
I appreciate that!
Thank you very much for this video. I have a question and maybe you have the time to answer it:
On the slide at 8:38, you said that the backward pass consumes a similar amount of memory as the memory requirements of the feature maps from the layers. Consequently, shouldn't there be a factor of two applied to "features_mem" in your function that approximates the total memory needed for the neural net?
The required memory of the parameter gradients is also neglected, isn't it? Independent of the optimizer a factor of two applied to the "parameter_mem_MB" should be reasonable since the gradient must always be calculated. Please correct me if I am wrong! In addition to that the memory requirement for the momentum that is used by some optimizers would require even a factor of three applied to the "parameter_mem_MB" as the information about previous iterations is needed for the momentum calculation.
Most of this is mentioned on the slide at 8:38 but not considered in the "get_model_memory_usage" function. Maybe you can give some feedback on this. That would be really nice.
Thanks Mr Srini for this video. Very useful
Wow. Simply amazing.
Glad you like it!
you can use mixed presision with float16 this will bring the size down by half.
Absolutely amazing video 👌🥰 is it possible to do a similar example for a GAN model? Such an example is the stack gan?
Very informative video. I have a query. I am trying to train standard UNet model on 512x512x3 dataset with batch_size=4 on RTX 3090 with 24GB GPU. I am getting memory full error. Please help me to resolve this issue.
Thank you for the response.
@@MARTIN-101 Actually, there were some issue with RAM. I have replaced it & now I am able to run it.
But if i increase the dataset or model parameters, then i have to do it in two steps.
(i) Train the model, save the weights & restart the kernel.
(ii) load trained weights and test.
Thank you very much for your knowledge sharing sir...
Sir can I use nvidia Jetson developer kit for training deep learning python codes ?
I recently learn a lot from your youtube tutorial sir! Anyway im working on some rnd regarding human segmentation (black and white) and make your tutorial as my guide.
Just want to ask, what would make a good segmentation dataset ya sir? Is it the quality? Variety of human body?
Respected sir for my cardiac mri segmentation work ,i used unet architecture , for that training i have taken in the ratio (80,100).The total images are nearly 900 but in the output only 224 images are displayed . Can u please explain me?
Hi, nice video, however have you ever considered or tried to adapt Gradient accumulation ? Even if your batch does not fit to your GPUs memory in Tensorflow there are technics to cut your big batch into mini batches. Maybe it would be worth to try it and make the video about it?
A great episode!
Thanks
i didn't get the "21 times" @ 2:31. Can you please elaborate.?
I meant to say that your input image has 3 channels and if your feature maps size is 64, it is like copying your input image 64/3 times which is approximately 21 times.
Thanks, Sreeni, I see a lot of machine learning learners recommend going to Google Colab Pro, what do you think about this?
I will probably record a video on the topic but the short answer would be - it depends. The free version of Colab is great but it disconnects frequently, you need to babysit the training. The pro version is a bit lenient when it comes to session time and gives you 24 hrs. So it may be worth it if you are annoyed by the constant babysitting of the free version. Although, I should mention that the pro version will not give you a lot more resources so it will not solve any large data problems, if that is your challenge.
Google Colab Pro is worth it if you cannot get RTX 3090 or Titan RTX or some GPUs better than these two. For Colab free users, you can only get Tesla P100 but for Pro users, you can probably get Tesla V100 GPU, which doubles the speed. Also for Pro version, the CPU memory is 25 GB but for the free version, the CPU memory is only 12 GB. Also for Pro version, you can use terminal, which is not existed in the free version. I train my GAN model on Colab Pro Tesla V100 GPU, it takes 200-210 seconds for each epoch (for free version Tesla P100 GPU, it takes about 350 seconds for each epoch) but if I train my GAN model on RTX A6000 (a $5000 GPU), it takes about 180 seconds for each epoch, only save about about 30 seconds for each epoch. I had already set the batch size to the maximum. Considering the price tags, you can see how Google is generous to offer us Tesla V100 on Colab Pro for only $9.99 a month.
Thanks for sharing your experience Yifei. I should also add that with colab pro you have a good chance of getting 26GB RAM but it is not guaranteed and also depends on your location. Same goes for GPU, you have higher chances of getting better piece of hardware compared to the free version but it is not guaranteed. I got frustrated too many times with colab pro so I paid extra to rent a virtual machine. It was (and is) expensive but it makes things predictable and gets the work done. I wouldn’t spend that type of money for hobby coding, it was for real work.
@@DigitalSreeni Thank you!
@@DigitalSreeni Hello Dr.Screeni, just let you know Google launches Colab Pro+ ($49.99/month). I has upgraded my Colab Pro to Pro+. However, Colab Pro+ still gives me Tesla P100 not matter how many times I disconnected it.
Thanks for this video. Super helpful. Can you please benchmark 3080ti vs 3090 for deep learning
There are 2 primary specs you need to consider when picking a GPU.
1. Memory: This is important if you work with large data sets. For example, if you'd like to work with large images or large batches. Please note that having small memory means you cannot work with large data, it stops you from getting the work done.
2. Processing speed: This is of course highly relevant, especially if time to results is important. Please note that having lower processing speed means your time to results is slow but you will get the work done. It will not stop you from getting the work done (generally).
Considering the above two dimensions you can now compare 3080ti and 3090.
The primary difference is in memory, 3090 comes with 24GB where as 3080ti comes with 12GB. This is a significant amount of difference. When it comes to processing speed, both cards have near identical specs for CUDA cores, tensor cores, etc.
In summary, can you get your work done with 12GB? Also, can you get away by using 12GB provided for free by colab or even $10/mo colab pro? If not, consider 3090.
@@DigitalSreeni Thanks for your reply. I will keep this in mind while making my purchase
Thank you, this is really helpful.
Glad it was helpful!
Thank you very much for this video
You are welcome
Thank you! But I feel the real situation is more complex than it. I do GAN research in my PhD program. I use the same model, same codes, same image dataset and same batch size on Colab Tesla V100 (16GB VRAM) and RTX A6000 (48 GB). On Colab Tesla V100, it takes about 16 GB VRAM. But on RTX A6000, it takes more than 30 GB. So I think the number of CUDA cores also affect the memory since more CUDA cores will do more parallel computing. RTX A6000 has about 10, 000 CUDA cores and Tesla V100 has about 5,000 CUDA cores.
I am not sure how number of cores affect the storage memory, may be I am missing something. Of course, it impacts the speed for calculations and may offload some overhead storage but the ability of the GPU to work with some data primarily depends on the model, batch size, and type of data (float, int, etc.). In any case, for learning purposes you need about 12GB GPU and for serious work you need at least 32GB.
I don't think so number of CUDA cores in any way affects the model size calculations.
@@khushpatelmd But my experiments shows it does.
@@DigitalSreeni Something should be missing here. My GAN is for image compression so the generator has encoder and decoder. I calculated the size of outputs of the encoder, and it gave 65.75 GB for a singe image. However, for a Colab Tesla V100 (16GB) GPU, I can set the batch size to 8 for my whole GAN model. Based on my previous experiments, which I had mentioned in my comment, I think GPU dynamically record the output images in the memory in order to save the memory, not record all output images at once. Your method calculates the upper bound of the GPU memory the model uses. But thank you for your effort again.
Very much interesting stuff, thanks a lot, but as usual the code isn't available within your amazing github stuff.
Idea. Call graph on memory.
so I think for most graduate students who do reserch about image in DP.Their lab should basically provide >=24 gb gpu.And i am starting to apply now😂single 4070 totally not work