Make Google Colab 25x Faster!

[@mariop3519]

Oh man, this was an incredible find!!!! Thanks, Brother, it took my Training time from hours to secs......🤘🤘

[@thequirkyengineer]

I make videos for comments like this! I’m glad you find it useful :)

[@priyasaha6773]

Thanks a lot, for this. It worked so amazingly with a big image dataset. I was so frustrated with importing it from google drive.

[@thequirkyengineer]

I'm glad it worked!

[@yusufyesilyurt2754]

hi, I have a big image dataset as well and trying to find a way to optimize the process. This trick seems fine but the disk space in colab seems limited even with pro it goes up to max of about 300gb I think. I need around 1TB of space. How much disk space you needed for your dataset and did you find a way to get around this? thanks

[@FakherEddinKhashman]

Great trick!
it took me a while to understand the difference because the script is not clear, I would suggest making it more clear and focus on it!
Thanks

[@thequirkyengineer]

Thanks for the feedback! I didn’t expect this video to be shared to this many people. I’m thinking making similar content and will make sure script and delivery is clear 👌🏻

[@geethakoritala4444]

Thank you so much.... such a time saver 👌

[@wrake069]

Are you a bot or what

[@Sagardeep_Das]

Now the duckers have made the faster GPUs paid. Slower ones like T4 GPU and v2 TPU can be used only once. It's becoming meaningless to use google colab anymore as the settings like yours don't show anymore.

[@yusufyesilyurt2754]

Merhaba, çok faydalı bir "trick" gerçekten, paylaştığınız için teşekkürler. Benim bir sorum var, yaklaşık 1TB boyutunda bir veri için de bu yöntemi kullanabilir miyim? Colab'da bu miktarda bir disk space almam mümkün mü normal pro aboneliği ile yoksa drive'a mahkum muyum? Teşekkürler.

[@yusufyesilyurt2754]

Biraz araştırdım ancak pro aboneliği disk space'i yalnızca 2 katına çıkarıyor galiba yani 300gb civarında bir alan var ama emin olamadım, çok yeniyim bu alanda bu yüzden görememiş olabilirim, bilginiz varsa paylaşırsanız sevinirim.

[@thequirkyengineer]

Merhabalar. 1 TB çok büyük bir dataset. Colab veya benzer machine learning platformlarında o kadar büyük data ile eğitim yapamayabilirsiniz. Ya drive'a mahkum kalacaksınız ya da Amazon, Microsoft, Google veya diğer cloud çözümlerine bakılabilir. Ona göre storage'ı yüksek ve gpu'lu makineler kiralamak en mantıklısı olacaktır. Alternatif olarak dataset eğer png formatındaysa jpg yapılabilir. Veya video ile uğraşıyorsanız tüm kareler yerine bazı kareler alınabilir. Onun dışında aklıma bir şey gelmedi 1tb gerçekten baya fazla :D

[@ahsanulkyumsiam8774]

Thanks man , it's great

[@thequirkyengineer]

You're welcome!

[@wrake069]

Wtf it's not working bot

[@StarEadrom]

Hi there! I have a question)
After extracting images from zip file, how should I call it? Like path = '/content/drive...' and directory to the path, or in other way?

[@thequirkyengineer]

Hello there!
I'm thinking your zip file has the structure:
-dataset
--1.png
--2.png
After extracting the images from zip file with that structure, it should be easy as path='dataset' or path='./dataset' as long as you stay in the same directory. Or if you know the absolute path, you can also do as you wrote. Btw, if you refresh the folders panel on the left, you will see the dataset in your current directory. You can click on that and get the absolute path if you want :) I hope it solved your question

[@olekmajkowski4139]

Dose it have any use-case in NPL? I’m using load dataset from datasets library. I have 5k json text-label dataset for text classification. I also use pandas to play around with it.
Thanks in advance for answering

[@louiserichard1729]

Ok Google Colab is connected to my Google Drive fine and my dataset is connected too with the good name for the zip file inside. Fine. And after that you say THats it !!! Ok fine but then WHERE DO WE GO AFTER THAT ??? its like I am stuck there and cannot go on training anything anywhere Thanks for your help

[@AjayYadav-ev2ej]

Nice 👍

[@sumayarofmita7699]

Not useful 👎

[@thequirkyengineer]

Hello. How is your data looking like? Is it multiple large files?

[@sumayarofmita7699]

@@thequirkyengineer yes

[@thequirkyengineer]

@sumayarofmita7699 so basically if you have maybe thousand or even millions of files, it creates a bottleneck when fetching all the data from drive. I’m just taking all those files into the colab’s directory, giving low latency. If it didn’t work for you, maybe you don’t have that much of files that will create that much of bottleneck

[@wrake069]

Agree 👍

[@yenimezgame125]

aha lan kesin türk drive kısmında türkçe yazı vardı insta bırakabilirmisin

[@thequirkyengineer]

Dostum merhaba yardımcı olabileceğim bir konu varsa söyle yeterki :)