hey! i hope you can give me an answer, what would be the minimun requirements for a system (RAM and GPU VRAM) to finetune gpt-J-6 ? From the video I could tell that 100% of your 128Gb RAM was used in finetuning. Would a System with 128GB RAM and a P100 or V100 (not sure how much VRAM they have) be sufficient for finetuning? My university has a Hardware Cluster available for their students and they have such P100 and V100 GPUS there but im not sure if this would be enough to fine tune on a dataset with roughly 120 datapoints
Great video, thanks. I don't have a video card that's that good, so I tried it on my HPE Gen9 server. Gave it 30 CPUs (4 physical Xeons, 3.6ghz) and it took 1 minute and 4 seconds to generate a result with your test input.
hi i got the error : "Output 0 of DequantizeAndLinearBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function." when run your colab, pls tell me how to fix it. Thank you
Yes. In all my previous videos with GPTJ I have been using the old method of converting the weights manually. I have a video of myself then using that model for fewshot learning.
@@Brillibits thanks man. I’m currently working on a project where I’m exploring the application of few-shot learning to generate datasets. I’m trying to setup the environment for the project but my system is a 16gb ram windows with 2gb nvidia graphics card. I’m not certain I’d be able to do the few-shot learning on my system. Or you reckon I could manage it since it’s not fine-tuning?
To just load the model and use it, you will need a significantly better system than that for GPTJ. You need roughly 32GB of RAM with swap to run it VERY slowly on the CPU and need a GPU with 16+ GB of VRAM to run the model with any speed.
Hi, when I'm trying the script of gpt-j-6B.ipynb I got an error "RuntimeError: probability tensor contains either `inf`, `nan` or element < 0” with RTX3090 and RAM of 128G. Any reply would help. thx
At the very end when I give it an input I get an error "AssertionError: Torch not compiled with CUDA enabled". I am using a 4090 & google searches have not helped. everything else went super smooth.
This may have to do with how new the rtx 4090 is and the lack of broad support for many popular applications. You can try compiling pytorch from source(I have a video doing that for rtx 3090 when it first came out). Or you try looking at latest video that goes over how to finetune these models using docker
@@Brillibits First Error : in your branch « original_youtube » at the cell where we convert to torch weights (the 4th one) when executing the line « params = reshard(params, old_shape).squeeze(0).T » i actually get this « AttributeError » Second Error : For the notebook in the main branch (this actual video) i get « ImportError : cannot import name GPTJForCausalLM from transformers (unknown location) » at the cell where we simply import every library we need. In both notebooks i was able to install every packages and libraries without any problem (i created the conda environment like you did). About the Second Error, i was able to make it work the first time, but then i get this importError every time even when recreating the conda environment. When it was working the first time, I got an error at the next cell when executing GPTJForCausalLM.from_pretrained(…) but i forgot what it was (may be the same as the First Error)
@@Brillibits i fixed the two errors by updating to jax 0.2.25. Actually I also had to update to transformers 4.12.5 and downgrade to tensorflow 2.5.0 to run the notebook on your main branch
It would be cool if you could make a tutorial on how to train GPT-J/Neo to understand other languages like spanish or italian with custom datasets, I've seen a few posts on it but they were all years old and for GPT-2 only. Thanks for the videos
RuntimeError: CUDA out of memory. Tried to allocate 1.88 GiB (GPU 0; 39.59 GiB total capacity; 36.94 GiB already allocated; 638.19 MiB free; 36.98 GiB reserved in total by PyTorch)
![⭐ฮิตในTikTok!! ( มะมะมะหมูเด้ง MooDeng ) Ver. แดนซ์ ReMix BY [ ดีเจกิต รีมิกซ์ ]](http://i.ytimg.com/vi/4noyAoMa7bQ/mqdefault.jpg)
Discord: discord.gg/F7pjXfVJwZ
Excellent video as usual. Thank you, Blake!
Thanks for watching!
Thank you so much for this video. Do you know if this repo is compatable to run on windows?
I do not know. I recommend dual booting with Linux to get GPU docker support to save yourself from many headaches.
hey! i hope you can give me an answer, what would be the minimun requirements for a system (RAM and GPU VRAM) to finetune gpt-J-6 ? From the video I could tell that 100% of your 128Gb RAM was used in finetuning. Would a System with 128GB RAM and a P100 or V100 (not sure how much VRAM they have) be sufficient for finetuning? My university has a Hardware Cluster available for their students and they have such P100 and V100 GPUS there but im not sure if this would be enough to fine tune on a dataset with roughly 120 datapoints
Hard to say what the minimum VRAM is without trying it. 24GB works and I would doubt much less than that would work.
Hey Blake what are your machine specs? :))
Currently 128 GB ddr4 ram. Ryzen 5950x. 2 3090s and one 1060.
@@Brillibits Nice!! Is it self funded or?
Yes my channel and things used for it are entirely self funded.
@@Brillibits Thanks! Keep it up!
@@TheAIEpiphany thanks!
Great video, thanks.
I don't have a video card that's that good, so I tried it on my HPE Gen9 server. Gave it 30 CPUs (4 physical Xeons, 3.6ghz) and it took 1 minute and 4 seconds to generate a result with your test input.
Thanks for sharing!
I'll be doing it on my HPE Gen7 With x2 X5650s. Mine should be slower, but I'll post my times if I can get it running.
Im having a lot of trouble with the packages , has the command changed for the first line of the jupyter notebooks?
Yes it has! Make a PR or I will try to update it when I can. Google "Pytorch getting started" and use the tool they have. Cuda 11.6 should be good.
hi i got the error : "Output 0 of DequantizeAndLinearBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function." when run your colab, pls tell me how to fix it. Thank you
you add .clone() at one of your functions' exit, I think it's F.linear(...).clone() or something similar if I recall well
Are there any cloud based platforms for training you recommend rather than buying a 3090 outright? Thanks for this really informative video.
Thanks for watching. I have used free Azure credits in the past to do this.
@@Brillibits how about google colab
Quick question blake, can the gpt-j slim version be downloaded and used for instances of few-shot learning?
Yes. In all my previous videos with GPTJ I have been using the old method of converting the weights manually. I have a video of myself then using that model for fewshot learning.
@@Brillibits thanks man. I’m currently working on a project where I’m exploring the application of few-shot learning to generate datasets. I’m trying to setup the environment for the project but my system is a 16gb ram windows with 2gb nvidia graphics card. I’m not certain I’d be able to do the few-shot learning on my system. Or you reckon I could manage it since it’s not fine-tuning?
To just load the model and use it, you will need a significantly better system than that for GPTJ. You need roughly 32GB of RAM with swap to run it VERY slowly on the CPU and need a GPU with 16+ GB of VRAM to run the model with any speed.
Thanks man. I find this stuff very interesting! I want to try this out when I have a better gpu.
I find it really interesting too! Grateful I get to work with this stuff.
Hi, when I'm trying the script of gpt-j-6B.ipynb I got an error "RuntimeError: probability tensor contains either `inf`, `nan` or element < 0” with RTX3090 and RAM of 128G. Any reply would help. thx
What part of the program?
At the very end when I give it an input I get an error "AssertionError: Torch not compiled with CUDA enabled". I am using a 4090 & google searches have not helped. everything else went super smooth.
This may have to do with how new the rtx 4090 is and the lack of broad support for many popular applications. You can try compiling pytorch from source(I have a video doing that for rtx 3090 when it first came out). Or you try looking at latest video that goes over how to finetune these models using docker
I have the following error : AttributeError : module jaxlib.xla_client has no attribute get_local_backend
Any idea how to solve this ?
Where are you getting this error? Which repo? Which line?
@@Brillibits First Error : in your branch « original_youtube » at the cell where we convert to torch weights (the 4th one) when executing the line « params = reshard(params, old_shape).squeeze(0).T » i actually get this « AttributeError »
Second Error : For the notebook in the main branch (this actual video) i get « ImportError : cannot import name GPTJForCausalLM from transformers (unknown location) » at the cell where we simply import every library we need.
In both notebooks i was able to install every packages and libraries without any problem (i created the conda environment like you did).
About the Second Error, i was able to make it work the first time, but then i get this importError every time even when recreating the conda environment. When it was working the first time, I got an error at the next cell when executing GPTJForCausalLM.from_pretrained(…) but i forgot what it was (may be the same as the First Error)
@@Brillibits i fixed the two errors by updating to jax 0.2.25.
Actually I also had to update to transformers 4.12.5 and downgrade to tensorflow 2.5.0 to run the notebook on your main branch
I wish I had the HW to fine tune a model like this.
Yeah it is expensive. I am grateful I was in a position to make the investment.
Thanks
Thanks for watching!
GOLD! Thank you
Glad it helped!
It would be cool if you could make a tutorial on how to train GPT-J/Neo to understand other languages like spanish or italian with custom datasets, I've seen a few posts on it but they were all years old and for GPT-2 only. Thanks for the videos
I have a video on creating a dataset. The idea for making a dataset for other languages is likely very similar
Thank you
how to train it on TPU please? it took 5 hours on dual K80 on azure
Didn't think K80 had enough Vram. If and when I make money from TH-cam, I will start making videos that cost money, such as renting a TPU.
My GPT-J says you are a vegan 😆 ...interesting. Thanks for the video. It helped me a lot 👍
Glad it was helpful!
Is discord channel still a thing? I wanna join.
Yes! Forgot to pin.
Discord: discord.gg/F7pjXfVJwZ
Can we use the same tutorial for code generation with GPT-J?
Yes
@@Brillibits Thanks 👍
What is your system configuration?
Ryzen 9 5950X, 128 GB of RAM, RTX 3060 and RTX 3090. Only ones that really matter for the video are the RAM and 3090
How can i fine tune gptj using 2 tesla t4 gpus??
Thats 16GB of VRAM right? I don't think its possible to do so with the amount of VRAM using the methods I go over in my videos.
I have one a100. Could you please suggest me a way through.
RuntimeError: CUDA out of memory. Tried to allocate 1.88 GiB (GPU 0; 39.59 GiB total capacity; 36.94 GiB already allocated; 638.19 MiB free; 36.98 GiB reserved in total by PyTorch)
Can you make a video on finetuning using sagemaker and hugging face Trainer
Potentially. Thanks for the suggestion.
Can this be done in Windows?
Maybe. I have not tested.
great
Thanks for watching!