This is probably where AMD's integrated graphics should shine, especially the most recent rdna 2 and 3 - because you can allocate your ram to be used as vram.
Hi, thanks for the video. Are there any differences in requirements if I only need to interrogate the model? For example through a chat? I do not need to fine tune it
I have realized that at least in windows you can assign extra paging memory from the SSD and this memory is also shared with the video card, it is not really necessary to have 2 3090. It would be really stupid, windows 11 comes with 16gb extra by default that it takes from the hard drive.
Looking at this today, with the likes of GPT-4 existing, 20B is in no way massive anymore, just a year later. Arguably it wasn't massive even a year ago.
Currently have a shopping cart with two 3090s in it and I'm just staring at the page. I would very much like to run an API service using something like this.
Hi Blake, thanks for the video. I tested the model using the host setup (initially I had some problems too, but creating a conda environment with python 3.8 resolved these), there is a weird thing however. Let's say I set top_p to 1, temperature to 0.7 and give it a prompt like "Hello, my name is". No matter how many times I run generate on it, it keeps putting out the exact same text, in this case "Hello, my name is Mabel, I'm a new member of the community...". If I change the temperature, the output changes too, but only once. After that, it keeps repeating again. I tried this in "interactive" mode as well, but the results are equally predictable after a restart. I'm no expert, but it seems like it's doing some sort of greedy search, instead of actual random sampling, making the output deterministic. Any suggestion would be much appreciated!
There is limited support for pytorch and apple silicon. If you have the memory, try running it using the huggingface model. I have a seperate video going over that.
@@albertstarfield GPT-NeoX is complete garbage and a light year away from even vanilla GPT-3 Davinci. You will easily see this if you use both back to back.
Awesome , I managed to get it working , may be next time please consider a video on fine tuning , few shot learning or train on new text.
Fine tuning will require the cloud. I may make a video on those topics in the future.
This is a really informative video. I'm buying a couople of 3090s soon to try this
Definitely use the Huggingface version. I have a video on that.
This is probably where AMD's integrated graphics should shine, especially the most recent rdna 2 and 3 - because you can allocate your ram to be used as vram.
possible to run in integrated graphics?
@@drkmnml9850 with rocm definitely
That would be interesting. It will definitely be slower, but maybe allow the use of larger models easier.
Hi, thanks for the video. Are there any differences in requirements if I only need to interrogate the model? For example through a chat? I do not need to fine tune it
I have realized that at least in windows you can assign extra paging memory from the SSD and this memory is also shared with the video card, it is not really necessary to have 2 3090. It would be really stupid, windows 11 comes with 16gb extra by default that it takes from the hard drive.
Thanks for watching and thanks for sharing!
How you have configured that?
Спасибо но это всё сложно когда уже появится установка в один клик?
can nvidia ReBAR work for the vram? I've got a 64gb ram system amd I'm looking into getting a 3050 or something with ReBAR to get this working
Not sure. EIther way I recommend using HuggingFace implementation now
when you were copying the get link and cloning the repo, where did you paste it is that command line or anaconda?
Does this model train on GPU (vRAM) or CPU RAM while fine-tuning?
This is not a video on finetuning(though a video on that likely will be coming soon). When finetuning, we use both.
can you tell me how to connect gpus in order to train
Looking at this today, with the likes of GPT-4 existing, 20B is in no way massive anymore, just a year later. Arguably it wasn't massive even a year ago.
Kinda standard size now lol
Currently have a shopping cart with two 3090s in it and I'm just staring at the page. I would very much like to run an API service using something like this.
If you have a use case, its a fairly reasonable investment. 4000s series may be coming out soon though.
@@Brillibits Yeah, I wonder what the premium will be for the 4000 series.
Just get some refurbished NVIDIA Tesla k80s. Very affordable.
@@timothynewton7500 I already have a 3090, will the nvidia K80 work with it…?
Allocate disk cache with windows 11
Hi Blake, thanks for the video. I tested the model using the host setup (initially I had some problems too, but creating a conda environment with python 3.8 resolved these), there is a weird thing however.
Let's say I set top_p to 1, temperature to 0.7 and give it a prompt like "Hello, my name is". No matter how many times I run generate on it, it keeps putting out the exact same text, in this case "Hello, my name is Mabel, I'm a new member of the community...". If I change the temperature, the output changes too, but only once. After that, it keeps repeating again. I tried this in "interactive" mode as well, but the results are equally predictable after a restart. I'm no expert, but it seems like it's doing some sort of greedy search, instead of actual random sampling, making the output deterministic. Any suggestion would be much appreciated!
That is odd. What did you set top_k to? I will look into this later myself.
don't use it like a chatbot
which wls you use with which terminal and which linux ?
I am using Ubuntu 20.04. I am not using WSL most of the time, including here.
Can this work on an M1 Max apple silicon chip ?
There is limited support for pytorch and apple silicon. If you have the memory, try running it using the huggingface model. I have a seperate video going over that.
😂
Hello Sir, Is it possible to run on CPU ?
Yes but very slow
could it work on two 3080's?
If you use HuggingFace int8 and have 12GB models perhaps.
what is ur console?
Powershell in this video
@@Brillibits w11? in v10 i don't see powershell like that
@@alexisvillegas1953 Its powershell but through the newish windows terminal. I believe you can install it through the MS store.
@@Brillibits i found it, thanks
could you run it on colab?
It may be possible to do this now. If will need a GPU with at least 24GB of VRAM though and use int8 through HuggingFace
this is better than openai gpt3?
GPT3 comes in many sizes. It's not better than the largest one, but it's close to the performance of comparable sizes
@@Brillibits what size openai GPT3 has?
@@goncharovblog The largest one is 175 billion parameters.
some sources says based on the benchmark that GPT-NEOX is really close to DaVinci model of OpenAI but OpenAI DaVinci GPT-3 is still slightly better
@@albertstarfield GPT-NeoX is complete garbage and a light year away from even vanilla GPT-3 Davinci. You will easily see this if you use both back to back.
You got the answer finally:
'The "meaning of life" is to make God happy and to be a good person.'
Well said!