except for the part that i have got no clue what the required software to start is. what os to run on nor how to run the code in step 2. the farthest i got was git bash for the transformer and thats about it. i just want to run some local ai, why does it have to be so obtuse??
@@norwegiansmores811 It's running in a Jupyter notebook - jupyter.org/ but you can run the code anywhere that can run Python code e.g. a Python REPL or a Python script.
Thanks for the quick and consise video, exactly what I wanted as someone that is already somewhat familiar with using models and just new to hugging face
I am getting an error/ info log from transformers (twice) stating "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained." The model then generates only a bunch of whitespace, no matter the input. I have followed through your steps and made sure the files were downloaded at the expected location. The behavior occurrs both with and without setting legacy=False.
1:03 - Thanks for this clarification. I'd done quite a bit of Google searching and scouring the Hugging Face website for this information. I found nothing of value. I'm a computer enthusiast / gamer and not a professional machine learning engineer. Since embarking on running an LLM locally on my previous daily use desktop, I've noticed its near impossible to find a model's resource needs. GPT4 says a 7b parameter model would consume about 48 GB memory. I asked it what size model would fit in my 12 GB Nvidia 3060, it said about 3.2 billion. My question for you is, why is it that everyone in this space who seems to offer a model (or talk about them) never includes something like a system requirements descriptor? Is it one of those situations where, if you need to ask, you probably don't have enough resources? Thanks for any insight you can give on this phenomenon.
My impression is that most of the models being created are assuming that you have insanely good GPUs to run them on! Since I created this video, there's been a lot of work done by a guy called TheBloke on Hugging Face to 'quantise' the models, which effectively means that the amount of resources required is reduced, but the quality of the model is slightly reduced too. I've found those models work a lot better on my laptop. The Bloke is using a format called GGUF, which is kind of a defact format for LLM models. I made a video showing how to run one of his models on my machine - th-cam.com/video/7BH4C6-HP14/w-d-xo.html. That video uses a tool called Ollama which works on Linux/Mac - th-cam.com/video/NFgEgqua-fg/w-d-xo.html There is also another library called CTransformers which lets you choose whether to run models on the GPU or CPU. I've found the 7B parameter quantised models work reasonably well even on the CPU. I should probably create a video about that I guess! But in the mean time, this is the link - github.com/marella/ctransformers
Hard to know exactly why - maybe connectivity with Hugging Face or maybe your internet or maybe the download tool?! You could try going to Hugging Face directly and click through to files and download them directly to see if it helps.
@@learndatawithmark You got to rerun that part couple times and make sure you have a fast connection. For example, pytorch_model.bin is 6.71gb that would take some time to download
If you're getting a wacky error trying to perform `AutoTokenizer.from_pretrained(model_id, legacy=False)`, do pip install protobuf==3.20.1 and restart the jupyter kernel
If it's running too slowly then maybe it'd make sense to try out some of the quantised models instead. Those ones are smaller and better suited for running on consumer hardware. I quite like Ollama and I've made a few videos on that. This is probably the best place to start - th-cam.com/video/NFgEgqua-fg/w-d-xo.html
thanks sir , however I want to know 1- how one can integrate specific set of models (pre-trained) ones in to Rstudio ? so that one can simply run examples on data "proprietary in my case " locally within R 2- is there a way to ask the inference API for tasks different from the typical sentiment classification of text for example "multi-entity tagging" , "modalities" ....etc your input is highly appreciated
Random idea, but a video on how to run an embeddings model on a laptop would be really cool 😀 Could even combine embeddings + text2text for more specific answers. Or even t5_3b + selenium to create something similar to bing chat. Anyway, wish you luck on TH-cam 😊
Sorry, I didn't see this reply! I've got a notebook with that idea sketched out, so I'll create a video for that soon. On holiday at the moment, but will do it when I get back home!
Hey, great tutorial! I also found your blog on this and followed that, but I always get this error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. This is my first time experimenting with LLMs, so any assistance would be greatly appreciated.
Oh I'm not sure about that error - I haven't seen that one before. Since I made this video I've been playing around with another tool called Ollama which I found easier to use. It might be worth giving that a try to see if that works for you? th-cam.com/video/NFgEgqua-fg/w-d-xo.html
I'm using the latest version of Jupyter Lab and I have it set to dark mode with pretty much every one of the views hidden so that I can use as much of the screen as I can. Not sure if that answered your question, so feel free to follow up!
Thanks Mark for this video. A quick question- Is this safe to pass some PII data to one of the open source hugging face models that require the hugging face API token ? If No, how can this be resolved in deployment so that there is no risk of data leakage ? Please guide through this.
It depends. If you are passing your HF API token because you're using the HF inference endpoint then your data is getting sent to the HF API. If you're passing it because you're downloading a model that requires token auth then your data will only be local to where you run the model that you download.
thanks Mark, very nice video, super clearly put! could you please suggest, what could be the reason if (when trying to set the wifi off) the output of those lines of code is "ModuleNotFoundError: No module named utils"?
utils should be referring to this file - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/utils.py - so in theory that's independent of WiFi connectivity. If it can't find that module you could copy/paste those functions into the notebook and use them like that.
I'm probably missing something, but where are you using the downloaded files? you are entering model_id in .from_pretrained(), how is it finding/using the downloaded model?
I deeply appreciate your video! Although I have a question, does this still works when the model file is a .safetensors or .pth file, not a .bin file? Thank you!
Open a Jupyter Notebook - jupyter.org/ Here's a link to the notebook I used in the video - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/LLMOwnLaptop.ipynb
What is the difference w.r.t to using the classical: from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) Thanks in advance!
really tried to follow along but I'm completely lost. I've never used python I'm assuming I put my own info in the parameters? Do I run each cell after filling it in? i just really have no clue what I'm doing but I've wanted to use hugging face for a while an no tutorial really helped me. it seems like every one expects me to know how to set up virtual environments right off the bat and do everything. if someone could just tell me what im actually putting where or what to run and when it would probably work i think
Awesome content, love your channel! Video is very informative and concise, thanks. As a friendly suggestion, you might want to give a couple of secs at the end for the video for slow people like me to hit that well deserved like button :)
I use a Mac M1 with 64GB RAM. I think it's a 2021 edition. I've found in general that the quantised models work better on my machine - either using Ollama or llama.cpp. th-cam.com/video/YDj_ScvBpKU/w-d-xo.html th-cam.com/video/NFgEgqua-fg/w-d-xo.html
Could you perhaps make a video with this very same theme... but assume there are those who don't know the first thing about installing this stuff? This is WAY too complex... why wouldn't there just be an installer (.exe) file you can just download and run, and just like any other program... Bob's you're uncle! It's ready to use?
Hi, i try to find someone who uses GGUF directly and locally without using a .bin to launch it because I would like to launch it under python, is this possible? Or should I do something else?
You can do this using CTransformers like I did in this video - th-cam.com/video/S2thmwdrYrI/w-d-xo.html I think you might even be able to do it with HuggingFace transformers, but I haven't tried it myself.
@@learndatawithmark if one day you make a video on this, I would like to see it, in fact what I would have liked was to discuss with the model directly with python without going through any interface and to give it a personality with json like we have could do it with webui (but without webui) I tried various methods and honestly I find so little explanation. I had the idea of making my own bot as I saw in "wifu" mode in the sense that it is totally customizable and we give it a personality with a long term memory. The basic idea was to have a small model just for me. I'm just frustrated to see bots that don't even remember talking to us 2 seconds before. xD
@@The_little_black_cat it sounds like you want to keep the history of the chat messages between you and the LLM? I showed how to do this in memory on this video using Ollama, but it can be adapted to another approach - th-cam.com/video/MiJQ_zlnBeo/w-d-xo.html. I can across a tool called MemGPT which I think attempts to solve this problem, but I haven't tried it yet - memgpt.ai/
Great vid, however I’m getting a value error, failure to import transformers error even though I used pip to do that, wondering if it’s a python version issue, I’m using 3.10, wonder if anyone has any ideas ? Thx
I keep following along until about 12 seconds in, where you start typing into something and you say let’s open up age something, and carry on typing into whatever it is you’re typing into - I can’t get that far, I don’t know what to type into
Yeh I think with a bunch of the models you're able to run them on the Hugging Face website on the right hand side of the page. And then in general there are many services that offer APIs that you can call. The approach describe in this video is only for if you don't want to use those services.
You explained completely and perfectly without wasting the audience's time! well done
Thanks!
except for the part that i have got no clue what the required software to start is. what os to run on nor how to run the code in step 2. the farthest i got was git bash for the transformer and thats about it. i just want to run some local ai, why does it have to be so obtuse??
@@norwegiansmores811 It's running in a Jupyter notebook - jupyter.org/ but you can run the code anywhere that can run Python code e.g. a Python REPL or a Python script.
@norwegianmores811 I think the easiest way to run local AI as of June 2024 is now this - lmstudio.ai/
Thanks for the quick and consise video, exactly what I wanted as someone that is already somewhat familiar with using models and just new to hugging face
Thanks a ton man... outstanding video and tutorials
Super video!
An API key is not needed if the model is downloaded and run locally.
So long as you have a behemoth of a machine. 16GB Ram = 100% memory usage 😭😂
I don't think anyone can afford an expensive laptop lol
Do you know how to run it on live servers!! How to get?
It is needed to download the model!
The api key is used to authenticate account for huggingface
Awesome! Thanks for this video.
Absolutely wonderful video! to the point and well explianed! way to go! thanks a lot!
Thanks - very kind of you :D
Thank you very much, you helped me a lot
Thank you for the contents.
Thanks for this video!!
This was an extremely informative video. Really appreciate it.
Thanks, glad you enjoyed it!
thank you very much for this great explaination
I am getting an error/ info log from transformers (twice) stating "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained." The model then generates only a bunch of whitespace, no matter the input. I have followed through your steps and made sure the files were downloaded at the expected location. The behavior occurrs both with and without setting legacy=False.
Does it work after you see that message?
@@learndatawithmark getting the same error and hang when running in colab or locally. seems like the pipeline("my query") never resolves
Same thing happens to me! Hanging and giving random replies! I am using mac m3
what was the interface you were using? Is that on huggingface somewhere?
Excellent thanks dude
1:03 - Thanks for this clarification. I'd done quite a bit of Google searching and scouring the Hugging Face website for this information. I found nothing of value. I'm a computer enthusiast / gamer and not a professional machine learning engineer. Since embarking on running an LLM locally on my previous daily use desktop, I've noticed its near impossible to find a model's resource needs. GPT4 says a 7b parameter model would consume about 48 GB memory. I asked it what size model would fit in my 12 GB Nvidia 3060, it said about 3.2 billion. My question for you is, why is it that everyone in this space who seems to offer a model (or talk about them) never includes something like a system requirements descriptor? Is it one of those situations where, if you need to ask, you probably don't have enough resources? Thanks for any insight you can give on this phenomenon.
My impression is that most of the models being created are assuming that you have insanely good GPUs to run them on!
Since I created this video, there's been a lot of work done by a guy called TheBloke on Hugging Face to 'quantise' the models, which effectively means that the amount of resources required is reduced, but the quality of the model is slightly reduced too.
I've found those models work a lot better on my laptop.
The Bloke is using a format called GGUF, which is kind of a defact format for LLM models. I made a video showing how to run one of his models on my machine - th-cam.com/video/7BH4C6-HP14/w-d-xo.html. That video uses a tool called Ollama which works on Linux/Mac - th-cam.com/video/NFgEgqua-fg/w-d-xo.html
There is also another library called CTransformers which lets you choose whether to run models on the GPU or CPU. I've found the 7B parameter quantised models work reasonably well even on the CPU. I should probably create a video about that I guess! But in the mean time, this is the link - github.com/marella/ctransformers
@@learndatawithmark thanks!
this video saved my day
That's Great! Thank you!
Super helpful and easy to understand!
Glad it was helpful :)
Awesome
hey um, i don't know if you'll read this in time, but I have a problem:
pytorch_model.bin: 0%| | 0.00/13.5G [00:00
Hard to know exactly why - maybe connectivity with Hugging Face or maybe your internet or maybe the download tool?! You could try going to Hugging Face directly and click through to files and download them directly to see if it helps.
@@learndatawithmark You got to rerun that part couple times and make sure you have a fast connection. For example, pytorch_model.bin is 6.71gb that would take some time to download
hi, thanks for the video. May I ask what's the meaning of legacy=False when using the pretrained model?
it says hf_hub_download isn't defined
What is the configuration of your local environment
If you're getting a wacky error trying to perform `AutoTokenizer.from_pretrained(model_id, legacy=False)`, do pip install protobuf==3.20.1 and restart the jupyter kernel
Good tip! I get that error somewhat randomly but never quite figured out the combination steps that result in it happening!
Amazing and outstanding. This video and presentation is awesome.
Thanks!
I have done as you say, but running the model pipeline is taking forever to work. It still has not worked, please what can I do?
If it's running too slowly then maybe it'd make sense to try out some of the quantised models instead. Those ones are smaller and better suited for running on consumer hardware.
I quite like Ollama and I've made a few videos on that. This is probably the best place to start - th-cam.com/video/NFgEgqua-fg/w-d-xo.html
thanks sir , however I want to know
1- how one can integrate specific set of models (pre-trained) ones in to Rstudio ? so that one can simply run examples on data "proprietary in my case " locally within R
2- is there a way to ask the inference API for tasks different from the typical sentiment classification of text for example "multi-entity tagging" , "modalities" ....etc
your input is highly appreciated
Super helpful 👍
Random idea, but a video on how to run an embeddings model on a laptop would be really cool 😀 Could even combine embeddings + text2text for more specific answers. Or even t5_3b + selenium to create something similar to bing chat. Anyway, wish you luck on TH-cam 😊
Sorry, I didn't see this reply! I've got a notebook with that idea sketched out, so I'll create a video for that soon. On holiday at the moment, but will do it when I get back home!
what that tool you are using called? the broswer ?
jupyter notebook
What is the editor you are using on localhost ?
I'm using a Jupyter notebook in the video
thx
Hey, great tutorial! I also found your blog on this and followed that, but I always get this error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
This is my first time experimenting with LLMs, so any assistance would be greatly appreciated.
Oh I'm not sure about that error - I haven't seen that one before. Since I made this video I've been playing around with another tool called Ollama which I found easier to use. It might be worth giving that a try to see if that works for you?
th-cam.com/video/NFgEgqua-fg/w-d-xo.html
Hi Mark, great video. May I know your notebook and the configuration? I’m thinking switching to MacOS to play around with Gen AI.
I'm using the latest version of Jupyter Lab and I have it set to dark mode with pretty much every one of the views hidden so that I can use as much of the screen as I can.
Not sure if that answered your question, so feel free to follow up!
Thanks Mark for this video. A quick question- Is this safe to pass some PII data to one of the open source hugging face models that require the hugging face API token ? If No, how can this be resolved in deployment so that there is no risk of data leakage ? Please guide through this.
It depends.
If you are passing your HF API token because you're using the HF inference endpoint then your data is getting sent to the HF API.
If you're passing it because you're downloading a model that requires token auth then your data will only be local to where you run the model that you download.
hi Mark - super helpful. can i run all of this in terminal?
You can. You can use a Python REPL or even the iPython CLI. Or you could put it all in a Python script and run that.
thanks Mark, very nice video, super clearly put!
could you please suggest, what could be the reason if (when trying to set the wifi off) the output of those lines of code is "ModuleNotFoundError: No module named utils"?
utils should be referring to this file - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/utils.py - so in theory that's independent of WiFi connectivity. If it can't find that module you could copy/paste those functions into the notebook and use them like that.
I'm probably missing something, but where are you using the downloaded files? you are entering model_id in .from_pretrained(), how is it finding/using the downloaded model?
It's reading from the ~/.cache directory. So it constructs a file path based on that directory & the model id
Hello there Mark i was wondering if i could use this method to download other ai models for example text to image models?
Yes you should be able to use a similar approach. There's a good guide on image to text over here - huggingface.co/tasks/image-text-to-text
I deeply appreciate your video! Although I have a question, does this still works when the model file is a .safetensors or .pth file, not a .bin file? Thank you!
Yeh I think it should work with both of those.
i want to work with a model that is tagged as 'text-generation' how do i run it?
Hi Mark! This video is very helpful, may I ask do you think fastchat can be used in combination with Qdrant for RAG? Thank you in advance
Yeh you can could combine it with any database to do RAG.
Hi, wich webinterface you use for python scripts? I want to use it to :)
This is Jupyter Lab - jupyter.org/
We're going to start by opening??? Start by opening what exactly?
Open a Jupyter Notebook - jupyter.org/
Here's a link to the notebook I used in the video - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/LLMOwnLaptop.ipynb
sorry i'm just starting with this, the code you're writing in the beginning, what is the website called?
Do you mean this one? huggingface.co/
What is the difference w.r.t to using the classical:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
Thanks in advance!
I think it's the same thing under the hood - no need to change from your approach!
Thank you! I finally downloaded a big llama model.. lol 😹
Winning!
really tried to follow along but I'm completely lost. I've never used python I'm assuming I put my own info in the parameters? Do I run each cell after filling it in? i just really have no clue what I'm doing but I've wanted to use hugging face for a while an no tutorial really helped me. it seems like every one expects me to know how to set up virtual environments right off the bat and do everything. if someone could just tell me what im actually putting where or what to run and when it would probably work i think
Awesome content, love your channel! Video is very informative and concise, thanks. As a friendly suggestion, you might want to give a couple of secs at the end for the video for slow people like me to hit that well deserved like button :)
Thanks for your kind words! Let me see if I can figure out a good way to implement your suggestion 🙂
Hey, can you tell me about your system info, i am using mac m3 and its not giving any response and running continuously?
I use a Mac M1 with 64GB RAM. I think it's a 2021 edition. I've found in general that the quantised models work better on my machine - either using Ollama or llama.cpp.
th-cam.com/video/YDj_ScvBpKU/w-d-xo.html
th-cam.com/video/NFgEgqua-fg/w-d-xo.html
How can I use this downloaded model next time ?
It will already be there so if you try to use it again there won't be any need to download it
Has anyone built a GUI for this?
sir if wifi is on then they model is working properly or not?
Yes it should work without wifi - but you will need a connection to the internet to download the model.
Thank you for this tutorial, this tutorial was really useful
Could you perhaps make a video with this very same theme... but assume there are those who don't know the first thing about installing this stuff? This is WAY too complex... why wouldn't there just be an installer (.exe) file you can just download and run, and just like any other program... Bob's you're uncle! It's ready to use?
Hi, i try to find someone who uses GGUF directly and locally without using a .bin to launch it because I would like to launch it under python, is this possible? Or should I do something else?
You can do this using CTransformers like I did in this video - th-cam.com/video/S2thmwdrYrI/w-d-xo.html
I think you might even be able to do it with HuggingFace transformers, but I haven't tried it myself.
@@learndatawithmark if one day you make a video on this, I would like to see it, in fact what I would have liked was to discuss with the model directly with python without going through any interface and to give it a personality with json like we have could do it with webui (but without webui) I tried various methods and honestly I find so little explanation. I had the idea of making my own bot as I saw in "wifu" mode in the sense that it is totally customizable and we give it a personality with a long term memory. The basic idea was to have a small model just for me. I'm just frustrated to see bots that don't even remember talking to us 2 seconds before. xD
@@The_little_black_cat it sounds like you want to keep the history of the chat messages between you and the LLM? I showed how to do this in memory on this video using Ollama, but it can be adapted to another approach - th-cam.com/video/MiJQ_zlnBeo/w-d-xo.html. I can across a tool called MemGPT which I think attempts to solve this problem, but I haven't tried it yet - memgpt.ai/
Great vid, however I’m getting a value error, failure to import transformers error even though I used pip to do that, wondering if it’s a python version issue, I’m using 3.10, wonder if anyone has any ideas ? Thx
Can you share a script with all the code you ran and I'll try to reproduce?
use 3.9 its more stable
your python installation is missing the transformer module 'pip install transformers' will do the trick.
I keep following along until about 12 seconds in, where you start typing into something and you say let’s open up age something, and carry on typing into whatever it is you’re typing into - I can’t get that far, I don’t know what to type into
I'm using a Jupyter Notebook, but the code would work in any Python environment or script
jupyter.org/
@@learndatawithmark ah thanks, that’s interesting - I’ve never heard of that
i personally found disabling your wifi from a jupyter notebook to be bad ass
Haha, thanks. It took me a little while to figure out how to do it!
So many steps missing in this video...
Bro I need to make offline chatbot using llm how to implement it please tell
hi. please help me. how to create custom model from many pdfs in Persian language? tank you.
Seems unnecessarily complex... isn't there like an online space to use this stuff without having to write a bunch of stuff just to download it?
Yeh I think with a bunch of the models you're able to run them on the Hugging Face website on the right hand side of the page. And then in general there are many services that offer APIs that you can call. The approach describe in this video is only for if you don't want to use those services.
Pinned
When I run this: "os.environ.get("HUGGING_FACE_API_KEY")" I get"None". Is it normal?
I guess I need to set the HUGGING_FACE_API_KEY variable to my token beforehand.
You would need to set that environment variable before running your Python environment otherwise yeh it'll be none