Running a Hugging Face LLM on your laptop
ฝัง
- เผยแพร่เมื่อ 3 ส.ค. 2023
- In this video, we'll learn how to run a Large Language Model (LLM) from Hugging Face on our own machine.
Blog post: www.markhneedham.com/blog/202...
Notebook: github.com/mneedham/LearnData...
Other videos showing how to run LLMs on your own machine
• Running Mixtral on you...
• LLMs on your own machi...
• Running Mistral AI on ...
• Hugging Face GGUF Mode... - วิทยาศาสตร์และเทคโนโลยี
![ภาพนี้ก็ฮาเหมือนกันนะเนี้ย #2 SS8 [ พากย์นรก MEME.EXE ] | easy boy](http://i.ytimg.com/vi/RMhUgxxD49Q/mqdefault.jpg)
You explained completely and perfectly without wasting the audience's time! well done
Thanks!
except for the part that i have got no clue what the required software to start is. what os to run on nor how to run the code in step 2. the farthest i got was git bash for the transformer and thats about it. i just want to run some local ai, why does it have to be so obtuse??
@@norwegiansmores811 It's running in a Jupyter notebook - jupyter.org/ but you can run the code anywhere that can run Python code e.g. a Python REPL or a Python script.
@norwegianmores811 I think the easiest way to run local AI as of June 2024 is now this - lmstudio.ai/
Thank you for the contents.
Awesome! Thanks for this video.
Thanks for this video!!
Super video!
Super helpful and easy to understand!
Glad it was helpful :)
That's Great! Thank you!
This was an extremely informative video. Really appreciate it.
Thanks, glad you enjoyed it!
thank you very much for this great explaination
Thank you very much, you helped me a lot
Absolutely wonderful video! to the point and well explianed! way to go! thanks a lot!
Thanks - very kind of you :D
this video saved my day
thx
thanks sir , however I want to know
1- how one can integrate specific set of models (pre-trained) ones in to Rstudio ? so that one can simply run examples on data "proprietary in my case " locally within R
2- is there a way to ask the inference API for tasks different from the typical sentiment classification of text for example "multi-entity tagging" , "modalities" ....etc
your input is highly appreciated
Super helpful 👍
Random idea, but a video on how to run an embeddings model on a laptop would be really cool 😀 Could even combine embeddings + text2text for more specific answers. Or even t5_3b + selenium to create something similar to bing chat. Anyway, wish you luck on TH-cam 😊
Sorry, I didn't see this reply! I've got a notebook with that idea sketched out, so I'll create a video for that soon. On holiday at the moment, but will do it when I get back home!
thanks Mark, very nice video, super clearly put!
could you please suggest, what could be the reason if (when trying to set the wifi off) the output of those lines of code is "ModuleNotFoundError: No module named utils"?
utils should be referring to this file - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/utils.py - so in theory that's independent of WiFi connectivity. If it can't find that module you could copy/paste those functions into the notebook and use them like that.
What is the configuration of your local environment
Hi Mark! This video is very helpful, may I ask do you think fastchat can be used in combination with Qdrant for RAG? Thank you in advance
Yeh you can could combine it with any database to do RAG.
i want to work with a model that is tagged as 'text-generation' how do i run it?
An API key is not needed if the model is downloaded and run locally.
So long as you have a behemoth of a machine. 16GB Ram = 100% memory usage 😭😂
I don't think anyone can afford an expensive laptop lol
Do you know how to run it on live servers!! How to get?
It is needed to download the model!
What is the difference w.r.t to using the classical:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
Thanks in advance!
Awesome content, love your channel! Video is very informative and concise, thanks. As a friendly suggestion, you might want to give a couple of secs at the end for the video for slow people like me to hit that well deserved like button :)
Thanks for your kind words! Let me see if I can figure out a good way to implement your suggestion 🙂
Hey, great tutorial! I also found your blog on this and followed that, but I always get this error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
This is my first time experimenting with LLMs, so any assistance would be greatly appreciated.
Oh I'm not sure about that error - I haven't seen that one before. Since I made this video I've been playing around with another tool called Ollama which I found easier to use. It might be worth giving that a try to see if that works for you?
th-cam.com/video/NFgEgqua-fg/w-d-xo.html
Hi Mark, great video. May I know your notebook and the configuration? I’m thinking switching to MacOS to play around with Gen AI.
I'm using the latest version of Jupyter Lab and I have it set to dark mode with pretty much every one of the views hidden so that I can use as much of the screen as I can.
Not sure if that answered your question, so feel free to follow up!
I deeply appreciate your video! Although I have a question, does this still works when the model file is a .safetensors or .pth file, not a .bin file? Thank you!
Yeh I think it should work with both of those.
Thank you! I finally downloaded a big llama model.. lol 😹
Winning!
Has anyone built a GUI for this?
hi Mark - super helpful. can i run all of this in terminal?
You can. You can use a Python REPL or even the iPython CLI. Or you could put it all in a Python script and run that.
Thanks Mark for this video. A quick question- Is this safe to pass some PII data to one of the open source hugging face models that require the hugging face API token ? If No, how can this be resolved in deployment so that there is no risk of data leakage ? Please guide through this.
It depends.
If you are passing your HF API token because you're using the HF inference endpoint then your data is getting sent to the HF API.
If you're passing it because you're downloading a model that requires token auth then your data will only be local to where you run the model that you download.
I'm probably missing something, but where are you using the downloaded files? you are entering model_id in .from_pretrained(), how is it finding/using the downloaded model?
It's reading from the ~/.cache directory. So it constructs a file path based on that directory & the model id
Hello there Mark i was wondering if i could use this method to download other ai models for example text to image models?
Yes you should be able to use a similar approach. There's a good guide on image to text over here - huggingface.co/tasks/image-text-to-text
What is the editor you are using on localhost ?
I'm using a Jupyter notebook in the video
Hi, wich webinterface you use for python scripts? I want to use it to :)
This is Jupyter Lab - jupyter.org/
I am getting an error/ info log from transformers (twice) stating "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained." The model then generates only a bunch of whitespace, no matter the input. I have followed through your steps and made sure the files were downloaded at the expected location. The behavior occurrs both with and without setting legacy=False.
Does it work after you see that message?
@@learndatawithmark getting the same error and hang when running in colab or locally. seems like the pipeline("my query") never resolves
Same thing happens to me! Hanging and giving random replies! I am using mac m3
Hey, can you tell me about your system info, i am using mac m3 and its not giving any response and running continuously?
I use a Mac M1 with 64GB RAM. I think it's a 2021 edition. I've found in general that the quantised models work better on my machine - either using Ollama or llama.cpp.
th-cam.com/video/YDj_ScvBpKU/w-d-xo.html
th-cam.com/video/NFgEgqua-fg/w-d-xo.html
sorry i'm just starting with this, the code you're writing in the beginning, what is the website called?
Do you mean this one? huggingface.co/
If you're getting a wacky error trying to perform `AutoTokenizer.from_pretrained(model_id, legacy=False)`, do pip install protobuf==3.20.1 and restart the jupyter kernel
Good tip! I get that error somewhat randomly but never quite figured out the combination steps that result in it happening!
i personally found disabling your wifi from a jupyter notebook to be bad ass
Haha, thanks. It took me a little while to figure out how to do it!
1:03 - Thanks for this clarification. I'd done quite a bit of Google searching and scouring the Hugging Face website for this information. I found nothing of value. I'm a computer enthusiast / gamer and not a professional machine learning engineer. Since embarking on running an LLM locally on my previous daily use desktop, I've noticed its near impossible to find a model's resource needs. GPT4 says a 7b parameter model would consume about 48 GB memory. I asked it what size model would fit in my 12 GB Nvidia 3060, it said about 3.2 billion. My question for you is, why is it that everyone in this space who seems to offer a model (or talk about them) never includes something like a system requirements descriptor? Is it one of those situations where, if you need to ask, you probably don't have enough resources? Thanks for any insight you can give on this phenomenon.
My impression is that most of the models being created are assuming that you have insanely good GPUs to run them on!
Since I created this video, there's been a lot of work done by a guy called TheBloke on Hugging Face to 'quantise' the models, which effectively means that the amount of resources required is reduced, but the quality of the model is slightly reduced too.
I've found those models work a lot better on my laptop.
The Bloke is using a format called GGUF, which is kind of a defact format for LLM models. I made a video showing how to run one of his models on my machine - th-cam.com/video/7BH4C6-HP14/w-d-xo.html. That video uses a tool called Ollama which works on Linux/Mac - th-cam.com/video/NFgEgqua-fg/w-d-xo.html
There is also another library called CTransformers which lets you choose whether to run models on the GPU or CPU. I've found the 7B parameter quantised models work reasonably well even on the CPU. I should probably create a video about that I guess! But in the mean time, this is the link - github.com/marella/ctransformers
@@learndatawithmark thanks!
Hi, i try to find someone who uses GGUF directly and locally without using a .bin to launch it because I would like to launch it under python, is this possible? Or should I do something else?
You can do this using CTransformers like I did in this video - th-cam.com/video/S2thmwdrYrI/w-d-xo.html
I think you might even be able to do it with HuggingFace transformers, but I haven't tried it myself.
@@learndatawithmark if one day you make a video on this, I would like to see it, in fact what I would have liked was to discuss with the model directly with python without going through any interface and to give it a personality with json like we have could do it with webui (but without webui) I tried various methods and honestly I find so little explanation. I had the idea of making my own bot as I saw in "wifu" mode in the sense that it is totally customizable and we give it a personality with a long term memory. The basic idea was to have a small model just for me. I'm just frustrated to see bots that don't even remember talking to us 2 seconds before. xD
@@The_Little_One_Of_Darkness it sounds like you want to keep the history of the chat messages between you and the LLM? I showed how to do this in memory on this video using Ollama, but it can be adapted to another approach - th-cam.com/video/MiJQ_zlnBeo/w-d-xo.html. I can across a tool called MemGPT which I think attempts to solve this problem, but I haven't tried it yet - memgpt.ai/
Great vid, however I’m getting a value error, failure to import transformers error even though I used pip to do that, wondering if it’s a python version issue, I’m using 3.10, wonder if anyone has any ideas ? Thx
Can you share a script with all the code you ran and I'll try to reproduce?
use 3.9 its more stable
your python installation is missing the transformer module 'pip install transformers' will do the trick.
sir if wifi is on then they model is working properly or not?
Yes it should work without wifi - but you will need a connection to the internet to download the model.
hey um, i don't know if you'll read this in time, but I have a problem:
pytorch_model.bin: 0%| | 0.00/13.5G [00:00
Hard to know exactly why - maybe connectivity with Hugging Face or maybe your internet or maybe the download tool?! You could try going to Hugging Face directly and click through to files and download them directly to see if it helps.
@@learndatawithmark You got to rerun that part couple times and make sure you have a fast connection. For example, pytorch_model.bin is 6.71gb that would take some time to download
How can I use this downloaded model next time ?
It will already be there so if you try to use it again there won't be any need to download it
under the "..." in huggingface there is a "clone this repo" which copies all stuff onto your PC. seems simpler to me.
Probably works for this one but sometimes there will be multiple different versions of the same model and it'll take up all your free space if you do that!
I keep following along until about 12 seconds in, where you start typing into something and you say let’s open up age something, and carry on typing into whatever it is you’re typing into - I can’t get that far, I don’t know what to type into
I'm using a Jupyter Notebook, but the code would work in any Python environment or script
jupyter.org/
@@learndatawithmark ah thanks, that’s interesting - I’ve never heard of that
hi. please help me. how to create custom model from many pdfs in Persian language? tank you.
So many steps missing in this video...
Huggingface is a single point of failure.
An index of alternative download URLs
for LLMs is needed for when Huggingface
is down, such as now (Feb 29, 0200 GMT)
Pinned
Seems unnecessarily complex... isn't there like an online space to use this stuff without having to write a bunch of stuff just to download it?
Yeh I think with a bunch of the models you're able to run them on the Hugging Face website on the right hand side of the page. And then in general there are many services that offer APIs that you can call. The approach describe in this video is only for if you don't want to use those services.
When I run this: "os.environ.get("HUGGING_FACE_API_KEY")" I get"None". Is it normal?
I guess I need to set the HUGGING_FACE_API_KEY variable to my token beforehand.
You would need to set that environment variable before running your Python environment otherwise yeh it'll be none