Set up a Local AI like ChatGPT on your own machine!
ฝัง
- เผยแพร่เมื่อ 1 ต.ค. 2024
- Dave explains the reasons why and the steps needed to set up your own local AI engine ala ChatGPT. For my book on the autism spectrum, check out: amzn.to/3zBinWM
Helpful links and tips:
Install ollama:
curl -fsSL ollama.com/ins... | sh
ollama serve
WSL Kernel Update Package: learn.microsof...
Run the UI Docker Container:
Run UI with Docker:
docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama
Dell Price: $35,594.36
Shipping: Free
Thank God the shipping is free.
"open the garage doors, HAl."
"I'm afraid I can't do that, Dave."
Open source ai models are more like Jarvis than Hal.
@@lonewitness just watch out for The Riddler
Dave? Why are you doing this, Dave?
@@javabeanz8549 nope it is the candyman for special Art-E-Fish'ale Philantrophy's.
did anybody seen karl marx at the sicknuts from disney ?
Lots of cleverness in these 2 lines... Well played Mr. fake W
I’m cool with living vicariously through Dave.
Installing docker via snap caused me issues with not being able to access my Nvidia graphics card to run the AI.
I believe this is because I'm running an "unsupported" Linux version.
Installing docker via "apt" fixed this.
So, for everyone who is struggling to get the wsl 2 set as your default, you need the command "wsl --set-version 2" in your PowerShell. Spent an hour figuring it out and know this will save a few headaches. Thanks for the video Dave. I hope to get this operational before bed.
New? It's been out for over 4 years now ;p
@@JustinEmlay Thanks for pointing out the spelling error, I typed that late into the night.
How do we link the Ollama AI to the OpenAI, seemed to skip that part?
So, we’re just casually summoning AIs at home now?
Been doing that for a while. Not too hard. The largest problem is having a good enough model to run that runs on what is a reasonably priced home computer.
lol I love this in this context
That's amusing, "saying, casually summoning ai models" like requesting your own personal butlers to remove the plates off the table once you are done eating. Or like using Uber, where is my ride "alexa" I called for it 30 minutes ago 😊
Daemon, not a demon.
"It's the work of the devil" - Mama Boucher
No it's not. It's just zeros and ones. On's and off's.
Same voltages and vibrations, vibes and grooves as the rest of the universe. The oneness is us. Oh eye see.
Summon sounds too archaic. Wait, not archaic enough.
We invoke them.
Why use WSL? Olllama has a windows installer.
I've been using LMStudio which also runs natively in Windows.
And just like this, 13minutes lead to an evening of successful tinkering. Thanks for the inspiration!
Glad to see some of us have 50K worth of hardware at hand 😅
@@heyheyhophop lol, just use your gaming machine, it still works quite fast. I certainly don't have the same beast of a machine.
@@eugene3d875 Right, was kidding, I hope my 12GB 3060 and 48GB of plain RAM should let me go relatively far -- esp. as now the layers / whatever can be partially offloaded to CPU, as far as I understand
@@heyheyhophop I found that 48 GB would be sufficient. I'm running inference on CPU only, due to incompatible graphics card, and it still performs quite well, while keeping the total RAM load under 32gb. So I think your setup will do great
@@EugeneShamshurin Many thanks for letting me know
The powershell command is wslinstall
Thanks!
Thank you, I found this after it didn't work, and it still didn't, but then tried "wsl --update" which then started the install.
... as I notice it says on Dave's very next slide!
comments coming to the rescue ;)
...and if you're having issues running that large Docker command copied from the video description, it has "sudo" missing from the start of it... and make sure you run it from the initial wsl command line rather than any other you may have opened.
Dave, As a retired Deccie, I love your appreciation of the pdp11. Worked on many pdp 11/34's way back when. Oh! and love the shirt.
Thank you. This helped me create my current girlfriend.
I had a feeling this was the Ollama model - I can verify as a Linux user that the install for this is as simple as installing ollama from the package manager / flathub; then running the two commands; ollama serve, then 'ollama run' - which automatically fetched the repository if it is not already there...
Two *very useful* commands within the chat interface are /load and /save. You can keep your AI 'alive' and contextually relevant by saving it before exiting.
5 minutes is my average prompt time, if anyone asks...
Sweet PDP11 shirt.
Just thinking the comparable size of a digital pdp11 to that thread ripper unit.
Please make more videos about this subject.
For those that don't know, (if you installed Debian), replace 'snap' command with the 'apt' command. BTW- Debian does all of this quite well also.
I believe you can use apt and snap both on Ubuntu, since it's Debian based, and snap is developed by Canonical.
@@JimmyS2 - use Mint, so snap isn't used.
I LOVE YOU! Stay awesome!
Snap didn't work but APT did! Now getting the following error after trying the web-ui command.
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Asked llama and chat-gpt for answers but they recommend systemctl commands that do not work on WSL 2 Ubuntu (I think). Is there any solution? (Really grateful for your time!)
This is brilliant Dave. But you are underestimating your own expertise and some of the configurations you already have in place on your machine - or simply do automatically. So lots of failures and error messages. But if you are an ageing weird computer nerd like me it's fun sorting it out :-) Thanks.
Sitting on an airplane with a laptop or in your shop next to a threadripper is exactly the same noise level.
Your computer’s specs make me want to cry.
Just wait a couple of years and it’ll be commonplace. Of course you’ll still be upset by his future setup, but your computer will be this powerful.
@@robbybobbyhobbiesMakes me want to wait a few years so ai compatible tech can mature and become cheap
@@alok.01 I saw some news somewhere that the cost or depreciation of AI development was like 97% which made it the fastest dropping market in history, so we're going there 😅
Just casually throwing out a 13min video that can completely transform your life and business... that's so Dave.
Well he is on the Spectrum so does this all the time, no big deal , ha ha. My family has no Idea why I get so happy using Ollama on my Pi5.
@@FlintStone-c3s Ollama on a Pi5? You are either a very brave or a very patient person.
I need to do some reading, but what are people actually using it for? I can't think of what I might ask it to do.
@@BastetFurrysays nothing, its all about the model
@@craigknights You can ask it what to ask
Yes. So glad you're covering this topic. Being able to use the files on your own PC without having to upload those files to other companies servers
That was excellent and I was able to get it up and running just like your instructions concisely provided. After trying it for several hours, I can say that it isn't a bad language model at all.
Can you remove the adult content limitation?
Dave, would you be willing to do a follow up video outlining a few lower levels of hardware? You don't even have to run the model on them (although that would be awesome), but describe some machines say in the 1k, 5k, 10k, and 25k range?
I’ve been running this exact setup in Linux (Pop!OS) for a couple weeks now, and it works fine on any modern (eg less than 6 years old) hardware. Initially I tried it with just my CPU (Ryzen 5800x), and it was fine albeit a little slow. But definitely usable. After I enabled GPU acceleration on my nVidia 2070 Super, the responses came back stupid fast. Like 10x faster than I could read them.
The only thing I would note is that either CPU or GPU (whichever one is enabled) is going to be pinned at 100% utilization while responses are being generated. The practical effect is going to be not trivial power draw, and for laptops much shorter battery life unless the unit is plugged into the wall. But don’t let that put you off. Even modest hardware ($1,000 PC brand new 5 years ago) is more than sufficient. Just be aware of your battery level if you’re going to do this on a laptop.
Ollama recommends a 10th gen i5 CPU or AMD equivalent and a NVidia GPU 20xx with 8GB or more VRAM.
Make sure your windows drive or host drive has enough diskspace as the models can easily rack up 100-400 GB at out of nowhere.
I spent 5 hours trying to Google this...to find no real answer. Then this. Thanks.
not sure just how much work it would have taken but with everything running through docker it likely is easy to test it on a lower end machine. While I think it's really cool to see performance an a machine that i may have something comparable to in 15-20 years... it would also have been informative to see the performance on anything close to normal consumer levels of performance as a comparison.
I'm running it on (don't laugh...it's paid for!) a Dell Opti790 with 32GB RAM (yes, you CAN install that much) and a poor ol' RTX 1070Ti vid card and it works like gangbusters! The graphic card is modestly overclocked. Turned off the overclocking and I'm not sure if it ran with just a slight performance hit or maybe that was just my imagination. Point is, although there's no doubt that the Threadripper would leave my system PAINFULLY smoked in the dust on more intensive work, the basic chat works great as it. For something that's isolated from web learning, I was very surprised in it's breath of knowledge (it even knew what WSL is...) A roughly 8+ year-old machine will run this configuration just fine...!
Yep, can confirm that it runs really well on my older AMD Ryzen 3600 with 32GB RAM and a 2070 GPU.
And just like that I've got an AI running locally on my machine. Feels kind of weird tbh. Awesome video guide, thanks a lot!
You can't load the open-webui container with the snap version of docker. Kinda disappointed in you Dave.
I did everything though I had to install docker for windows and use it's WSL integration, I can display the web gui but there is no model available there whilst it's working in the CLI.
Would have been funnier if the question was "What is the airspeed velocity of an unladen swallow?'"🤭
When I run the docker command, after downloading, it gives me the following error:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.
Same here. Would love to learn how to fix this.
Same, frustrating. Any ideas anyone?
Remove the --gpus=all parameter in the docker command. I have this running on a VM in proxmox using just the CPU and it fixed my issue.
@@jqoutlaw Thanks. That worked.
In my casual sleuthing of the problem, it looked as if the NVIDIA gpu had something to do with it. I found that I have an AMD Radeon 780M gpu, so I looked to see how to run it with that, but none of the solutions I found worked.
So I guess I'll just run it using the cpu instead.
My guess is that it's complaining about not finding the NVIDIA CUDA toolkit (only works if you have an nvidia gpu as @jqoutlaw mentions). Also do an update/upgrade: 'sudo apt update; sudo apt upgrade'
Wouldn't it be easier/better to take a machine and just run Linux as a default OS without using a virtual machine and sharing resources with your main OS?
Yeah it is. WSL 2 has many disadvantages if any app doesn't support it. And to my knowledge only vscode is supported other than the terminal ofcourse
This is pretty cool. I installed it on a Lenovo laptop, Windows 11 Home, 13th Gen Intel i7-1355U,10 core, 16GB ram, SSD. Runs decent enough to experiment with. I am only running from a command prompt.
Already running InvokeAI (Stable Diffusion) and text-generation-webui (for Llama) for months locally. This is the first year I especially bought a GeForce gfx card (with 16GB RAM) not primarily for gaming but for Generative AI. The times they are a-changing. ;)
Stable diffusion runs on Pi5 8GB, a bit slow, 3 minutes per image. Hoping the Hailo AI Hat can run it faster.
I personally really like using LM Studio. It does everything, including downloading and loading models in a simple UI
How can we be sure that Dave made this video, and not his AI model?
That's what a bot would say!
@@DavesGarage sounds like something a synth would say.
Thanks Dave, I have a couple RHEL Linux systems at work I shall try this.
For those with less hardware, the Llama 3.2 3B Instruct model is good for chat and requires a lot lower specs to run. I am able to run it on GPU using an Nvidia GTX1070Ti with only 8GiB of VRAM. So far it has been on par with Llama 3.0 & 3.1 for my use while being a lot faster. To get it, run the following: ollama pull llama3.2:3b-instruct-q4_K_M
Thank you very much for the video. FWIW, your final incantation to launch the web UI didn't work on my system - something to do with packages installed with snap not being able to see the graphics card drivers (?) Following the installation instructions on Open Web UI's page seemed to do the trick though.
Thanks - now pretty please download the most ridiculous model available (maybe Llama3.1:405b ?) and show us what killer hardware can do !
Did your wife divorce you for buying that machine? LoL
It's a loaner! I can't afford that kind of hardware!
thank you. Decentralized, uncensored, 100% private, etc. AI/AGI really is important.
‘The path to hell is paved with good intentions’ is a quote that comes to mind when I hear governments, corporations, etc. trying to limit freedoms of individuals. AI is something that is too important to not have individuals be able to have absolute freedom over their own AI’s/AGI’s.
Well, toss aside the speed/hardware/cost issue. That's something you can grow into or even Moore's-Law your way into (i.e. be patient).
The *real* issues are: dialog & scripting with persistence that's even forkable, so that you can maintain contexts just like people maintain repositories and even do multi-party engagement with it. That's especially the case, if you're going to put up a public-side interface to this.
That's a primitive form of knowledge-base integration, which gets to the next item: real-time updating and learning, not just training by some batch process at "update time". That's not trivial, and it's an issue that is *independent* of the scaling issue alluded to in the first sentence: how to integrate short-term memory into the long-term memory that is the model, itself. Trainability is also an important issue, yes, but I'd be more concerned with the ability to interface with components for a hybrid architecture that includes a logic and math engine and knowledge-base engine.
The "advantages" that the greater resources put into the major AI-providers' models diminish exponentially because of the neural scaling law, so you can go a long way to getting into the same ballpark as them, without the blow-up up in resources that they have or used to get there. Hybridization could blow through that wall, slingshotting right past the big players, if it's done right - in a race to get there before they do. A model of your own is good, but you really need hooks into these other things to go with it, or you're just cosplaying OpenAI in the minor leagues. I want to move this to a more modular form, actually, as curriculum training; and also to mold a personality type. An already-provided pre-trained model is just a starting point to launch this off of, but only if the extra hooks are integrated into its design.
I'm sorry Dave, I'm afraid I can't do that!
Thanks!
This is the tutorial I didn't know I needed, until now.
Even though I've been running local inference and RAG for about a year now I still stopped everything to listen to Dave's explanation.... Because Dave..🕺🤖
What I wanna know is if I can get an AI on my steam deck that I can use as a personal assistant computer for creating Contant strictly based on all of the data and information I create I just put all my information inside of a single file of course it’s not gonna be a single file. There’s multiple different layers of stuff. I’m going to put it of course information but I just wanna be able to give like a custom GPT or llama three all this information and just have it so that it’s a personal assistant for one topic alone, but it’s an expert in that topic.
Docker UI Container command: You'll probably need to add 'sudo' at the start
It would be great if you could talk to it and activate it using your voice, just like Siri.
That's what I'm waiting for. 🤞
All the Linux nerds that see wsl and hear windows second language because Linux is primary.
Wow, Dave! I had no idea. I don't have the beast machine you have but I'm going to put this to use. Thanks.
Hey Dave, as an former employee of Digital Equipment Corp for over ten years, I love the t-shirt.
Thanks, but how do you add your own information like information about your own large code base?
Great vid...now all i need is to free up 5Gb of space XD
Hello Dave, can you share the link where you got the t-shirt ?
Great tutorial. Would like see to one on how to create a custom model or add training to an existing model.
Love the shirt, was there slightly after PDP days but got to see the old Alpha starting with EV54. You made this too easy to install on a laptop :)
I've been running LLM locally for a while. The most useful thing i made it do was create weather reports from weather data from NWS.
I have 2 rtx 4060 ti (16gb each) and my old rtx 2060 (6gb) in my server. Its really just a gaming desktop moonlighting as a server, it can run a 70B model decently though.
How do you like the complexity of the answers you get? Really annoyed openai is forcing people to use their hardware especially since most people use chatgpt for personal use. Is it worth it to setup something locally?
@@organicdinosaur5259 its pretty good honestly. I'm running a Q4 model (Llama-3.1-70B) but despite that its rather accurate. For me personally, I say its worth it. mostly because I can just download a random model then throw it on the server so you're not just stuck with chatgpt. Qwen is a really good general model, comes in a bunch of sizes.
You can also run most 8B and lower models on CPU at a pretty brisk speed, but its better if you have a GPU with at least 8gb of vram. ollama is super easy to setup on both windows and linux so its absolutely worth at least giving it a shot.
Amazingly, I have an Arc A770 and it comes with an AI playground so I could build my own graphic maker and chat system.
I run into this every time i try to install docker: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown. :(
same here
You are absolutely the best at explaining everything so simply and well! Thanks for another great clip!
Dave, getting close to 1mil subscribers!
8:30 Dave flexing 388MB/s downloading
Oh this is interesting! Please more of this, Dave!
I've been waiting for a straightforward tutorial like this to share after setting one up myself! Mistral has an amazing 12B small model that most GPUs can run.
I'd like to add that ollama plays very nice with the Continue VScode extension which means....private local github copilot too!
This is incredible, exactly what I have been looking for! ❤
Next video: how to train your own model!
Didn’t understand a word but it was interesting
braincell explosion in my head
Just a little note: the wsl arguments like "install" require double dash like "wsl --install" in the video you show one dash for type I guess
I bet you’re gonna hit 1M subs with this one! Deal for the next video on how to train your local AI? Also, how much power does that beast of a workstation pull? Can't to see how long it'll take to get a response on a human desktop...
I've been using Ollama on my PC for a while now (I opted for a Windows install with AnythingLLM as my front end...easy and no Linux needed). It's pretty good over all. Not bleeding edge. Maybe not even cutting edge. Regardless, it does fine if you pick the right model(s) for your needs. It definitely wants to stretch its legs, though. More disk space (for larger versions of models) and more CUDA cores (speed, baby) are definitely more better.
Great informative video. I used the "cheap" and almost premade option of an Intel Arc A770 with their AI Playground program.
Got inspired.Thanks.
Thanks for this. I've been running LM Studio on my windows box and experimenting with a few different models. Was looking for inspiration to build a docker based AI server, and this hit the spot.
Or you can just install Msty :)
‘Do you want to play a game?’
Finally! I've been looking for a way to learn to make a GPT AI to consume rulebooks and modules for Old School Essentials so I can ask it questions and generate random encounters.
How about you make this all in your OS instead of explaining? Would have been better. Do this and that is not really beginner friendly.
Openwebui is the bees knees. I have every API and local model running through it for the past month and i just love it. I don't use docker rubbish though, much easier install on Linux
Love the "digital" t-shirt
Another great episode
In the early 80's I had a program called 'Whatsit?' maybe you know it, it was a early learning piece of software running on CP/M that did the same, it learned as long as the things you put into it. AI is based on these efforts. Tried later to make a similar program in AmigaDOS with a friend and it was fun. Just text base.
There is no way you made anything like LLMs on a CP/M.. Imagine thinking you invented something on a computer that could barely have enough ram for a primitive OS and that has only been in research for the last 10 years.
That program's ability to learn was really a simplistic classifier, in that it had limited scope and could not really learn
AI is based on a lot of things that have gone before us, Whatsit included, although it was more that Whatsit was based on other fundamentals in it's time
@@TheBodgybrothers
LLM's have been around as a concept well beyond 10 years!
There have been LLM's that ran in a batch mode that were large but because of the limits of systems and ram, they were so slow as to be unusable but they existed
In regards to MrKiilerno1, of course he didn't run an LLM on a CP/M machine, I don't think that was his point, he was referring to a system that engages in a conversation and that tracked that conversation across multiple interactions, in this regard, he is correct, role-play games and learning systems have been using this for decades now
In terms of Ram, there are older OS's who can easily run programs beyond the constraints of their system memory, OpenVMS for example has both swapping and paging mechanisms. OS's now focus on speed so they demand more memory rather than use concepts like swapping to run extremely large programs but systems of old use to run large application in very limited memory. I worked on one that had 16K of memory and ran a whole accounting ledger for a large municipality of over 1 million people
@@stultuses And still I had a lot of fun with it. Most days when I talk to Alexa, Siri or Google, they tend to do sometimes what they want. I appreciate the way to communicate with them by voice, this on itself, I think, is a masterpiece in programming.
@@TheBodgybrothers That time I was working for a company that made medical database software, to store all their data about medicine and patients in. It had to be big machines, they were very pricy at the time and had a large storage device on them, harddrives. It was also the time 16 bit computers were coming and Microsoft took hold of many branches. Luckily this software evolved into the current database system it is these days. It all started by one man and his machine, selling his product and hardware to numerous institutions and health practitioners (doctors offices). As long as you had the thousends of dollars, you could buy it.
My wife would kill me.
I stay away from snap
Ollama has Windows client
I would hug you if only for the amazing person you are
Wow, THANK YOU! Great video again! Would be interesting how much faster YOUR SETUP is compared to my 10 core Laptop, 32GB, and a 4GB 3050 . Right now it kinda crawls on most questions, but one can always come back 10 minutes later. The amazing thing is IT DOES give answers standalone.
Still pointless then... I'll keep using CoPilot for now..
I bet your issue is the 4GB of VRAM on the GPU. I ran it on just my CPU (5800x, 8 core Zen 3), and responses took less than a minute with no GPU acceleration. You might get better performance by cutting out the GPU entirely and letting it run just on the CPU so the model doesn’t have to load into VRAM piecemeal on every query.
Nice video, but a few parts i did miss:
Installing nvidia driver support in wsl ubuntu for docker. Without that it will not use GPU acceleration!
You also did miss the part that to run docker as non superuser you need to configure permissions of docker (adding non superuser to group)
Alternative manual installation of docker in wsl ubuntu would be nice snap did not work for me
love the shirt
I want your Digital tee Dave. "Best job I ever had..."
Fast Hardware Working Hard. Me too!
Future me installing this on my $100 quantum computer set-top box and thinking how quaint Dave looked installing this on a $50k server. Now, if only I had a time machine. Thanks Dave, really cool!
Don't have Dave's budget? I run LLM's on Raspberry Pi5 8GB. Some tiny LLM's will run on the Pi4 4GB. About to order the Hailo AI HAT as they are now available Downunder.
can you make the same for the Mac and also what models are best to use for what purpose?
And if you run Linux, you can skip the whole WSL install part :) This is great... hopefully it runs on my laptop.
Sorry, Dave. Respect for the era you were involved with MS, but I bailed at your first mention of Windows. Nope. Never again. My only involvement with that mess is fixing the problems it constantly creates with my elderly parents' systems.
Are you going to do a subsequent video that compares performance of this high end loaner with you're personal computers and something we might actually have (say
Great stuff Dave. Love your channel
Dave' gone in a really round about way to run ollama, without much explanation. For 99.99% of you just install ollama directly into windows without going through WSL. WSL is good for some things rather than others. But why spending 30 minutes setting up WSL to run when ollama can be installed in 30 seconds?
Does this only work with NVIDIA cards?
No. It uses the GPU on my Mac quite nicely, even!
On Debian it configures automatically and works with AMD ROCm as well, at least on my W7900 pro card with 48GB VRAM.
Lost me at the 50K machine.