try AIJoel - Multi Generator: text, code, image (create sticker, remove image background, add color to black white image, image to video, logo design) and (music and video are in beta mode)
One new SSD and a full install later.... It works!! Docker was giving me an error which I couldn't resolve but it was all straightforward on a clean installation!
Hey bro I just want to say that you grind is on another level. So good that you can go so for many years and sitll create new channels and succeed. Nice transition to AI content and your views are great too. Hope you get a lot of sponsorship and affiliate deals.
PC USERS IF YOU ARE GETTING THE WSL ERROR WHEN RUNNING DOCKER - Enable virtualization in BIOS: This process varies depending on your computer manufacturer and model. Generally, you'll need to restart your computer and enter the BIOS settings (often by pressing F2, F10, or Del during startup), then look for an option related to virtualization or VT-x and enable it.
@@SkillLeapAI No I understand, and it makes perfect sense. Just as someone who's used a terminal at school and at work for the past decade... the delivery of line just tickled my funny bone
the really interesting part around 07:15 is that in a few months computing power and size of these models will make it possible to run them on practically anything. when they get more effcient we'll have them in our phones.
No joke. I've always hated Docker. Hated everything about it. However, seeing you run the Openweb UI command and it just randomly showed up in the Docker app is making me reconsider. Holy moly! Might have to reinstall it on my Raspberry Pi 5.
Not private, its a hybrid model, caches data when offline then when connected back you can see it sends data. Use wireshark it will show you its sending data
This is an important comment because many channels are saying OpenWebUI and Llama3 is private to your machine. Is there any way to turn off the cache sending process? I would like to analyse documents that are not permitted to be publicly uploaded
Llama3.1 8B runs much faster than you can read on an RTX 2060 Super. Not dirt cheap, but I built my PC for $1.4K 4 years ago - should be cheaper now, and I built it for gaming, not AI. You can host it on a desktop and connect your other devices with a VPN like Tailscale. Don’t spend extra money for less performance by going for small form-factor/laptop.
I have a 9 year old windows computer that runs 13b and 8b models the same speed as your brand new $5000 Mac 👀 it was purchased at a Walmart for $270 in 2015 😂
@bassamel-ashkar4005 As a machine learning engineer and master developer - over 25 years now, I must say you need to learn a bit about inference to speak. 8b can even run on edge devices like orange pi or Raspberry Pi. When models are converted to rknn for such devices, you can achieve greater speeds than base inference. Just replacing the matmul function in transformers with a more optimized function (that uses less 'compute') a model can generate 3x more tokens/ second. Pruning downstream layers and using sporadic attention, a KernelAttentionNetwork or even a KAN (not the same) in place of transformers can gain greater speeds. Until you can design an LLM architecture from scratch...shhh... Not to mention our novel model architecture using stock CPUs can generate >3000 tokens per second - on base mobile phones with only 4GB RAM and an 8 core processor (cost of such phone $100-400). We have demonstration after demonstration available online and social platforms like LinkedIn- where we discuss the architecture advancements with other real engineers. Things now just coming to light, we've been playing with for nearly a decade. 🤗 Nothing is new under the sun. A known company is currently licensing the architecture from us for their new mobile devices to be released later this fall. #NDA for collaboration. 🙏
Wow did you do any special modifications? My device is also has 8GB ram and an Intel i3 processor do you think I can possibly run it with decent speed?
@@SkillLeapAI awesome, thank you. I have a specific use case requiring very specific data to train the model, would love to see how effective it can be. Also concerned with the data flowing back online, can you elaborate on what is sending date back out when the machine is reconnected to the internet as others have mentioned?
Is there a way to install Ollama on a Windows drive not labeled C? I want to install on a D drive, I transferred downloaded exe file to D drive, tried to install from there, no luck, no selection option with install tool?
Everything seemed fine until I clicked the link in Docker. The website page opened with an error message stating, "This page isn’t working." Can anyone offer assistance?
Thanks for the vid, a couple of confusions to share: 12:18 how can a LLama 3.1 model not know anything about LLama 3 because of delay? it doesn't make a lot of sense Plus you compared 8B on a text exchange while you gave the 70B model a python code to decipher, then you gave the 8b a text file to summarise. We can't compare execution times unless the task is identical
Hello! What kind of hardware setup do you have to have to run OpenSource LLMs? LLMs are actually quite heavy and they require high GPU, RAM, and CPU power. Can you do a TH-cam video about which computer power parameters or PC brands are optimal for running them smoothly? I love your content. Keep up the good work! :)
They are indeed quite heavy but in case you haven't noticed, in each of the LLMs documentation there should be a section dedicated to the hardware requirements, google has it, meta has it and the other probably do too. You could look them up like "Llama 3.1 hardware requirements" and the first link should take you there
Great video. Question: can I redirect all models downloaded (installed) in Ollama, to a secondary drive inside my laptop? C: Primary System M.2 SSD 2T D: Secondary SSD 2T
Nice video. I dont like u have to sign up for webui and don't like docker for security reasons but your fist two steps helped alot.. btw you can upload or pass large files on the command line. finally microisoft llama does not meet the traditional definition of open source, because it's not. What they did is create a new definition, their definition, and put it in the terms of service..... must be nice to be able to change the def of words willy nilly. also nothing we have today meets the official comp sci definition of AI, not by a long shot.
The Docker "thing" is only for embedding the model installed before to the webUI? Or its something that pushes all of the stuff to cloud and I don't need that NASA computer? Thanks
Nice video. What are the use cases for running AT locally on your computer at home? Whats the purpose when people can get a subscription? I can think of a few but i would like to know what others think. Cheers!
hey i keep getting a notification that llama 3.5 is ready to update from llama3.1? i click the notification in windows 11 but nothing happens. how can i verify or update llama 3.1?
How much vram is needed to run the model depends if the model of quantised or not. This should be explained in this video instead of giving contradictory information.
Hi. First of all, thanks for the video. It's really useful and easy to follow step by step. I am running M1 mac air and seems like it's not good enough to run llama 3.1 as it seems to freeze my computer. Which model would you recommend for M1 mac air?
Hi, I couldn't run docker on my GPU, I have RTX 4090 which should help a lot, while on CPU is slow, do you have any idea how to enable my GPU on Docker - Windows 11?
hugging face llama 3.1 model repo storage is 60 gb but it run very slow for one response and ollama run same llama 3.1model faster but size of llama 3.1 is around 5 gb. what is the difference
Hi, is this model multimodal like ChatGPT 4 Omni? I mean, can it generate code, solve mathematical problems, etc., or is it purely a linguistic model capable of easy conversation but unable to handle complex issues? In other words, my question is, do only the models hosted on the servers of OpenAI, Anthropic, or Meta have the capability to manage complex problems, or does this offline model also have that capability? Thank you.
They have some of that capability but online models are going to much better. It’s very difficult to run the better models offline. The best version of llama 3.1 is too complex to run on a computer and OpenAI and Anthropic dont have an open source model that you can run locally
this is great, thanks for puting in such simple and understandable way. I can run locally now, is there a way where I can point it out this local models to my python code, or my langchain/llama_index application code?
If I installed the 8B model first and then I want to install the 70b, Will I have both installed or the largest one will overwrite the 8B? Can I uninstall the models, just to save up some storage space 😅 (asking because I'm not too technical about it, I'm using Windows btw). Thanks in advance.
I downloaded the Windows file "OllamaSetup.exe" and installed Ollama. What now? After clicking on the app-icon nothing happenes. Is there any tutorial for Windows out there? Running the app in CMD, I cannot use the shortcuts: C:\Users\user\AppData\Local\Programs\Ollama>show The command "show" is either misspelled or could not be found.
Hey! Open cmd once again and type "ollama run llama3.1" you can play with it in command window. I'm looking for interface now, to run and play it in a nicer way - docker mentioned by the OP need subscription - could be 0 USD, but I don't like to subscribe.
@@fotszyrzk79 Thank you! This worked. I can chat in the command window. But I wanted to have an UI and I don't understand what the icon in the taskbar window is for. View logs and Quit Ollama is all I can do.
@@HaraldBendschneiderOpenWebUI gives you the windows chat interface. I use it on Windows 11 and it works fine with Ollama and Docker. You can look at Matthew Berman’s channel for simple install instructions as well. Bottom line: most tutorials assume Mac but it’s not hard to get it all working on Windows - you have to hunt a bit more for work arounds
I believe modern Macs run a unified memory model which combines and distributes the GPU and CPU memory as needed. PCs don’t do this so need a specific amount of GPU ram on dedicated Nvidia GPUs to run models. I have an RTX8000 which is 24GB VRAM which runs all 8B models fine - but completely chokes on 70B models regardless of quantisation. For Macs it’s all about your total memory available and having enough modern GPU cores to do the CUDA processing as I understand it
Check out our updated course on running private AI chatbots on your computer.
bit.ly/skill-leap
try AIJoel - Multi Generator: text, code, image (create sticker, remove image background, add color to black white image, image to video, logo design) and (music and video are in beta mode)
so I installed ollama and asked it, are you cloud based or are you running locacly on my computer? It replied that is cloud based Ai. Why is that?
Can upload image and analyze that imge in jpeg format?
One new SSD and a full install later.... It works!! Docker was giving me an error which I couldn't resolve but it was all straightforward on a clean installation!
Any luck?
Hey bro I just want to say that you grind is on another level. So good that you can go so for many years and sitll create new channels and succeed. Nice transition to AI content and your views are great too. Hope you get a lot of sponsorship and affiliate deals.
I appreciate that!
PC USERS IF YOU ARE GETTING THE WSL ERROR WHEN RUNNING DOCKER - Enable virtualization in BIOS: This process varies depending on your computer manufacturer and model. Generally, you'll need to restart your computer and enter the BIOS settings (often by pressing F2, F10, or Del during startup), then look for an option related to virtualization or VT-x and enable it.
I get this error but can't see anything in BIOS like you mention. My M/B is the Gigabyte X570 Aorus Elite.
@@joeduffy52 Enter the bios in advanced mode - go over to Tweaker tab - go down to advance CPU settings - and enable SVM mode
Just returned to leave this comment
"We need to go to an app that a lot of people have never used before.... its called 'Terminal'". OMG that line had me dying!
I’ve made videos for 8 years and 99% of people never used terminal
@@SkillLeapAI No I understand, and it makes perfect sense. Just as someone who's used a terminal at school and at work for the past decade... the delivery of line just tickled my funny bone
Absolutely hilarious...
Same dude : ) And then my wife came into the room and said "are you really?" HAHAHA
the really interesting part around 07:15 is that in a few months computing power and size of these models will make it possible to run them on practically anything. when they get more effcient we'll have them in our phones.
That's the future of this technology. Not that I endorse it or judge it in any way, but it's the way the development is heading.
OpenELM from Apple is already runable on mobile devices I think
@@thinhngo7244 i ran LLAMA 3.1 8b on my laptop with docker and ollama. like a breeze.
there are LLMs that work on phones already.
Brilliant.
I have been looking for this videos since meta announced 3.1.
Hats off.
@SkillLeapAI I should be really thankful if u can make a video to deploy it on server.
Perfect ‼️ Just the video I was looking for with intro to how to upload files to the model‼️ THX‼️
You have great teaching skills. Thank you for making it so simple!
Thanks! This video was awesome and really detailed! Can’t wait to start trying this.
Great - I will try 70B - ) thanks for instruction how to do it:)
What's your PC specs that can handle 70b?
@@burntdoug6823
Cpu: intel i5 2400
Ram: 4gb
Gpu: integrated Graphics
Storage: 120Gb Hhd
great video for local ai. easy and clear explanation. thank you!
No joke. I've always hated Docker. Hated everything about it. However, seeing you run the Openweb UI command and it just randomly showed up in the Docker app is making me reconsider. Holy moly! Might have to reinstall it on my Raspberry Pi 5.
Not private, its a hybrid model, caches data when offline then when connected back you can see it sends data. Use wireshark it will show you its sending data
This is an important comment because many channels are saying OpenWebUI and Llama3 is private to your machine. Is there any way to turn off the cache sending process? I would like to analyse documents that are not permitted to be publicly uploaded
@@longboardfella5306 Ollama and Llama3 are private, but OpenWebUI might send some analytics and other data.
Is this true? Any resource that supports this statement?
@@longboardfella5306 run it in a VM or container that has the network adapter disconnected from the internet, or wifi off entirely
Llama is literally private, maybe openwebui…. However what would the ui need to cache and send over the internet? Doesnt make sense to me
subscription incoming!! great content, keep up the good work
Awesome! Thanks for the post!!!
Thank you for your will to share with us!
Excellent - Thanks for sharing.
Sounds like step one is buy a $5,000 M3 if you want to run it locally now at least before they release smaller quantizations of Llama 70B and 8B
Llama3.1 8B runs much faster than you can read on an RTX 2060 Super. Not dirt cheap, but I built my PC for $1.4K 4 years ago - should be cheaper now, and I built it for gaming, not AI.
You can host it on a desktop and connect your other devices with a VPN like Tailscale. Don’t spend extra money for less performance by going for small form-factor/laptop.
I run 8b on my 2020 M1 MacBook Air (8gb ram and 7 core GPU) and the output token rate is much faster than a normal speech cadence.
I have a 9 year old windows computer that runs 13b and 8b models the same speed as your brand new $5000 Mac 👀 it was purchased at a Walmart for $270 in 2015 😂
Bullshit detected @@mcombatti
@bassamel-ashkar4005 As a machine learning engineer and master developer - over 25 years now, I must say you need to learn a bit about inference to speak. 8b can even run on edge devices like orange pi or Raspberry Pi. When models are converted to rknn for such devices, you can achieve greater speeds than base inference. Just replacing the matmul function in transformers with a more optimized function (that uses less 'compute') a model can generate 3x more tokens/ second. Pruning downstream layers and using sporadic attention, a KernelAttentionNetwork or even a KAN (not the same) in place of transformers can gain greater speeds. Until you can design an LLM architecture from scratch...shhh...
Not to mention our novel model architecture using stock CPUs can generate >3000 tokens per second - on base mobile phones with only 4GB RAM and an 8 core processor (cost of such phone $100-400). We have demonstration after demonstration available online and social platforms like LinkedIn- where we discuss the architecture advancements with other real engineers. Things now just coming to light, we've been playing with for nearly a decade. 🤗 Nothing is new under the sun.
A known company is currently licensing the architecture from us for their new mobile devices to be released later this fall. #NDA for collaboration. 🙏
I hope they give access to internet. Thank you for this
Nice, straight forward video.
Can you make a Hardware buying guide please?
I can see microphone icon on the right side typing window, can it be used to summarize meetings ?
Thank you very much for this video.
Huge respect
Hi, does anyone knows what is needed to run well full 70B Llama 3.1 model? so it responds quickly?
thanks dude, I was able to run llama3.1 8b on a linux laptop with 8gb ram and am impressed..
Wow did you do any special modifications? My device is also has 8GB ram and an Intel i3 processor do you think I can possibly run it with decent speed?
@@shampaghosh1241 No special mods, just selected the smallest model and followed the steps from this video and ran some prompts.
i would love for you to create a tutorial how to train these models on specific data. any chance?
Adding to the list
@@SkillLeapAI awesome, thank you. I have a specific use case requiring very specific data to train the model, would love to see how effective it can be. Also concerned with the data flowing back online, can you elaborate on what is sending date back out when the machine is reconnected to the internet as others have mentioned?
Thank you for tutorial.
Can we store the files on cloud storages like AWS S3 and then run it or deploy it ??
worked with issues hats off
you are the perfect man thank you
The steps doesn't show how the models gets loaded onto WebUI. When I'm in WebUI, there are no models.
They should automatically. I didn’t do anything in webui to add them. They were added through terminal in easier steps
On the top corner there is choose model
讲的很清楚，很棒的入门指引
Is there a way to install Ollama on a Windows drive not labeled C? I want to install on a D drive, I transferred downloaded exe file to D drive, tried to install from there, no luck, no selection option with install tool?
Everything seemed fine until I clicked the link in Docker. The website page opened with an error message stating, "This page isn’t working." Can anyone offer assistance?
Thanks for the vid, a couple of confusions to share:
12:18 how can a LLama 3.1 model not know anything about LLama 3 because of delay? it doesn't make a lot of sense
Plus you compared 8B on a text exchange while you gave the 70B model a python code to decipher, then you gave the 8b a text file to summarise. We can't compare execution times unless the task is identical
Why do I get 400: 'NoneType' object has no attribute 'encode' error anytime i try to upload a document?
Is there a way to bring this local running Llama 3.1 onto my website? I want to share my trained AI with other people. Thanks!
There are special server on digital ocean .. but simply you installon your website and make it available throught an API.
when i upload an image to it all it says is "i cannot see the image"
when I go to workspace I don't seem to have the "Documents" tab.
When I try to run open-web ui, Flowise opens up. I guess they are listening to the same ports =/ ? Can I solve this in any way?
Thanks for the great tutorial. Can you let me know the defaut path to store chat histories? Is it possible to change it?
i use ubuntu my web ui doesnt show any models after correct installation
+1
My docker won't show the address/ports after installing everything. What can be the problem? I'm trying to google it all day and can't find an answer
Hi. TY for this video. I'm wondering, do I need to train the model that I install?
No you can just use it after install
Can we do a APi call of this local LLM in my code ?
Than you so much. Can you tell me how to acces openAi with WiFi on or off?
I do appreciate this informative walkthrough though! TQVM!!
you have a great internet :D
wanted to try 70b with 4070ti 12gb ryzen 5 7600x 64gb 6000mhz ram and it is pretty slow, it takes 20 seconds to response "hi" prompt 😄
7600x is a weak cpu for AI.
It's a mere 6 core cpu lol.
There is probably ways to get it better maybe.
Wait, what does docker do if the LLM is running?
Hello! What kind of hardware setup do you have to have to run OpenSource LLMs? LLMs are actually quite heavy and they require high GPU, RAM, and CPU power. Can you do a TH-cam video about which computer power parameters or PC brands are optimal for running them smoothly? I love your content. Keep up the good work! :)
They are indeed quite heavy but in case you haven't noticed, in each of the LLMs documentation there should be a section dedicated to the hardware requirements, google has it, meta has it and the other probably do too. You could look them up like "Llama 3.1 hardware requirements" and the first link should take you there
Great video. How can I run the docker for open webui on my Linux local server, but ollama running on my windows pc?
Great video.
Question: can I redirect all models downloaded (installed) in Ollama, to a secondary drive inside my laptop?
C: Primary System M.2 SSD 2T
D: Secondary SSD 2T
Great video. Did you test how many documents you can upload at once and have it summarized?
I haven’t yet but it think it’s a good amount
Inference looks fast in local what configuration of Mac laptop are using please let me know. Thank you for nice tutorial.
It’s the top of the line m3 Mac with 64 gig of ram
9:20 i believe that there is also a docker credentials package that also works, and doesn't require the GUI bloat.
Nice video. I dont like u have to sign up for webui and don't like docker for security reasons but your fist two steps helped alot.. btw you can upload or pass large files on the command line. finally microisoft llama does not meet the traditional definition of open source, because it's not. What they did is create a new definition, their definition, and put it in the terms of service..... must be nice to be able to change the def of words willy nilly. also nothing we have today meets the official comp sci definition of AI, not by a long shot.
The Docker "thing" is only for embedding the model installed before to the webUI? Or its something that pushes all of the stuff to cloud and I don't need that NASA computer? Thanks
Nice video. What are the use cases for running AT locally on your computer at home? Whats the purpose when people can get a subscription? I can think of a few but i would like to know what others think. Cheers!
hey i keep getting a notification that llama 3.5 is ready to update from llama3.1? i click the notification in windows 11 but nothing happens. how can i verify or update llama 3.1?
Can we locally remove restraints on LLaMA models, including ethical safeguards?
Thanks
How much vram is needed to run the model depends if the model of quantised or not. This should be explained in this video instead of giving contradictory information.
as much as you are correct i am fairly certain "giving contradicting information" wasn't the intent.
Hey do you know where we can get a comprehensive explanation to what we downloaded? i cant afford his course
Are there way to select hard drive when install?
My C Drive is almost full, how can I select my other drive?
same issue. Please reply here if you find a solution.
th-cam.com/video/uj1VnDPR9xo/w-d-xo.html
@@R_RYT th-cam.com/video/uj1VnDPR9xo/w-d-xo.html
How much total of the storage it requires anyway?
Can I fine tune the 70B model and use the open web ui?
Hi. First of all, thanks for the video. It's really useful and easy to follow step by step. I am running M1 mac air and seems like it's not good enough to run llama 3.1 as it seems to freeze my computer. Which model would you recommend for M1 mac air?
Thanks
could you put the file on your web server then use it to be your search and or help?
can you do a tutorial on how to do it on windows, thanks
Are you able to upload images to use with prompts?
Most open models don’t have vision and if they do, they are not good. I would use ChatGPT for that
Lava models are pretty good for image analysis
Perfect. Thanx
is there a how to for window 11?
What's your hardware setup you run this model on?
I’m on m3 Mac 64 gig ram. I can run the 70B model and the small models respond almost instantly
@@SkillLeapAI mac book pro? mac pro? how much VRAM? (complete configuration). this has to be a $5K+ setup i suspect.
select a model does't work for me :(
Can I train my data?
how to upload file *.xlsx. When i use this apple can't upload file. Help me
i have macbook m3 pro with 18 gb ram and 500 gb harddisk will this be sufficient
How is it offline, when you have to login to webUI?
You just have to create the account. You can turn off your WiFi after that
Hi, I couldn't run docker on my GPU, I have RTX 4090 which should help a lot, while on CPU is slow, do you have any idea how to enable my GPU on Docker - Windows 11?
Use --gpu all Option :)
Thank you for your effort. Any reason why my LLAMA3.1 would respond one letter @ a time? 70B
@HitsInSandbox 4090 with 128 gigs of ram....i dont use an antivirus
By the way, how can the internet be connected to get real time answers with this?
It can’t do real time data from the web
Thank you.
hugging face llama 3.1 model repo storage is 60 gb but it run very slow for one response and ollama run same llama 3.1model faster but size of llama 3.1 is around 5 gb. what is the difference
can i add image ?
Will the website version be capabe of uploading documents anytime soon? My computer isn't large enough to support running the offline models.
Not sure. I hope so
Can you do a video like this for using llama in Python
I installed llama3.1. Now how to remove previously installed llama3?
Wow the 405B model file size is still smaller than some of the Call of Duty games. LOL
Yea that’s true but it’s basically text. If it was video, it would a million times bigger
Hi, is this model multimodal like ChatGPT 4 Omni? I mean, can it generate code, solve mathematical problems, etc., or is it purely a linguistic model capable of easy conversation but unable to handle complex issues? In other words, my question is, do only the models hosted on the servers of OpenAI, Anthropic, or Meta have the capability to manage complex problems, or does this offline model also have that capability? Thank you.
They have some of that capability but online models are going to much better. It’s very difficult to run the better models offline. The best version of llama 3.1 is too complex to run on a computer and OpenAI and Anthropic dont have an open source model that you can run locally
Crazy cool.
this is great, thanks for puting in such simple and understandable way. I can run locally now, is there a way where I can point it out this local models to my python code, or my langchain/llama_index application code?
If I installed the 8B model first and then I want to install the 70b, Will I have both installed or the largest one will overwrite the 8B? Can I uninstall the models, just to save up some storage space 😅 (asking because I'm not too technical about it, I'm using Windows btw). Thanks in advance.
It will install both and you can choose between them. It won’t override. And yes you can remove. My new video covers that and some new upgrades
@@SkillLeapAI Thank you, I will watch it. I managed to install them with this video!
What would be an affordable, yet ideal, desktop workstation
Great job! I have a M1 mac with 16gb ram. Can i run 8b?
I think so
@@uncannyrobotdo you also train models and would care to elaborate? i'd be all ears
@@uncannyrobot i will find one, thank you!
I downloaded the Windows file "OllamaSetup.exe" and installed Ollama. What now? After clicking on the app-icon nothing happenes. Is there any tutorial for Windows out there? Running the app in CMD, I cannot use the shortcuts: C:\Users\user\AppData\Local\Programs\Ollama>show
The command "show" is either misspelled or
could not be found.
Hey! Open cmd once again and type "ollama run llama3.1" you can play with it in command window. I'm looking for interface now, to run and play it in a nicer way - docker mentioned by the OP need subscription - could be 0 USD, but I don't like to subscribe.
If you type just ollama it will print the available commands for you.
@@fotszyrzk79 Thank you! This worked. I can chat in the command window. But I wanted to have an UI and I don't understand what the icon in the taskbar window is for. View logs and Quit Ollama is all I can do.
@@HaraldBendschneiderOpenWebUI gives you the windows chat interface. I use it on Windows 11 and it works fine with Ollama and Docker. You can look at Matthew Berman’s channel for simple install instructions as well. Bottom line: most tutorials assume Mac but it’s not hard to get it all working on Windows - you have to hunt a bit more for work arounds
Would like to learn to do Llama + Groq + iPhone shortcuts to run llama.
still confused on github ... it didnt take me there so i dont have a clue as to where your at there
So, you don't need a fancy GPU to run Llama locally? It does say that you need an Nvidia GPU...but you're running a Mac? Please elaborate.
I have the built in Apple GPU. These are my specs. Chipset Model: Apple M3 Max
Type: GPU
Bus: Built-In
Total Number of Cores: 40
The 8B model should run on variety of GPUs
@@SkillLeapAI Ah, I see. Thanks for explaining. :)
I believe modern Macs run a unified memory model which combines and distributes the GPU and CPU memory as needed. PCs don’t do this so need a specific amount of GPU ram on dedicated Nvidia GPUs to run models. I have an RTX8000 which is 24GB VRAM which runs all 8B models fine - but completely chokes on 70B models regardless of quantisation. For Macs it’s all about your total memory available and having enough modern GPU cores to do the CUDA processing as I understand it
How to add knowledge in this like gpt search whole internet and adding intelligence