How to run Mistral LLM locally on iPhone or iPad
ฝัง
- เผยแพร่เมื่อ 12 ธ.ค. 2023
- Based off the following tutorial: / using-llms-locally-ipa...
LLM Farm: llmfarm.site/
Mistral: huggingface.co/TheBloke/Mistr...
Description:
📚🔍🌐 Join us as we explore the cutting-edge world of AI with a step-by-step tutorial on installing a ChatGPT-like large language model (LLM) locally on your Apple device. Based on the insights of Maciek Jędrzejczyk, Senior Cloud Infrastructure Architect at Amazon Web Services (AWS), this guide is tailored for anyone looking to run an LLM on their iPad Pro Gen or iPhone.
🚀 In This Detailed Walkthrough, You'll Learn:
Preparation: The prerequisites for installing an LLM on your Apple device, including required RAM and storage space.
Installing TestFlight and LLMFarm: A step-by-step guide to using TestFlight to install LLMFarm, an open-source client supporting Apple Silicon, on your device.
Downloading and Setting Up the Mistral-7B Model: How to download the Mistral-7B model from huggingface.co, and set it up using LLMFarm.
Configuring Chat Settings: Detailed instructions on setting up your chat context window, including prompt formatting and resource management settings.
Testing and Using Your LLM: Finally, testing your setup to interact with the model and check the accuracy of results.
✨ Why This Tutorial Is Important: This video is perfect for privacy-conscious individuals, AI enthusiasts, and tech-savvy users who want to leverage the power of LLMs on their portable Apple devices.
🔔 Stay Updated with AI Innovations: Subscribe and hit the bell icon to keep up with the latest tutorials and insights into AI applications and data privacy.
💬 Your Experience and Questions: Have you tried setting up an LLM on your device? Share your experiences or any questions you have in the comments - let's discuss the exciting world of local AI models!
📲 Connect with Us for More Tech Insights: Check out the links in the description for more information and updates. Follow us on LinkedIn and other social platforms for regular content on AI, technology, and privacy.
#ChatGPT #LLM #Apple #iPhone #AI #mistral
![[Full Episode] MasterChef Junior Thailand มาสเตอร์เชฟ จูเนียร์ ประเทศไทย Season 3 Episode 4](http://i.ytimg.com/vi/e7Oqqo90x_w/mqdefault.jpg)
The fact that LLM can be run locally is already fascinating. Excited for 2024.
Yeah will be exciting to see what Gemini is able to do being designed for mobile
This is amazing! I got it working on an iPhone 14 with only 6GB RAM. It works great, but is very slow (3 tokens a second) and puts my phone on its knees! I wouldn’t use it day to day but it’s a wonderful profit of concept and performs amazingly well. Even Gemini makes similar mistakes. I didn’t turn on mloc as advised by someone else in another comment.
Amazing! I’m running like 5 models on my iPhone, I love it! Especially for small talk
Oh wow!! What other models have you tried?
@@kylebehrend I have llama chat 7b, orca 3b and dolphin mistral 7b. All q4
@@paulkiragu8120what’s your favorite?
Great video. Mistral just launched v2 model 🎉
Nice 👍
Hi Kyle! Thank you for your video, I was trying to run this on my iphone 13 but using microsoft/Phi-3-mini-4k-instruct-gguf and whenever I try to run it, my phone completely crashes. Any ideas on why this is and how can I solve it? Thanks again!
Hi, thanks for the video! Would you mind sharing what kind of screen recording/editing software you use? I’ve seen it used I a few other tutorials and am quite impressed with the style and quality.
Sure thing ScreenStudio :)
That's a great tip; thanks for your video. By the way, I wonder when Apple is going to release their own LLM, and since Apple controls their own silicon, I guess they will put some emphasis on LLMs that take advantage of Apple's 'Neural Engine.' I noticed that in features like iOS 17, it clones your voice. Personal Voice is trained locally on your device.
Apparently iPhone 16 will have AI capabilities and no doubt 2024 will be huge
Can it be run on a modern smartphone or just on ios?
I think iOS but new phones will have local LLMs too
Followed to the letter and I just get a bunch of gobble about derivatives and mathematical jumbo :(
Oh no :( You may need to force shut the app and try again. It's quite experimental at this stage :)
I’m IOS17, everytime i done this (i tried remove and do it again many times) and when i chat it’s look like my device not enough Ram, very lag and I can’t do anything even when my device turn off screen I can’t touch it, my device is hot, sometime it’s reboot,...
Mine does the same thing, I didn’t care to look to fix it, but I did notice when looking through downloads I had a setting that made me download the model to my iCloud and not my actual device storage this might possibly be the problem
What device you on? iPad is better than iPhone unless it’s high spec
For Minstral 7B settings template: Llama2 chat 7B new .. inference turn on Llama and turn of mlock all will work well. Also in system setting stop the app from running in the background and it will stop the lagging
for the life of me I cannot get this to work on my 9th gen ipad! I have followed the directions step by satep several times even using the original article which is a different model then the video. anyways each time i get model load error eval [error] load model, or load model error: [done] its so frustrating! please redo this with the new ipad LLM farm interface and a working model or teach us how to download a model that is available in the downloads and get it to work.
Probably better to wait a little, I'm sure there is a better way of doing this now. I'll take another look
possible to run on android?
I haven't seen it done yet but I am sure there is a way :)
mine crashes?
I probably wouldn't recommend doing this anymore. Was more an experiment :)
Don’t turn on MLock if your using metal this will cause your device to freeze
Thanks for the tip! Whats MLock?
@@kylebehrend metal lock
@@kylebehrend locks the loaded model into memory
@@kylebehrend settings template: Llama2 chat 7B new .. inference turn on Llama and turn of mlock all will work well. Also in system setting stop the app from running in the background and it will stop the lagging
@@pjth3g0dx Ooh interesting thanks will have to check that out.
Done that, LLM Farm spews complete nonsense. First it generated some sort of novel-type text after simple "hi!", now it creates code after the same prompt.
I can't speak for LLM Farm specifically, since I never used it, but this looks like an issue with the model or the settings, not with the app.
Try decreasing the "temperature" setting; "0.75" is a good baseline.
I'd rather pay OpenAI cents to run the latest LLM model on the cloud,
than buying an iPad Pro to run, whatever this is, on an 8 GB shared RAM.
Me too :) This is just a demo of whats possible and if I had a trip planned with no WIFI this would be on my download list
Well, of course you wouldn’t *buy* an iPad Pro to run this, this would be something for people who *already have* an iPad Pro and who want to try it out.