How to Run LLMs Locally on Phone with ChatterUI App - AI Model on Phone

[@vivekkarumudi]

this app is lovely especially the text to speech TTS feature on an existing power of LLM.🤩

[@fahdmirza]

indeed

[@ManananggalNaStroke-x5w]

Can someone please provide the right template for GPT-2? I can't get it to work because I remember using ChatGPT-1 and 2 before, and it's definitely not that good. But it's okay. It is a satisfaction. Right now, I really just want to make the GPT-2 model work in this app because it keeps saying random stuff due to the wrong template. And there is no tutorial on how to create the right template for the model. 😭

[@pneuma23093]

Thank you, Mr. Mirza. That’s exactly what I needed to know. I’m moving to Android for more RAM and faster charging, and the Snapdragon 8 Elite and Dimensity 9400 are looking really promising this year.

[@fahdmirza]

enjoy

[@eado9440]

Early testing, but seems more than stable than others .and running the lattest granite ibm 2b and smoll 1.7 b, models, haven't measured tokens vut i guess its around 11 tokens per second, definitely usable, . Phone specs 8.1 chip 16 gb RAM. And the Note: For devices with Snapdragon 8 Gen 1 and above or Exynos 2200+, it is recommended to use the Q4_0_4_8 quantization for enhanced prompt processing.

[@fizzyfizzgigcouple957]

Thank you this is exactly what I was looking for for the last year. I just set it up for my Motorola edge 2022 plus version. I am able to run the nous Hermes 3 8b model on my phone although it is very slow similar to the video. I am still very happy that I can at least run it locally on my phone I am going to try the 1 billion now and see if that is smooth.

[@fahdmirza]

Great, please keep us posted about the performance. Keen to know how the models are performing on different phones. I have also done videos yesterday on some other apps, please also test it on them if you like, thanks.

[@fizzyfizzgigcouple957]

@fahdmirza I tried pocket pal it kept crashing on me. I only got one of the models to load up the 1 billion llama

[@HelmutBemboka]

I load Llama 3.2 3B. When I say "Hi" it says this
" What can I help you with?
import random
def check_greeting(user_text):
if user_text == 'Hi ':
return check_user_intention('greeting', 'greetings')
else:
return check_user_intention('greeting', 'greetings')
def check_user_intention(name, responses):
response_msg = random.choice(responses)
print(response_msg)
return response_msg
"
Is that normal?

[@ManananggalNaStroke-x5w]

Definitely not normal. Yesterday, I was also confused like you, but today I understand why the model keeps going insane. It's all because of the incorrect template. 🤦 Since your model is Llama 3.2, you should select Llama 3.2 as the template in the instruct window.

[@mastermanhwas]

Thanks on my Samsung galaxy A35 5g it takes 1 second to respond i have tried in the airplane mode this was really grateful thanks for your work, Same like any model that can run on like this for the text to image generation? Can anyone please tell me that.

[@Boss-ru9ul]

Super

[@fahdmirza]

Thanks