Demo: Rapid prototyping with Gemma and Llama.cpp
ฝัง
- เผยแพร่เมื่อ 31 มี.ค. 2024
- Learn how to run Gemma locally on your laptop using Llama.cpp and quantized models.
Checkout more videos of Gemma Developer Day 2024 → goo.gle/440EAIV
Subscribe to Google for Developers → goo.gle/developers
#Gemma #GemmaDeveloperDay
Event: Gemma Developer Day 2024 - วิทยาศาสตร์และเทคโนโลยี
I've used gemma for a benchmark in a research project I'm working on, where I compared human results against AI, gemma was the closest after bloom 176B, followed by models like mistral instruct 7Band llama 34B, even the 2b version did pretty well, great work team 👏🏻
Hi mate, if you'd like to compare diff models for the same question, you do so on Wordware. Highly recommending it! ;)
@@polish4932 thank you
Very cool, thank you!
I like this format with demos. We are developers!
Thanks for the demo!
This is a fantastic format for a talk, insightful and engaging for a viewer!!!
Glad you enjoyed this video! 😎
0:21 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉
Really cool and simple to understand
Good demo, nice tooling suggestions out of this
Wow this is so cool 😎, developers changing the world
Cool demo, I'll try it
Minimum System Requirements for running a model locally?
I was looking for this! Thanks! 😎🤖
Thank you for the discussion.
Like it very good sharing.
Very nice 👌 cool Demo
how do you connect the LM Studio to llama.cpp? you used a preset which was probably custom.
This is so cool :)
CODE would be awesome, as well as knowing the SPEC of the notebook as it’s easy to buy a sub1k or 5K+ at apple store depending on so many factors…
Very nice!!!!
Wrapped in the llamafile runtime it is an even better single file .. oh yes!
This is so cool.
cool demo
How about windows environment? Is LMstudio/Ollama a better choice?
Very interesting demo!
What kind of extra work would be required to run this without LM Studio? Does Llama.cpp contain the necessarry functions to load models as servers you can interrogate?
Cool. Will definitely use this soon
At 9:03, Gemma shows some solid satire
Is this meant to be super accessible? If I have an APU, on a laptop with no GPU or NPU(?), that means I can expect it to run fairly well? Also considerations for a lighter yet usable model?
I tried that in an old APU, very slow generations but the same result.
Nah, Gemma is just a parot. It is released for fine-tuning, aka research purpose.
Very cool demo 👍. Any chance we can get the source code 😀
Anyone notice ram usage of 16.68 GB? What was that?
Very nice, but what is your computer spec? Memory and chip?
Probably m3 max 128gb
Gemma 7b can run on a rtx 4060 laptop
good, thank you
where can we learn to set this up ?
Great choice of demo app!!
Has anyone else tried doing this? I tried following this exactly with LM Studio using the exact model and prompt but I am consistently getting atrocious outputs; the gemma model is just outputting gibberish or incorrectly formatted JSON.
I wish there were more details on the presets used.
If anyone faces an error on Mac about "network error: self signed certificate", close the app and use terminal, run "NODE_TLS_REJECT_UNAUTHORIZED=0 open -a "LM Studio" "
This reopens the app and the error should go away. I have no idea where to put this info sooooo...
That's a lot of RAM to run this for me.
Just my two cents: 1. No explanation on how to connect LM Studio to the Llama.cpp, 2. newest hardware required - at least it doesn't work on my M1 with eight performance cores and 32 GB Ram
damnn its using 15 gb of ram i have an 8gb m1 i dont think it will work for me..
You can download a quantized version 8 maybe 4-5 bits
@@lorenzo9196 okay i will try
Pls provide source code also
What’s the required hardware specs?
did gemma should use GPU ? so I wanna try to learn this model, but I didn't want to use GPU
Awesome.
game using lm studio, interesting.
It's cool now, but what if each application starts deploying local models. It'll turn our phones into what data centers were meant for thereby reducing costs for large corporations. Trading a few megabytes for faster and more expensive chips.
Please Elaborate. What were data centres meant for? Asides hardware to run inference of worldwide requests. If it’s done locally, surely it’s better for redundant tasks. Also, the data centres use a lot of megabytes and expensive chips.
wow
Just focus on your existing products Google. Which are a mess 😂
never trust google products.
Google seems to be trying out too many new things. to me thats a turn off and red flag. everyone is different 🙂