Hi! Thank you for watching and for the kind words! Regarding your question, unfortunately, the LLama 3.2 model with 11 billion parameters won’t be able to run fluently on an RTX 3050 with just 4GB of VRAM. Even with aggressive quantization (like INT8 or INT4), an 11B parameter model requires significantly more VRAM than 4GB. You would need at least 16GB or higher to run it smoothly offline.
I am running 3.18b Q5 KM on an RTX 3070 with ollama in Docker (with Nvidia GPU Runtime ) and it runs just fine. Very responsive. -- Ubuntu Linux 22.04 LTS with 64GB ram.
Llama vision is not very good for transcribing text - it makes a lot of things up. I show that in the latest video on my channel. Claude is currently miles ahead of anything else. GPT-4o not far behind. Llama a bit of a joke.
Hi.
Great video.
I whish to know if llama 3.2 the 11b param model can run fluently offline on my RTX 3050, 4gb VRAM laptop.
Thanks
Hi! Thank you for watching and for the kind words! Regarding your question, unfortunately, the LLama 3.2 model with 11 billion parameters won’t be able to run fluently on an RTX 3050 with just 4GB of VRAM. Even with aggressive quantization (like INT8 or INT4), an 11B parameter model requires significantly more VRAM than 4GB. You would need at least 16GB or higher to run it smoothly offline.
@@AIFusion-official thanks for the info.
@@JaneCacti yeah, I am currently running the 3.1 8b params model. Though it is somehow slow 😅
I am running 3.18b Q5 KM on an RTX 3070 with ollama in Docker (with Nvidia GPU Runtime ) and it runs just fine. Very responsive. -- Ubuntu Linux 22.04 LTS with 64GB ram.
@@AIFusion-official hello im download vison instruct model 11b but how can import openweb ui?
It is actually a terrible VL model. English only.
Of course you could also use the LLM to do a language translate, either in or out...
Llama vision is not very good for transcribing text - it makes a lot of things up. I show that in the latest video on my channel. Claude is currently miles ahead of anything else. GPT-4o not far behind. Llama a bit of a joke.