Hugging Face SafeTensors LLMs in Ollama

In this video, we're going to learn how to use Hugging Face safetensors models with Ollama on our own machine.
We'll also learn how to quantize the model to reduce the memory required and increase the number of tokens generated per second.
#llms #ollama #safetensors
Code from video - github.com/mneedham/LearnData...
Ollama Quantization options - github.com/ollama/ollama/blob...
SafeTensors - www.google.com/search?q=safet...
Hermes Model - huggingface.co/NousResearch/H...
Model downloader - github.com/bodaay/HuggingFace...
• Running Mistral AI on ...
• Few Shot Prompting wit...
• Ollama adds OpenAI API...

[@saramirabi1485]

Hello, Thanks for the great videos. It's been about several ours I am browsing in your channel. Just a question is it possible to use Ollama and doing fine-tuning with that?

[@csepartha]

Nice

[@bocilmillenium7698]

how to tinstall modeldownloader? i try git clone, and then try hfdownloader in cmd, its still error its not recognized as an internal or external command. thx

[@learndatawithmark]

I used the instructions from the Quick Download section - github.com/bodaay/HuggingFaceModelDownloader?tab=readme-ov-file#quick-download-and-run-examples-linuxmacwindows-wsl2

[@janithaoshan9227]

hello, im stuck at the quantize part can you help? I'm using terminal on macos with ollama. pls send me the terminal commands to quantize safetensors llm with the create -q command on ollama(Q5_K_M). thank you

[@learndatawithmark]

All the code that I used is here - github.com/mneedham/LearnDataWithMark/tree/main/ollama-own-model
Let me know which bit didn't work and what error you get?

[@generolas121]

i keep getting incorrect function, any advice?

[@learndatawithmark]

Incorrect function? Can you explain a bit more

[@ZenitoGR]

I can't get find help about 'Error: llama runner process has terminated: signal: aborted' can you assist me?

[@peteralex5921]

I get this: Error: llama runner process has terminated: exit status 0xc0000409.. Could be nice if someone figured out whats wrong here :D

[@learndatawithmark]

Have you tried posting it on the Ollama GitHub repository? I haven't come across that error yet!

[@parthwagh3607]

Thank you so much. I am having problem running models downloaded from hugging face having safetensor file. I have these files in oobabooga/text-generation-webui. I have to use this for ollama. I followed everything, even created modelfile with path to safetensor directory, but it is not running >> ollama create model_name -f modelfile. Please help me.

[@learndatawithmark]

What happens when you run the command?

[@parthwagh3607]

@@learndatawithmark
Thank you so much for quick response. Your videos have helped me a lot.
I am running this on windows 11. I did follow steps:
1) created Modelfile with this script:
"FROM C:\Users\PARTH\Downloads\text-generation-webui-main \text-generation-webui-main\models\TheBloke_dolphin-2.7- mixtral-8x7b-AWQQ
TEMPLATE = """{{ if .System }}system
{{ .System }}{{ end }}
user
{{ .Prompt }}
assistant
"""
PARAMETER stop
PARAMETER stop
PARAMETER stop "
2) I ran following command on terminal opend from where this modelfile is stored.
"ollama create mixtral:dolphine -f .\Modelfile"
3)It showed me this error:
"Error: command must be one of "from", "license", "template", "system", "adapter", "parameter", or "message""
4) I only made file with FROM statement without parameter and template, It ran, but gave this error:
"C:\Users\PARTH\.ollama>ollama create mixtral:dolphine -f .\Modelfile
transferring model data
unpacking model metadata
processing tensors
Error: couldn't find a layer name for 'model.layers.0.block_sparse_moe.experts.0.w1.qweight'"
5) I ran again with another models, but gave same error:
"C:\Users\PARTH\.ollama>ollama create slimorca:13b -f .\Modelfile
transferring model data
unpacking model metadata
processing tensors
Error: couldn't find a layer name for 'model.layers.0.mlp.down_proj.qweight'"