Ollama can run LLMs in parallel!

In this video, we're going to learn how to run LLMs in parallel on our local machine using Ollama version 0.1.33.
#ollama #llms #llama3 #phi3
Code - github.com/mneedham/LearnData...
Ollama 0.1.33 - github.com/ollama/ollama/rele...
Blog post - www.markhneedham.com/blog/202...

[@123arskas]

Thanks for the code and the information

[@36mcast]

Fantastic video and thanks for sharing!
A few notes for I already have ollama installed and I have to turn it off before running the commands.
1. Turn off the ollama on ubuntu or mac ` systemctl stop ollama.service` before running `OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve`
2. Line 12 in your code will not work on some streamlit version as the API key is ` api_key="ignore-me"` but should be `api_key="ollama"` by default

[@learndatawithmark]

Hey, thanks for your kind words. Regarding:
1. Yes, you are right. I had manually killed it on my machine, but your way is better. And if you want those environment variables to persist you'd want to set them in your .bashrc file or similar.
2. Do you mean on some openai library versions? I didn't realise it was used unless you were calling OpenAI itself, but I will use 'ollama' from now on!

[@36mcast]

@@learndatawithmark for 2. This would be for any open source model for some reason, it did not work me. This might be related to the package/ollama versions. GenAi libraries are updated about 2x per week to point where what you see on the internet today does not work next week

[@thomasrodermond6057]

Very good work. Thank you!

[@learndatawithmark]

Thanks!

[@ajmalm1]

Can you explain how to use this parallel capability with the ollama python library.

[@tlubben972]

I agree , would be great if you can provide the code for doing this in python 😊

[@learndatawithmark]

Do you mean in Python in general or specifically how I did it to build the UI I used here?

[@123arskas]

Can you show the Parallelism of Ollama through LangChain too? Thank you

[@learndatawithmark]

What would be a good LangChain example - showing how to call Ollama multiple times via LangChain? Is that what you had in mind?

[@123arskas]

@@learndatawithmark
Summarization of multiple TH-cam Transcripts one by one. Lets say 100 of them.

[@karthikb.s.k.4486]

What IDE are you using please let me know.

[@learndatawithmark]

The code is in Vim

[@learndatawithmark]

But I'm only using Vim for the video as it lets me make the text really big. Usually I use VS Code

[@anilrajshinde7062]

Your all videos are great. Can you prepare few videos on LLM OS where we can use Ollama?

[@learndatawithmark]

I haven't heard of LLM OS before - can you explain it a bit more?

[@anilrajshinde7062]

@@learndatawithmark th-cam.com/video/6g2KLvwHZlU/w-d-xo.html