man thank you this is the most comprehensive easiest way to install this. I had ollama before but with open webui it's a game changer. Do i need to run the app docker before each time or it will auto boot when computer starts ?
You can change the services that start with the computer by typing MSCONFIG after hitting Windows+R. If you see Docker in the list it will boot with computer
What was your PR about? ❤ Watching this video makes me wanna go add some stuff too, Things I caught from Matthew: RAG loading indicator until files are avail. Get new models in UI rather than term. Things I wish to add: litellm, groq adding pipelines and flows not just prompts, adding #folder to context, easier installer that checks docker etc with Houston assistant. OpenInterpreter integration. OpenUI integration + Developer Mode, Tester Mode, Rapid Feedback, Obvsf API Pools, Import ChatGPT & Claude History, Stars, Topics (connected threads), Timelines, Search Hist, Actions, Prompt Refinements, Self Improvement, Skill Library, Image Gen & Editing. Song Gen, Incognito thread, global/group chats. Sign in from phone/multi device. Agents and Clients Dashs.
Wow I just built a virtual AI girlfriend using ollama. I'm trying it on Llama3 model, and recently migrated to one of the uncensored models. Good-bye wife, hello AI - LOL.
I've been with you for over a year and it's been amazing watching you dominate this LLM news space. For example your snake game has become a standard in the industry now!!! Like you, I'm constantly in the LLM lab and I'm constantly coming across your name with a quote regarding a large language model. Awesome job carving out a niche - MUCH LOVE FROM NEW ORLEANS 🔥💪
Few more cool features 1. Image generation. I hooked this up to both local automatic1111 and DallE3 with api key. It's a bit of an odd workflow. You prompt it and the response will have a little pic button under it. I loaded a model finetuned for image prompts so response is cool. 2. Hook up openai models for chat choices with your api key. 3. Pull any llava model and you can hit the plus button to load a picture and ask questions about it.
As someone who is completely new, this is so damn confusing LOL first off, what is docker. Second, where are you and is this a windows terminal? Im extremely lost and maybe link your old videos?
I'm not sure if you've gotten any answers to your questions here, but I thought I'd try to help a bit where I can. To your first question, docker is a program that allows you to run software in something called a container. You can think of containers as a space in your RAM that is separate from the rest of the programs that are running on your computer. Docker is nice because it makes it easy for you to run things that are dependent on certain versions of Python , shared libraries, and other stuff without you having to worry about changing versions back and forth to make other things work on your computer. Secondly, from what I could see, buddy is on a Mac computer (the UI he shown at the top where there was a llama kind of was the giveaway) which unless he changed it to bash or something, means he is using a zsh terminal, but that's not important. If you are using a windows computer look up how to get WSL (a.k.a Windows Subsystem for Linux) on your computer and you should be able to follow most of the instructions. Helpful tips: Linux has different distributions but they fall into 3 main families- Debian(some people say Ubuntu but I'm my own brand of snob here), RedHat, and Arch. This is important to you as far as getting started working with linux because it'll determine your package manager. (Debian uses apt, RedHat uses dnf, and Arch uses pacman) they all have their own quarks as far as what they name things and how they function. But your going to want to ensure you have git installed in you WSL environment as well as having the most up-to-date version of things on there for security reasons to start off. Learn about the basics of bash shell. Learn the basics of git. There will be some differences as far as folder structure once the program is installed, but I think you're smart of enough to figure that out 😉. Hope this helps. Now go on out there and cry like the rest of us. 🫡
I don’t usually post messages, but your video changed that. Very well done! I followed your steps, and within minutes, I had LLama3 running on Open-webUI with Docker on Ollama on a Windows computer. Thank you, Sir. Keep up the great work!
It would be great to see a comparison between ollama and lm studio explaining the benefits and reasoningg of when to use each. The one thing i havent seen much of is how to leverage (if possible) other models from huggingface within ollama. This is easy to do in LM studio. For most other thigs i prefer ollama but i tend to use LM studio to test new models that ollama might not have readily available.
Same here. I prefer LM Studio and use it across networks hosted from a server. Not quite the same with ollama. Going to experiment with open-webui and see if it can connect to LMStidio with some code tuning.
For one thing, Open WebU is open source, which means that you'll be able to use it forever for free, for commercial or personal use (that and all the other advantages of it being FOSS, like more security, a nice and supportive community, etc.). That being said, LMStudio is easier to install.
I am new to LLM development. How do I get the documents(PDF) loaded and RAG setup in Ollama UI to be part of my development in AUTOGEN 2.0? If you load the documents with Ollama UI and I use Ollama as a local LLM for AutoGen, will I see the embeddings in my prompt data through AutoGen? Thanks!
Llava model allows you to ask questions about images. Hit + next to prompt field to upload pic. Then ask away. Wasn't great generating code from mock, but answered general questions ok.
Matthew, could be awesome a podcast with David Ondrej. He is inviting you in his videos. I'd love you both to talk about this tools. I'm using Anything LLM with LM Studio. What is the difference with Ollama with Ollama UI? thank you for your videos.
HEADS UP your first acct is your admin acct, if you update owu or ollama, or dkr, make sure you follow the instructions or it will lose your user name and password and you wont have admin anymore, and its a pain to straighten it out, doable, I did it, but a pain.
Does anyone know how to make the open-webui interface accessible from my phone on the same network? I'm used to being able to add --listen to a gradio webui and access it anywhere on the network, but I haven't seen anyone do that with this one.
I used yarn in the plane to discuss with phi3 about astrophysics. It was epic. At some point, it becomes slow (llama3 started out slow), so I would read for a minute something else and return to its response and continue the chat. It was great. I so hate the offline time on planes, but having a gpt (or 10) on you computer really helps. Also very funny how my battery would drop on burst every time I press enter, lol. It takes juice generating those responses. But again, it's pretty cool. I think at some point I wanted to install ollama ui but the docker part lost me.
This comes so close Quick question I am looking for a single installation that allows running LLMs locally and has a front end that includes text-to-speech and speech-to-text, with a backend that allows me to attach from the local network from other machines to provide LLM services the kicker is I would like to to add all interactions into a rag from both front end and back end services to allow "learning" The idea is to have 1 pc providing all my AI needs locally for home automation, assistance, and various library/research/content generation services on my local network and continuing to learn about me and my needs as time passes (ultimately to be my interface with the internet in general) and help/lead would be greatly appreciated.
Thanks! Great video! Can you do bit different scenario : LLM engine (Ollama, llama.cpp or something else) running on dedicated hw with GPU acc. and frontend in local machine (vm/container/conda env). I'v been using ollama on hw with GPU's and frontends in VM or container since they started supporting OpenAI API .. but very mixed results. Idea is that opensource LLM backends are quite stable now .. few upgrades, new features and bugfixes. Just download new model when something interesting appears. But frontends and other implementations are still Wild West and better kept in easily recyclable format :)
I have tried the blulk load feature in the documents , it does not work , cannot change the embedding models as well , after the restart it keeps defaulting to the original. There is a ticket in the github for exactly this . However I have not heard back from the team.
Another superb, insanely useful how-to vid! A noob question: Setting this up with Docker vs Anaconda, the config with Python version and all the supporting libraries and package becomes the default for that user account on your computer, correct? (I’m on a Mac like you, and don’t think this is an issue at all for me, just wanted to understand the setup better.)
I dont think i understand, can this be used as a UI for a particular platform or device, or is it just a standalone thing? I mean, would this work as a UI instead of windows or Android, or is this a toy?
Question about these embedding models. I _thought_ that adding your own document makes and "embedding" loaded into the LLM for you to search your document. But what are those embedding models that you can define in the gui settings? I can't quite figure out why i'd want them or what they are used for..
I just learn this myself. Apparently embedding is a seperate thing or app or code that does that part so you actually need the database and an embedding model only to be able to save stuff in. vector you don't need the llm yet only once it's saved by the embedder then you can retrieve from the db with your lmm and rag, the main llm doesn't save the data only fetches it later
@@jarad4621 Thank you for your reply! I've read it about 10 times but still don't think i'm following you entirely. I get that you don't train your main LLM for your custom data. That's what the whole RAG thing is about. But with RAG you end up with a vector database that your main LLM can use to provide better results for specific subjects. Thus far i get it. But what is that "embeddings llm" like "nomic-embed-text"? I can't figure out how you use it, where you use it or what to do with it.. Just guessing here, or is there some kind of model hierarchy? Does it follow a path like llama -> nomic-embed-text -> own vector db? In which case the "nomic-embed-text" would be the one interacting with my data, not llama like it would be in a classic RAG setup? 99% guesses here, i just don't know.
*** I posted this for someone else here thought it may help you a little bit too. If anyone wants to correct or add anything please do.*** I'm not sure if you've gotten any answers to your questions here, but I thought I'd try to help a bit where I can. To your first question, docker is a program that allows you to run software in something called a container. You can think of containers as a space in your RAM that is separate from the rest of the programs that are running on your computer. Docker is nice because it makes it easy for you to run things that are dependent on certain versions of Python , shared libraries, and other stuff without you having to worry about changing versions back and forth to make other things work on your computer. Secondly, from what I could see, buddy is on a Mac computer (the UI he shown at the top where there was a llama kind of was the giveaway) which unless he changed it to bash or something, means he is using a zsh terminal, but that's not important. If you are using a windows computer look up how to get WSL (a.k.a Windows Subsystem for Linux) on your computer and you should be able to follow most of the instructions. Helpful tips: Linux has different distributions but they fall into 3 main families- Debian(some people say Ubuntu but I'm my own brand of snob here), RedHat, and Arch. This is important to you as far as getting started working with linux because it'll determine your package manager. (Debian uses apt, RedHat uses dnf, and Arch uses pacman) they all have their own quarks as far as what they name things and how they function. But your going to want to ensure you have git installed in you WSL environment as well as having the most up-to-date version of things on there for security reasons to start off. Learn about the basics of bash shell. Learn the basics of git. There will be some differences as far as folder structure once the program is installed, but I think you're smart of enough to figure that out 😉. Hope this helps.
Great video! I installed Open WebUI on top of Ollama yesterday. It's freaking awesome. Looks and feels just like ChatGPT. Keep up the good work on your videos. I think they are "best of breed" when it comes to AI videos on TH-cam. By the way, the response tokens per second in your video is very high. What is your hardware, especially your GPU(s)?
I have already downloaded many models that I use with Oobabooga and LM Studio (Thank you for suggesting LM Studio, it's a good software. Shame it does not support EXL2). But, I don't want to download models specific to Ollama.
Will try it This coming week! Have you or anyone tried the "Chat with RTX" by NVIDIA? I am trying to go through the setup giving it directories to intake BUT it failed on most expecting zip files... I recall you had covered "Private GPT" before, would it be able to do the same thing intended for Chat with RTX? I bought a new system with a 4060TI 16gig and 32 gig RAM, 1 TB SSD with 2 TB HDD, windows 11... It should run well but currently crawling while Parsing Nodes... any guidence woudl be greatly apreciated!
Great video thanks for sharing. Sam Altman talks about GPT 4 being an embarrassing model. Hopefully his team see this video and are equally embarrassed about their piss poor UI.
I really wish people would start being more original with their ui. If I wanted my front end to look just like ChatGPT then I would just use ChatGPT free lol. That’s why I like LMStudio. Only thing holding me back from using it more often is the lack of voice features.
Hey matt, can you tell us how you setup your terminal? (which terminal software you use, addons, etc). Was also hoping you can create a video testing which LLMs do the best at text summarization for large documents / transcripts
What we need is multi-prompt templates (series of prompts, one at a time), including step repeat. This way we can have the LLMs reflect on its previous answer before executing the next step in the series.
Thanks for creating this video with clear instructions!! After running this UI with llama3, the responses that I get is very slow even to those simple questions, e.g. how are you?. Is having GPU in my machine must to use this with proper speed?
Your video about this software comes way too late to be honest. The guy behind this project had made this thing awesome like year and a half ago. But it is nice that you did it anyway people need to know about this project and support it since rly rly good. Keep up the good work ;)
0:50 "Look how fast that is" well, the response being fast is not a function of the UI, it's a function of the hardware you're running it on. so please.
Another extremely informative video! I have an idea for a follow-up video: I would love to see a tutorial on how to secure the docker container with https / ssl. I am trying to figure this out through experimentation and have so far been unsuccessful. Thank you again for your great content!
what gpu server provider you use for your language model deployment? or do you only need gpu power for training? can you create chat agents from those ollama deployed chat models? im just getting into stuff and some of your videos are nice to follow
in case this helps someone. to run linux images on windows, you need to switch to linux containers, which will use wsl2. you do this by rightclicking on docker running in taskbar, and selecting switch to linux containers.
I was hoping someone else would handle all the heavy lifting with the UI, as it was somewhat of a turn-off for me to read replies in the shell. Thank you for sharing it!
This is definitely my local go-to now. What an amazing project.
Могу я там исп llmstudio?
Wow, you too?
man thank you this is the most comprehensive easiest way to install this. I had ollama before but with open webui it's a game changer. Do i need to run the app docker before each time or it will auto boot when computer starts ?
You can change the services that start with the computer by typing MSCONFIG after hitting Windows+R. If you see Docker in the list it will boot with computer
Is posible install this in a Rpi5 with Coral and use from multiple sources like Home Assistant or mi custom app?
As a contributor (merged one single PR 😊) but mostly a very early adopter of this project, I'm always stoked to see people talking about open-webui
great project! I just covered it on my channel too. congrats!
Nice
What was your PR about? ❤
Watching this video makes me wanna go add some stuff too,
Things I caught from Matthew: RAG loading indicator until files are avail. Get new models in UI rather than term.
Things I wish to add: litellm, groq adding pipelines and flows not just prompts, adding #folder to context, easier installer that checks docker etc with Houston assistant. OpenInterpreter integration. OpenUI integration + Developer Mode, Tester Mode, Rapid Feedback, Obvsf API Pools, Import ChatGPT & Claude History, Stars, Topics (connected threads), Timelines, Search Hist, Actions, Prompt Refinements, Self Improvement, Skill Library, Image Gen & Editing. Song Gen, Incognito thread, global/group chats. Sign in from phone/multi device. Agents and Clients Dashs.
I have been using it for a while. A great UI
@@fire17102awesome yeah please add all that stuff
You can actually skip the git clone step. everything is contained in the docker image.
f docker!!!!
Wow I just built a virtual AI girlfriend using ollama. I'm trying it on Llama3 model, and recently migrated to one of the uncensored models. Good-bye wife, hello AI - LOL.
I was fishing yesterday... unbelievably, thank you! Ps.: Can you make a video about WebUI + Open Interpreter + LLM Local or LM Studio? Thanks
I've been with you for over a year and it's been amazing watching you dominate this LLM news space. For example your snake game has become a standard in the industry now!!! Like you, I'm constantly in the LLM lab and I'm constantly coming across your name with a quote regarding a large language model. Awesome job carving out a niche - MUCH LOVE FROM NEW ORLEANS 🔥💪
Few more cool features
1. Image generation. I hooked this up to both local automatic1111 and DallE3 with api key. It's a bit of an odd workflow. You prompt it and the response will have a little pic button under it. I loaded a model finetuned for image prompts so response is cool.
2. Hook up openai models for chat choices with your api key.
3. Pull any llava model and you can hit the plus button to load a picture and ask questions about it.
As someone who is completely new, this is so damn confusing LOL first off, what is docker. Second, where are you and is this a windows terminal? Im extremely lost and maybe link your old videos?
I'm not sure if you've gotten any answers to your questions here, but I thought I'd try to help a bit where I can. To your first question, docker is a program that allows you to run software in something called a container. You can think of containers as a space in your RAM that is separate from the rest of the programs that are running on your computer. Docker is nice because it makes it easy for you to run things that are dependent on certain versions of Python , shared libraries, and other stuff without you having to worry about changing versions back and forth to make other things work on your computer. Secondly, from what I could see, buddy is on a Mac computer (the UI he shown at the top where there was a llama kind of was the giveaway) which unless he changed it to bash or something, means he is using a zsh terminal, but that's not important. If you are using a windows computer look up how to get WSL (a.k.a Windows Subsystem for Linux) on your computer and you should be able to follow most of the instructions.
Helpful tips:
Linux has different distributions but they fall into 3 main families- Debian(some people say Ubuntu but I'm my own brand of snob here), RedHat, and Arch. This is important to you as far as getting started working with linux because it'll determine your package manager. (Debian uses apt, RedHat uses dnf, and Arch uses pacman) they all have their own quarks as far as what they name things and how they function. But your going to want to ensure you have git installed in you WSL environment as well as having the most up-to-date version of things on there for security reasons to start off.
Learn about the basics of bash shell.
Learn the basics of git.
There will be some differences as far as folder structure once the program is installed, but I think you're smart of enough to figure that out 😉. Hope this helps.
Now go on out there and cry like the rest of us. 🫡
if you using docker you dont have to clone that git repo to run it.
Really awesome. This is something I’ve been looking for for a long time. The one I built myself is terrible.
Unsubscribed. This shit has been available for more than a year.
do we really need to do the git clone. I think it is enough we do docker run
I don’t usually post messages, but your video changed that. Very well done! I followed your steps, and within minutes, I had LLama3 running on Open-webUI with Docker on Ollama on a Windows computer. Thank you, Sir. Keep up the great work!
I HATE docker!!! this is all to confusing, grrrr! you all can do this, I'm stupid :(
How does it compare with Anything LLM?
Anything LLM is way clunkier, all models i have tried act like quite dumb
My go to channel for learning how not to write the game snake in python
It would be great to see a comparison between ollama and lm studio explaining the benefits and reasoningg of when to use each. The one thing i havent seen much of is how to leverage (if possible) other models from huggingface within ollama. This is easy to do in LM studio. For most other thigs i prefer ollama but i tend to use LM studio to test new models that ollama might not have readily available.
Same here. I prefer LM Studio and use it across networks hosted from a server. Not quite the same with ollama. Going to experiment with open-webui and see if it can connect to LMStidio with some code tuning.
Ask perplexity pro new model Bout it really good now based on llama 70b
Thank you. I just NEED to decide on which GPU to buy.
Btw. Why do 6% of the people in your poll believe GPUs should be regulated?
Maybe 6% of his audience are OpenAI employees. 😂
What's the advantage of this over LMStudio?
Not asking which one is better, but on what use cases they excel over the other.
For one thing, Open WebU is open source, which means that you'll be able to use it forever for free, for commercial or personal use (that and all the other advantages of it being FOSS, like more security, a nice and supportive community, etc.). That being said, LMStudio is easier to install.
I am new to LLM development. How do I get the documents(PDF) loaded and RAG setup in Ollama UI to be part of my development in AUTOGEN 2.0? If you load the documents with Ollama UI and I use Ollama as a local LLM for AutoGen, will I see the embeddings in my prompt data through AutoGen? Thanks!
Does it support image processing or is there anyway in which you could add image processing capabilities
Under settings there is a way to link this UI to an existing image generator like Automatic111 or ComfiUI.
Llava model allows you to ask questions about images. Hit + next to prompt field to upload pic. Then ask away. Wasn't great generating code from mock, but answered general questions ok.
Thanks for the vid, any Chance you May explain how to deploy such a chatbot for usage of customers or even implement or in Something Like c# Code?
Hey, seriously? I mean, I know the Walmart version is surprisingly fast, but come on, I'm using an RTX 3060 here! What's up with that?
can you link the video where you install docker.. i cant seem to find it..
How do you feel this compares to the newest LM Studio? It looks like it’s lacking on the advanced settings.
This thing needs docker engine unavailable for mac. Luckily, I'm a linux guy
Matthew, could be awesome a podcast with David Ondrej. He is inviting you in his videos. I'd love you both to talk about this tools. I'm using Anything LLM with LM Studio. What is the difference with Ollama with Ollama UI? thank you for your videos.
HEADS UP your first acct is your admin acct, if you update owu or ollama, or dkr, make sure you follow the instructions or it will lose your user name and password and you wont have admin anymore, and its a pain to straighten it out, doable, I did it, but a pain.
Are their any videos showing open we ui with agents, function calling and Web search?
Oh interesting, that documents feature I might have to try attempting to link it to all my Obsidian files. Ollama + Obsidian seems like a good match.
Does anyone know how to make the open-webui interface accessible from my phone on the same network? I'm used to being able to add --listen to a gradio webui and access it anywhere on the network, but I haven't seen anyone do that with this one.
I used yarn in the plane to discuss with phi3 about astrophysics. It was epic. At some point, it becomes slow (llama3 started out slow), so I would read for a minute something else and return to its response and continue the chat. It was great. I so hate the offline time on planes, but having a gpt (or 10) on you computer really helps. Also very funny how my battery would drop on burst every time I press enter, lol. It takes juice generating those responses. But again, it's pretty cool. I think at some point I wanted to install ollama ui but the docker part lost me.
I've been using text-generation-webui, with extensions, but I might have to switch if I can get this working correctly.
This comes so close
Quick question I am looking for a single installation that allows running LLMs locally and has a front end that includes text-to-speech and speech-to-text, with a backend that allows me to attach from the local network from other machines to provide LLM services the kicker is I would like to to add all interactions into a rag from both front end and back end services to allow "learning"
The idea is to have 1 pc providing all my AI needs locally for home automation, assistance, and various library/research/content generation services on my local network and continuing to learn about me and my needs as time passes (ultimately to be my interface with the internet in general) and help/lead would be greatly appreciated.
It´s wonderful, but how can I add Hugging Face models to Open WebUI also ? I have tons of them ! ❤❤❤
Thanks! Great video!
Can you do bit different scenario : LLM engine (Ollama, llama.cpp or something else) running on dedicated hw with GPU acc. and frontend in local machine (vm/container/conda env).
I'v been using ollama on hw with GPU's and frontends in VM or container since they started supporting OpenAI API .. but very mixed results. Idea is that opensource LLM backends are quite stable now .. few upgrades, new features and bugfixes. Just download new model when something interesting appears.
But frontends and other implementations are still Wild West and better kept in easily recyclable format :)
I have tried the blulk load feature in the documents , it does not work , cannot change the embedding models as well , after the restart it keeps defaulting to the original. There is a ticket in the github for exactly this . However I have not heard back from the team.
If it's local why does it need a sign in. There should be a way to create a version that doesn't need that
Another superb, insanely useful how-to vid!
A noob question: Setting this up with Docker vs Anaconda, the config with Python version and all the supporting libraries and package becomes the default for that user account on your computer, correct? (I’m on a Mac like you, and don’t think this is an issue at all for me, just wanted to understand the setup better.)
Instead of Model centric, I wish an agent centric UI with RAG capability
Now what if we made your own model ! and we want to use that model on your website how could we do that , any tools easy for non programers
Does it have a memory feature? Wonder if it can be used as an ongoing "database" per-se
This sucked. appears only for the ultra technical who can use stuff like docker.
Hey! Love your videos! and yes, I'm your first commenter of this video
Supposedly they are working on implementing a perplexity style search too! Pretty slick
Like web?
@@jarad4621 Yes web
If they get something like that it's game over for the rest of the opensource interfaces!
Can you use page-assist with openweb ui?
Why would I use this over autogen? Looks pretty similar
Awesome video. Can you, going forward, mention the system resources needed for the local stuff like that, to run it at reasonable speed? Thanks.
Did not explain how to update and still keep all the settings intact!
I dont think i understand, can this be used as a UI for a particular platform or device, or is it just a standalone thing? I mean, would this work as a UI instead of windows or Android, or is this a toy?
its local on your pc
Absolutely amazing, thanks!
I wonder how it could be best accessed via iPhone GUI?
there is no voice calling feature in it like chatgpt has now on the phone right?
its on Pinokio as well
Yes I knew talking about anticipation would put you over the top to release😂
There is an agentic system that runs on Groq, might be worth checking out, I can't remember the name.
AutoGroq?
This is fine. but AnythingLLM FTW!!!! XD
Thank you. Best tutorial on youtube. Very clear.
can I run an remote model with api key on this?
I wonder if there will be a tool called Barack Ollama
Question about these embedding models. I _thought_ that adding your own document makes and "embedding" loaded into the LLM for you to search your document. But what are those embedding models that you can define in the gui settings? I can't quite figure out why i'd want them or what they are used for..
I just learn this myself. Apparently embedding is a seperate thing or app or code that does that part so you actually need the database and an embedding model only to be able to save stuff in. vector you don't need the llm yet only once it's saved by the embedder then you can retrieve from the db with your lmm and rag, the main llm doesn't save the data only fetches it later
@@jarad4621 Thank you for your reply! I've read it about 10 times but still don't think i'm following you entirely. I get that you don't train your main LLM for your custom data. That's what the whole RAG thing is about. But with RAG you end up with a vector database that your main LLM can use to provide better results for specific subjects. Thus far i get it.
But what is that "embeddings llm" like "nomic-embed-text"? I can't figure out how you use it, where you use it or what to do with it.. Just guessing here, or is there some kind of model hierarchy? Does it follow a path like llama -> nomic-embed-text -> own vector db? In which case the "nomic-embed-text" would be the one interacting with my data, not llama like it would be in a classic RAG setup? 99% guesses here, i just don't know.
I'm not on a Mac. :(
*** I posted this for someone else here thought it may help you a little bit too. If anyone wants to correct or add anything please do.***
I'm not sure if you've gotten any answers to your questions here, but I thought I'd try to help a bit where I can. To your first question, docker is a program that allows you to run software in something called a container. You can think of containers as a space in your RAM that is separate from the rest of the programs that are running on your computer. Docker is nice because it makes it easy for you to run things that are dependent on certain versions of Python , shared libraries, and other stuff without you having to worry about changing versions back and forth to make other things work on your computer. Secondly, from what I could see, buddy is on a Mac computer (the UI he shown at the top where there was a llama kind of was the giveaway) which unless he changed it to bash or something, means he is using a zsh terminal, but that's not important. If you are using a windows computer look up how to get WSL (a.k.a Windows Subsystem for Linux) on your computer and you should be able to follow most of the instructions.
Helpful tips:
Linux has different distributions but they fall into 3 main families- Debian(some people say Ubuntu but I'm my own brand of snob here), RedHat, and Arch. This is important to you as far as getting started working with linux because it'll determine your package manager. (Debian uses apt, RedHat uses dnf, and Arch uses pacman) they all have their own quarks as far as what they name things and how they function. But your going to want to ensure you have git installed in you WSL environment as well as having the most up-to-date version of things on there for security reasons to start off.
Learn about the basics of bash shell.
Learn the basics of git.
There will be some differences as far as folder structure once the program is installed, but I think you're smart of enough to figure that out 😉. Hope this helps.
Ollama UI only accept 1 OpenAI-compatible servẻ end-point (vLLM), while accepting multiple GGUF. Weird !
Great video! I installed Open WebUI on top of Ollama yesterday. It's freaking awesome. Looks and feels just like ChatGPT. Keep up the good work on your videos. I think they are "best of breed" when it comes to AI videos on TH-cam. By the way, the response tokens per second in your video is very high. What is your hardware, especially your GPU(s)?
Is there a way to make custom ui?
I have already downloaded many models that I use with Oobabooga and LM Studio (Thank you for suggesting LM Studio, it's a good software. Shame it does not support EXL2). But, I don't want to download models specific to Ollama.
Will try it This coming week! Have you or anyone tried the "Chat with RTX" by NVIDIA? I am trying to go through the setup giving it directories to intake BUT it failed on most expecting zip files... I recall you had covered "Private GPT" before, would it be able to do the same thing intended for Chat with RTX? I bought a new system with a 4060TI 16gig and 32 gig RAM, 1 TB SSD with 2 TB HDD, windows 11... It should run well but currently crawling while Parsing Nodes... any guidence woudl be greatly apreciated!
Great video thanks for sharing. Sam Altman talks about GPT 4 being an embarrassing model. Hopefully his team see this video and are equally embarrassed about their piss poor UI.
@Mathew ollama with webui on Linux os
I really wish people would start being more original with their ui. If I wanted my front end to look just like ChatGPT then I would just use ChatGPT free lol. That’s why I like LMStudio. Only thing holding me back from using it more often is the lack of voice features.
Hey matt, can you tell us how you setup your terminal? (which terminal software you use, addons, etc). Was also hoping you can create a video testing which LLMs do the best at text summarization for large documents / transcripts
What we need is multi-prompt templates (series of prompts, one at a time), including step repeat. This way we can have the LLMs reflect on its previous answer before executing the next step in the series.
i run it without docker or ollama lol.
So helpful! Thank you!!!
I should add that your style is just as engaging as the dude from The Why Files channel. When are you going to get a cool sidekick like HeckleFish?
Still not sure about the name, Open WebUI. Very generic and not descriptive of its functionality.
Thanks for creating this video with clear instructions!!
After running this UI with llama3, the responses that I get is very slow even to those simple questions, e.g. how are you?. Is having GPU in my machine must to use this with proper speed?
Your video about this software comes way too late to be honest. The guy behind this project had made this thing awesome like year and a half ago. But it is nice that you did it anyway people need to know about this project and support it since rly rly good. Keep up the good work ;)
0:50 "Look how fast that is" well, the response being fast is not a function of the UI, it's a function of the hardware you're running it on. so please.
We have been using Chabot-ui and enjoy the built in tools and assistants libraries.
Keeping an eye on these other repos until a true leader emerges.
Does it have web?
@jarad4621 web search? It needs to be added as a tool/skill.
What HW are u running the LLM on?
Another extremely informative video! I have an idea for a follow-up video: I would love to see a tutorial on how to secure the docker container with https / ssl. I am trying to figure this out through experimentation and have so far been unsuccessful. Thank you again for your great content!
MSTY is way better. I’ve actually switched to MSTY for my work.
Impressive, this is having all - except support for my local language (what ChatGPT do have).
I prefer ooba text Gen webui as it’s more customizable. The RAG implementation is not something I’ve tried before. I guess I’ll give it a try.
Great video. Thanks! fyi - In the install section, it's not necessary to clone the repo before running docker.
Nice solution with the docker!
Can it handle the processing of documents with tables? Will an OCR like Tesseract be needed? Any suggestions
have you tried Silly tavern? its more geared towards characters and roleplay but has some nice features and a fully customizable ui
what gpu server provider you use for your language model deployment? or do you only need gpu power for training?
can you create chat agents from those ollama deployed chat models?
im just getting into stuff and some of your videos are nice to follow
in case this helps someone. to run linux images on windows, you need to switch to linux containers, which will use wsl2. you do this by rightclicking on docker running in taskbar, and selecting switch to linux containers.
Could this work on Jan?
Thank you for your clear documentation, really helpful to setup a complete system in a few steps. Great job!
amazing tutorial, thanks!!
how do you know the chunking methods for the embedding?
I was hoping someone else would handle all the heavy lifting with the UI, as it was somewhat of a turn-off for me to read replies in the shell. Thank you for sharing it!
Can we generate plots on this
I clicked hella fast because I thought I missed an update from ollama XD love openui though!
just make sure you don't use update models functionality if you have some models as ollama author pointed, until they implement model hash comparisons
we dont need fast inference we need quality inference, im willing to wait minutes for responses for complex questions as long as they are coherent
Anyone know which terminal software Matthew uses for his? That auto complete is a nice feature and would love to add it to mine