Thanks for this awesome tutorial. I took it as a reference and built a user_id based map to keep history in an in memory database. This helped me to keep history for each user. { "user_1" : [{}], "user_2": [{}] }
@@gears2525 Hey why not use a flask API and you load your local Ollama in that and try to connect that with flask and share it to the other PC Maybe it'll work
Is there an example of a chat UI application that uses the ollama inference endpoint and is then deployed in the cloud (AWS, GCP, etc)? I have managed to create the app and it is running on local, but I'm struggling to deploy it in cloud - specifically, I'm stuck at creating an appropriate Dockerfile, as it seems there needs to be two deployments, one for the ollama inference endpoint and one for the UI. Therefore, an example showing how it's done would be awesome!
I havent built out any GUI's because its hard to improve upon the usability of the cli in this case. Everywhere else I prefer a gui, but here none of them are good enough to beat out the cli. If you want to deploy with docker, there are plenty of tutorials online to get you up to speed. A friend, Bret Fisher, has a popular Docker course. But no need to start there. Just deploy it to the cloud without docker. Spin up an instance somewhere and run it. The bigger challenge there is securing your system. If you don't need to share it, then look into tailscale.
When I use Ollama from the terminal with the llama3 model, it works very fast, almost instantly. However, when I try to make a request to localhost from the same machine using curl, it is incredibly slow. Why could this be?
What is the command you are running? The cli uses the same api as curl. There is no difference. Best thing to try is to go to the cli then type ‘/set verbose’ and then ask the question. Then in the curl look at the stats in the final json blob and compare the numbers.
If ollama itself is not multithread and async, how much wrapper like this can help? I have 2 laptops, A and B. 'A' has a database with more than 10k records in a table and code written Scala to fetch data from the database. 'B' has ollama running on it and llama3:latest model is loaded on it. If I am fetching data from the database from 'A' and sending it to chat and generate endpoints API of ollama on 'B'. I observed that ollama responds to the chat or generate request within few milliseconds like 100, 125, 200, 300. But that's just in the beginning. Later on the response time increase to 10 minutes and about for a request by the time 2k requests is send and still 8k requests still need to send. From this behavious, it doesn't look that ollama is supporting concurrency. I created a python script with Flask to achieve async but the behaviour from ollama remained same. Do you know to solve this problem? Or is it like there no problem and hence no solution? OLLAMA_NUM_PARRAL value was 4.
If you are seeing long times like that there is something wrong with your setup. You should ask on the discord and try to resolve the issue with your setup.
@@technovangelistLove these videos keep them coming! I know you can produce embeddings from ollama, and can store them in a database, but I don't know how that is useful. Can you explain?
@@technovangelist I get a delayed response (while the AI is producing a response), once the response comes back, all the chunks are received in a single response, rather than getting the chunks one at the time (i.e. the time taken to generate a complete response is the same, but the chat api lags, and only send a response after the complete response is generated, even with stream=True). This is vastly different to the generate api, where one character is sent back at a time.
Thanks for this awesome tutorial. I took it as a reference and built a user_id based map to keep history in an in memory database.
This helped me to keep history for each user.
{
"user_1" : [{}],
"user_2": [{}]
}
That awakward silence at the end though :D
Thank you sir for adding this QoL. Super helpful indeed!
Thanks for writing this terrific server, models, and tools!
Glad you like it!
What an absolute legend
For some reason, I’m unable to hit the endpoint from another computer on the same network
@@gears2525 Hey why not use a flask API and you load your local Ollama in that and try to connect that with flask and share it to the other PC
Maybe it'll work
I actually had no idea you were part of the ollama team too thats super cool
I was but not anymore. Focusing on videos
Still super cool! Love the vids! Keep em comin!@@technovangelist
How can I do those requests from another pc connected to the same network?
Is there an example of a chat UI application that uses the ollama inference endpoint and is then deployed in the cloud (AWS, GCP, etc)? I have managed to create the app and it is running on local, but I'm struggling to deploy it in cloud - specifically, I'm stuck at creating an appropriate Dockerfile, as it seems there needs to be two deployments, one for the ollama inference endpoint and one for the UI. Therefore, an example showing how it's done would be awesome!
I havent built out any GUI's because its hard to improve upon the usability of the cli in this case. Everywhere else I prefer a gui, but here none of them are good enough to beat out the cli. If you want to deploy with docker, there are plenty of tutorials online to get you up to speed. A friend, Bret Fisher, has a popular Docker course. But no need to start there. Just deploy it to the cloud without docker. Spin up an instance somewhere and run it. The bigger challenge there is securing your system. If you don't need to share it, then look into tailscale.
When I use Ollama from the terminal with the llama3 model, it works very fast, almost instantly. However, when I try to make a request to localhost from the same machine using curl, it is incredibly slow. Why could this be?
What is the command you are running? The cli uses the same api as curl. There is no difference. Best thing to try is to go to the cli then type ‘/set verbose’ and then ask the question. Then in the curl look at the stats in the final json blob and compare the numbers.
If ollama itself is not multithread and async, how much wrapper like this can help?
I have 2 laptops, A and B. 'A' has a database with more than 10k records in a table and code written Scala to fetch data from the database. 'B' has ollama running on it and llama3:latest model is loaded on it. If I am fetching data from the database from 'A' and sending it to chat and generate endpoints API of ollama on 'B'. I observed that ollama responds to the chat or generate request within few milliseconds like 100, 125, 200, 300. But that's just in the beginning. Later on the response time increase to 10 minutes and about for a request by the time 2k requests is send and still 8k requests still need to send.
From this behavious, it doesn't look that ollama is supporting concurrency. I created a python script with Flask to achieve async but the behaviour from ollama remained same. Do you know to solve this problem? Or is it like there no problem and hence no solution? OLLAMA_NUM_PARRAL value was 4.
If you are seeing long times like that there is something wrong with your setup. You should ask on the discord and try to resolve the issue with your setup.
Very interesting, I had no idea. What possible roles are there besides "user", do we just make them up or is there a predefined set?
The three possible roles for now are user, system, and assistant
@@technovangelistLove these videos keep them coming! I know you can produce embeddings from ollama, and can store them in a database, but I don't know how that is useful. Can you explain?
is there a way i can fetch the whole response in a single response object
Sure. Set streaming to false
Hello 👋 can we connect our local ollama with a self hosted N8n server ?
Hi Matt! Thanks from the awesome work! Is there a way to include vLlm into Ollama?
I don’t think so. They are alternative ways of doing the same thing
Is some way that you can help me with my proyect ? thanks from Chile Claudio
separate question. What's that browser? It has a very nice interface. TIA
That’s Arc. It’s great.
@@technovangelist thank you kindly sir! Please do more videos 🙏
Why does the chat api respond all at once, even with streaming turned on?
Not sure what you mean by that
@@technovangelist I get a delayed response (while the AI is producing a response), once the response comes back, all the chunks are received in a single response, rather than getting the chunks one at the time (i.e. the time taken to generate a complete response is the same, but the chat api lags, and only send a response after the complete response is generated, even with stream=True). This is vastly different to the generate api, where one character is sent back at a time.
are those models fine_tunable ??
Most of the models used in Ollama are fine tunable.
Do you have github repo for this video?
github.com/ollama/ollama/examples
Is there a link to doscord channel ?
There is a link on the main website for ollama
hey can you post a github link to the code?
That’s a good point. I find it and post
@@technovangelist did ya manage to get it?:D
Oh, its already in the ollama repo
Yes, its in the main ollama repo
Hi! Thank you for a nice and descriptive video. Can you please link to repo? Can't find it :)@@technovangelist