nice clip jan, thanks its working like a charm. hardest part was getting rid of amazon aws stuff in vs code that kept on installing amazon Q and stealing the code completion ^^.
Thanks for the interesting video. Can I run this setup on a PC with a RTX 2060 GPU having only 6 GB of VRAM? Or do I need at least 4,7 GB (llama3.1:8b) + 1,8 GB (starcode 2b)? Or does my GPU VRAM only has to be large enough so that the starcoder LLM can fit into it?
From what I could do with a 3070, take the lighter/medium ones so that your GPU won't struggle. Well, now I have a 4090, and I still don't know what can stop this card, lol.
I don't know if I did it right but after 8 months from publishing this video some things were different. So I copied the "models" array of your config.json file, which is: "models": [ { "title": "Gemini 1.5 Flash", "model": "gemini-1.5-flash-latest", "contextLength": 1000000, "apiKey": "AIzaSyAIJ49oNQAMSEionN0v5R7fWE1fHgmeMuo", "provider": "gemini" }, { "title": "Llama 3", "provider": "ollama", "model": "llama3" }, { "title": "Ollama", "provider": "ollama", "model": "AUTODETECT" }, { "model": "AUTODETECT", "title": "Ollama (1)", "completionOptions": {}, "apiBase": "localhost:11434", "provider": "ollama" } ],
@@scornwell100 did you check the issues listed in their GitHub repo? github.com/continuedev/continue You can also join their Discord server to get more detailed help: discord.gg/vapESyrFmJ
wish you posted your full config.json on pastebin or somewhere since default config is not even near that you show and guide is useless without full setup
i cant believe no one else here mentions this in the comment section, he just skipped like a couple of steps that he didnt even mention just make a very short video.
I run it on 16 GB. The processor and GPU are quite important for Ollama as well. 16GB RAM are recommended: github.com/open-webui/open-webui/discussions/736#
I run it with 11GB of VRAM from command line and it seems fine, but inside VSCode I can't get it to respond, it throws errors that the stream is not readable.
@@PedrinbeepOriginal Not really. Granted, I don't do much other heavy work when I'm coding but it runs super smooth with the 16GB. The only time I wish I had more RAM is when doing video editing lol
@@iamjankoch Building a code assistant agent to work inside VSC editor, to help me with other projects. actually I will ask for your advice: I have a trading bot written in NodeJS react while this project written by another programmer, for some parts struggling with it development. like to have a useful AI assistant to help me find bugs and understand the app structure. the app runs server with browser client while it have about 130 files. tried to use open AI GPT but it's to many files for it, while it's loosing the context, with the other issues it have. I came into the conclusion that the best way is to have a local llm running on my machine. if you have any recommendations for the right AI assistant you would use, I'll appreciate you advice.🙏
forgot to mention how much you need to pay for a computer that can handle this otherwise it will not even start or take literally ages to do anything or you have to use the lowest llms that cant do s&T(t. it's all hype to get money into openai and otehrs like
Mine still working without the apiBase field... Tutorials on Continue and Ollama is so rare. Thank you man
@@hackeymabel1617 so glad you got value from it!
nice clip jan, thanks its working like a charm.
hardest part was getting rid of amazon aws stuff in vs code that kept on installing amazon Q and stealing the code completion ^^.
I'm glad I didn't have AWS connected to VS Code in that case :D Glad it worked well for you!
Thanks for the interesting video.
Can I run this setup on a PC with a RTX 2060 GPU having only 6 GB of VRAM? Or do I need at least 4,7 GB (llama3.1:8b) + 1,8 GB (starcode 2b)?
Or does my GPU VRAM only has to be large enough so that the starcoder LLM can fit into it?
From what I could do with a 3070, take the lighter/medium ones so that your GPU won't struggle. Well, now I have a 4090, and I still don't know what can stop this card, lol.
@sifly4683 thanks for the answer.
I tried it and it works with my 6GB GPU and qwen-coder:3b
I don't know if I did it right but after 8 months from publishing this video some things were different. So I copied the "models" array of your config.json file, which is:
"models": [
{
"title": "Gemini 1.5 Flash",
"model": "gemini-1.5-flash-latest",
"contextLength": 1000000,
"apiKey": "AIzaSyAIJ49oNQAMSEionN0v5R7fWE1fHgmeMuo",
"provider": "gemini"
},
{
"title": "Llama 3",
"provider": "ollama",
"model": "llama3"
},
{
"title": "Ollama",
"provider": "ollama",
"model": "AUTODETECT"
},
{
"model": "AUTODETECT",
"title": "Ollama (1)",
"completionOptions": {},
"apiBase": "localhost:11434",
"provider": "ollama"
}
],
Doesn't work for me. I can do it from command line but Continue plugin seems not working at all. Did all configuration and it responds with nothing.
@@scornwell100 did you check the issues listed in their GitHub repo? github.com/continuedev/continue
You can also join their Discord server to get more detailed help: discord.gg/vapESyrFmJ
What about using rtx 4060ti 16 vram with ryzen 9 5950x could works well ?
@@rodrigoaaronmartineztellez3572 yes, that should handle Ollama quite well
wish you posted your full config.json on pastebin or somewhere since default config is not even near that you show and guide is useless without full setup
Here you go: gist.github.com/jan-koch/9e4ea0a9e0c049fe4e169d6a5c1e8b74
Hope this helps
i cant believe no one else here mentions this in the comment section, he just skipped like a couple of steps that he didnt even mention just make a very short video.
Dope. I'm going on an RV trip for two weeks so will have spotty service but can still ship 🚢
Sounds awesome, enjoy the trip!
Is 8gb RAM sufficient? I have enough Storage but when I try to use this after installing, it just doesn't work. Keeps loading.
I run it on 16 GB. The processor and GPU are quite important for Ollama as well.
16GB RAM are recommended: github.com/open-webui/open-webui/discussions/736#
I run it with 11GB of VRAM from command line and it seems fine, but inside VSCode I can't get it to respond, it throws errors that the stream is not readable.
This is awesome. Currently using codeium, but will install good later and give it a go.
@@codelinx let me know how it goes 💪🤖
Thank you! I was looking for this. Wat are the specs of your mac?
Glad you enjoyed the video! It’s a 2023 MacBook Pro with M2 Pro and 16 GB RAM
@@iamjankoch Nice, thank you for the fast answer, I was thinking if I need a M3/M2 Max with a lot of ram to load Llama 3 in the MacBook.
@@PedrinbeepOriginal Not really. Granted, I don't do much other heavy work when I'm coding but it runs super smooth with the 16GB. The only time I wish I had more RAM is when doing video editing lol
Great video ! really helped me setting things up!!
@@caiohrgm22 glad to hear that!!!
Great video my friend. Worked like a charm.I thank you very much!.
Glad to hear that, happy coding!
simple and great explanation ! thank you
@@Ilan-Aviv you bet, glad the tutorial was useful! What are you building with AI?
@@iamjankoch Building a code assistant agent to work inside VSC editor, to help me with other projects.
actually I will ask for your advice:
I have a trading bot written in NodeJS react while this project written by another programmer, for some parts struggling with it development.
like to have a useful AI assistant to help me find bugs and understand the app structure.
the app runs server with browser client while it have about 130 files.
tried to use open AI GPT but it's to many files for it, while it's loosing the context, with the other issues it have.
I came into the conclusion that the best way is to have a local llm running on my machine.
if you have any recommendations for the right AI assistant you would use, I'll appreciate you advice.🙏
hatur nuhun pisan ... sehat sehat terus kang
Thnaks alot! =)
You’re welcome!
Great explanation
@@tanercoder1915 thank you!
codearea is to small cant see
Thank you so much!
@@2ru2pacFan glad you enjoyed the tutorial!
luv u bro
@@user-13853jxjdd glad you enjoyed the video!
forgot to mention how much you need to pay for a computer that can handle this otherwise it will not even start or take literally ages to do anything or you have to use the lowest llms that cant do s&T(t. it's all hype to get money into openai and otehrs like
@@josersleal I have an M2 Pro MacBook Pro with 16GB, for reference