Cole, you’ve done an outstanding job! Your videos consistently make complex topics clear and easy to follow, which is a rare talent. I appreciate how you break down each concept, making it accessible for viewers at all levels. Your teaching style not only engages but also inspires confidence in learning. I’m eagerly anticipating your next iteration. It’s always exciting to see how you evolve your content and introduce new ideas. Keep up the fantastic work, and thank you for sharing your knowledge with us!
Thank you very much!! I really appreciate the level of detail in your kind words - it means a lot to me to know that my work is noticed in such a thoughtful way and that I'm hitting my target of breaking down complex topics well!
@@ColeMedin Yes indeed! You explain topics with such a good flow between ideas and concepts that rivals that of other popular tech youtubers such as Networkchuck, Linus Tech Tips and Christian Lempa
I have to agree. Literally just getting started with local AI. Was about to skip past this video and thought, maybe it’s something I can use that I didn’t know existed. BAM! This video is going to be my beginning into what my vision is for my local AI. Really appreciate you made this understandable!
@@Hatch3dLabs the author tells well, and help me understand more clearly. I am getting started too, what tool are you using? I'm using xark-argo/argo on github, it's simple to install on Mac and Windows, what about you? I'd like to keep in touch for more learning.
This is a very good step-by-step tutorial. Following the instructions in this video will get you started with local AI. For people trying M1 and above, the ollama must be installed separately, and the rest are the same.
@@ColeMedin If you want to use GPU/NPU-accelerated LLM rather than CPU on Apple Silicon (which doesn't have either AMD or Nvidia GPUs), you'll need the actual Ollama desktop app on your host Mac and pull the models from there rather than using the Ollama container in the docker compose file. That's why in the Quickstart they call for starting the Docker compose stack without a profile - it doesn't even start the Ollama container. Docker will still be able to reach Ollama using the docker internal hostname, but you'll get much faster processing using the Apple Silicon GPU and NPU (what pieces are accelerated depend on what the Ollama team have done with the app over time). It took me a few minutes to figure it out, but once I did it works just fine.
@@scottstillwell3150 Since I could not open the credentials, I tried to setup new ones. They say they could connect, but I am not able to use the ollama node in the demo workflow. It can't fetch any models. This is super confusing.
By far the best tutorial and overview on Local RAG and also dropping gems on the little improvements you've made from the original repo. Workflow is amazing too!! One of my ideas is playing some of older rpg's back in the day on the steam deck but with less time that I have now for other priorities, its nice to just query the walkthrough docs and ask where to go next etc.
Genius, this is like a “medior ai engineer” tutorial video if someone builds the same thing then tweaks it to make a unique llm app out of it. I think a lot of companies would appreciate their engineers to know all this
Local is a good start. As a practical application, I think a good project would be to distribute the containers and have a login system for businesses.
Yes I definitely agree! Wish I could cover that here without making the video way to long, but I will be making content on this kind of thing in the future!
Yes definitely!! I love using pgvector so I'm 100% behind you there. I focused on Qdrant in this video just to show the entire package, but often times using less services (so using Postgres both for chat memory and RAG) can be the way to go if you find it works well for your use case.
@@ColeMedinthat was my question answered 😅 simplified the stack, if you get it to work with supabase you have all the db you need for different functions in this pipeline
Outstanding work, Cole. Love it. I will implement it today. Looking forward to more of your excellent content. You are not verbose, just straight to the point and deliver value to boot. Thank you!
15:20 this is truely the most important part of the logics. It's absolutely necessary to have a function in order to handle contingency regarding file duplicates
This looked great, right up to the point you try to access the web interface and you find you can't proceed until you have made an account with n8n. I must have missed where that was shown in the video.
Something I'd like to see is building in mixture of agent frame works and tool use and an actual chat interface. This is a great start and that is exactly what I'm going to start working on lol
Your doing a great job keep making content much appreciated ....i have some amazing ideas but unfortunate dealing a shitty computer phone n lack of the 5 months spent know i realize why i struggled so hard ...so i thank you for clear explanations.....if u can help be so grateful or anyone their as soon can enough or hit streets and hustle im buying anew computer cheers everybody
Thank you sooo much.. I had to change the document import a bit to work with local network shares for my company but it works .. SO GOOD. The deleting of documents already in the system before adding more is really important, ** I cant wait for your front end video **
My pleasure!! I'm curious - what exactly did you have to change for local network shares? I'm glad it's working great for you! I can't wait to add the frontend into this setup - I appreciate you calling that out :)
@@ColeMedin I am happy to do a short clip on YT showing the local file import change, we are a large mining house and have hundreds of policies "PDF" now you can ask the AI assistant to compare policies from different operation / counties and highlight inconsistencies, or to find the one relevant to you or just to summarize the content to get you up 2 speed.. will reply with a link to the clip :)
I have been trying to make Llam3.1 work using llama-stack but felt it was too complicated or still unfinished. Docker and Postgres? Oh yeah, this one sounds more like it for me! Subbed.
Great content, thanks! What minimal computer specs do you recommend so the response feels natural. I want to connect home assistent to it, so it can ve controlled with vocal commands.
Thanks for video. A lot of web ui chat tools compatible with ollama nowadays can now do the RAG just right out of the box. Like Open Web UI. Auto triggers part with n8n is a good one, if you need to automatically process a lot of documents.
Yes, would love to see setting like Redis caching, supabase auth, next.js project, stripe payment system to have some template for a Saas. God bless you
This is looking fantastic, I shall try this later, but I suspect maybe a little too much was skipped for my abilities. We shall see. Thank you either way for the video :)
Amazing video Cole. Quick question, I downloaded the json file and try to use it but looks like the Clear Old Vectors node doesn't exist? Also, if i do this with pinecone and open AI how would i change the code? Thanks man
Thank you! That's really weird... it could be a difference in versions of n8n. There's just nothing there where the Clear Old Vectors node should be? For using Pinecone you would just have to swap out everything Qdrant with Pinecone! Also check this out for clearing old vectors with Pinecone: community.n8n.io/t/enabling-upserts-for-vectorstore-using-langchain-code-node/47633
Thank you!! I have a link to install Docker Desktop in the description! I can put it here too: www.docker.com/products/docker-desktop/ It's a very quick and easy install!
Thank you - I'm glad it was easy to follow! I haven't encountered this error before... did you make any custom changes to the workflow by chance? Otherwise, it seems like it could be an issue with your Supabase credentials since it's an authorization error. I'd check to make sure the API key you set for the creds there in n8n matches the one in your project
Great work Cole. I plan to set up RAG for my business as I’ve followed RAG developments for about a year. Things have come a long way. I plan to model your work and would like to connect to Supabase since I plan to use for some of my other App work.
Awesome video, but I think I am a little bit confused. When I follow the instructions in the video, it seems like I am basically installing the original n8n version, as this is what we are pulling. How do we actually get your version with the custom workflow?
Thank you and I understand the confusion! My version of n8n isn't anything special besides the custom workflow. But I have the custom workflow as JSON file you can download from the link in the description, and then you can import it into your n8n by creating a new workflow, clicking on the options in the top right, selecting "Import from file" and then selecting my JSON file once you download it to your machine.
There are two more additions that need to be added for this local dev environment. Open Web UI and ceph nano with S3 enabled With this you have your own local dev cloud environment then you can build functions and tools in open web UI that call n8n workflows, and store files using S3 protocol
Amazing Cole! at 20:18 when you download files from the google drive, where are they downloaded too? If instead of getting files from google drive, I get it from my local computer (mirrored in the container in data/shared) with the Local File Trigger, do I need the step of downloading files? Thank you so much, I subscribed already!
Thank you very much! The Google Drive files are downloaded inside the container running n8n. So if you are using the local file trigger you don't need the download step!
My issue at the moment is I believe I followed all the steps, but I am unable to connect to the Postgres account. I get an error saying password for user "root" failed. I tried the default 'password' and also the one I set in Visual Code Studio whilst following along with your steps, but neither work.
I'm glad you found it helpful! For running an LLM locally to help with coding, you could use a really lightweight model like Microsoft's Phi3 Mini! It does depend a lot on your hardware though. If you have a really good GPU (8-16GB+ of VRAM), you could run any of the 8b parameter models to help you with coding without sacrificing performance too much.
That's what I thought too but I wasn't able to connect to Postgres within n8n until I exposed the Postgres port! Maybe I was just setting up the URL incorrectly at that point, you could totally be right. But that's what ended up working for me!
Have you looked into how to extend the intelligence by using `o1-preview`, `o1-mini`, `claude-3.5-sonnet`, `4o` and so forth as high-level thinkers/orchestrators that manage many small agents to pull intelligence in and process?
You sir are speaking my language! haha I have been experimenting with this a bit with some initial success. I'll almost certainly make a video on this kind of idea in the future!
First off, amazing work, I'm puzzled why you chose to use google drive if the aim is to be local? I'll be using your workflow as inspiration for local files, not knowing n8n, can it work with local files/folders? lets hope I can get it working! Thanks for sharing.
Thank you very much! And that is a really fair point! I have worked with businesses in the past where they wanted a local AI solution, but they were fine storing sensitive information in Google Drive. The biggest concern is more sending their data into a cloud LLM that will be trained with it. In the end, cloud storage is just so convenient that almost no one wants to store all their files locally (on something like a NAS box) even if it's sensitive data. So that is why I consider this setup fully local even though it uses Google Drive. If you do want to work with local files, n8n can certainly do that! I would check out the local file trigger for n8n: docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
@@MustRunTonyo Honestly I'm not sure how my comment got removed! TH-cam says it is there still but when I try to view it I don't see anything. All I said was Google Drive is generally considered "safe" to store company data. Numerous businesses I work with have sensitive data there even though they want to run LLMs/RAG locally for privacy. Of course it's up to each business to determine if this is really the case for them! However, I am looking into using the local file triggers in n8n to make this setup entirely locally! I will probably be making content on that in the near future!
@@ColeMedin There are many existing self-hosted n8n users, and we don’t want to start from scratch. Hopefully, this idea can inspire you to create a tutorial on how to onboard the AI starter kit with an existing DigitalOcean self-hosted setup 🙂
Very nice video, thank you. Just a question, any ideas about the minimum configuration for a server running LLama3, RAG Stack and more, to have useful performance? On my Mac silicon LLama3 take minutes to answer. I'm talking about RAM size GPU type CPU numbers and so on. Thanks
You bet! And good question! Could you clarify which version of Llama 3 you are trying to run though? As in number of parameters (8b, 70b, etc.). You could also try a tool like Air LLM to run larger models on not as powerful as hardware: github.com/lyogavin/airllm
Great question! Everything you mentioned here is fantastic. I've more recently liked Claude 3.5 Sonnet more than GPT-4o, but that would be the only thing I'd maybe change. Once o1 supports function calling in the API though, OpenAI will probably be the way to go again!
Jarad I appreciate you replying to so many comments on this video! There's a few things that could be considered missing for production that maybe you're thinking of. Some sort of logging/tracing tool like LangSmith, or maybe caching using something like Redis? Also some sort of reverse proxy service like Nginx would probably be needed to host this in a VPS.
installed docker desktop and git desktop on W10 (12 cores, 48GB machine, Nvidia GT710) and it seems to keep blue screening on me. WTF? never did that before. Pretty sure it is docker desktop and WSL that is the problem, I don't know why.
Hello! Thank you for sharing this wonderful video! I am interested in changing the workflow, specifically the source of the knowledge base files. Can we upload from a local source like my system C drive? Perhaps we can try to upload to the Docker storage and use this as a source?
I created a folder on n8n container using docker, added it to the yaml file under the n8n service>volumes section , format .\Folder:path, then tested the imported using docker cp command. I can import in the local file trigger but stumbling around the conversion and import to vector DB. Any recommendations? It looks like the google trigger takes in the data as a binary file(text), and sends it to an Extract trigger. I need to replace the steps where the Download File is completed and the Extract step is completed but using local path and PDF file.
My pleasure George! Great question and I see you figured it out yourself - thanks for following up on that! I'm sure others will benefit from what you've done here too.
Great questions! So the API key for Qdrant can actually be anything since it is running locally. That parameter isn't actually used (because it is local). Which also means it is fully local to your second question! It is just set up this way for n8n to make the node compatible with the hosted version of Qdrant if you wanted to use that.
Nice video thanks! Is any way to create a permanet memory based in the conversations with the chat bot? I mean: I just with the bot and it keeps all the learnings from this conversation for ever. Thanks
Thank you, my pleasure! :) This kind of thing you're describing where an LLM can learn from conversations over a long period of time is definitely possible, but it isn't a quick and easy implementation! Essentially, you have to tell the LLM (usually in the system prompt) how to determine what is important to remember in conversations. Then, you would set up a tool for it to add parts of conversations to its knowledge base (for RAG) when it determines something is important. I will be making content around this in the future so hopefully that will make this much clearer!
Thanks man! It all depends on the LLM you are using. Your laptop could definitely run smaller models like Microsoft Phi3 Mini! And then you can certainly try the new Llama 3.2 models (I'd test out 3b and 11b) and see what kind of speeds you get!
@@ColeMedin thanks for the swift response brother!! I managed to set it up on my Ubuntu Linux 20.04 laptop with 20gb ram ddr4 and I am actually running some tests since my mac is out of storage lol
Very useful content. I think if the camera is moved a little lower, your face will be closer to the middle of the frame, and this will create a more balanced angle. Thank you very much.
Light mode is objectively better if you're working during the day. The brain needs a bright day to work well, sitting in the dark is quite counter productive.
Thank you so much. Love the videos. I am trying to setup a rag ai chat bot for work. I’d love to run this from a SharePoint site or teams channel. Would some like this be possible to do. Change the google drive to the SharePoint or teams channel? Also be cool to see a next.js chat interface.
Thank you so much! n8n does have a Teams integration so you could easily swap out Google Drive for Teams! There isn't a direct SharePoint integration, but there is OneDrive and I know a lot of companies sync Sharepoint to OneDrive. Including Next.js with this local AI package for a frontend is on my list of ways to extend it! So coming soon!!
Hi dude, thank you very much for share your knowledge. Talking about RAG, right now the Graph RAG looks like the best option for get better answer, I found projects about but every single one needs code, would you do a video building a flow with neo4j? Please
Great video! What about making this runnable in a digital ocean VPS, and having some authentication so an API could be exposed only to those authorized. That's what most companies need, you could sell it in Gumroad even.
Great questions! The whole local AI starter kit takes about 6GB (5.3GB of that is the Ollama container). Then the system requirements all depend on the LLM you want to run locally. Almost any computer would be able to run an LLM like Microsoft's Phi3 Mini! Then for a model like Llama 3.1 8b, you'd want a graphics card with at least 8GB of VRAM. Something like the RTX 3090.
Hey there mate thanks heaps for this tutorial and everything else you do, your doing really well as your well aware of :).. Was wondering if anyone had some advice please, if im running a i7 147000 and a RTX 4070 12G and plenty of RAM how much can I bump up the chunk size??
Thank you very much, it's my pleasure! A good recommendation for a chunk size is somewhere between 1000 and 2000. Depends a lot on your use case so something to play around with. But your setup can definitely handle that kind of chunk size with a model like Llama 3.1 8b!
My pleasure! And I appreciate the suggestion! I honestly didn't think about sending in a pull request to update their README but I will consider that! As long as they are open to it!
Hello and thank you for the video. I don't understand what the point of having n8n locally is, since if I switch off my computer, the workflows no longer work? And how can I avoid losing my data?
You are welcome, and good question! You can host n8n locally but do it within a machine you rent in the cloud so you can have it running 24/7 and not lose any data!
You made an amazing video, I learned a lot and I have subscribed to your channel. May I ask a silly question: Why does conversational memory use a traditional database(postgresql)? Why not do the embedding and save it to a vector database?
Thank you very much, I'm glad you learned a lot from the video! :) Good question! Conversational memory uses a traditional database like PostgreSQL because it’s simple and reliable for storing structured data like chat messages. Vector databases are meant more for unstructured text - using a vector database with embeddings is fantastic for complex searches or matching based on meaning, but for just saving and retrieving straightforward info, a regular database gets the job done more efficiently.
Yes, a lot of local LLMs hallucinate a ton, but that can be helped with good prompting and techniques like mixture of agents, chain of thought, and multi-shot prompting! I agree though that they do have a long way to go. Thankfully progress for open source LLMs is insane!
Such a cool idea. One question, I can't connect to the Google API when I run on localhost, what is your (or anyone's) workaround for that? Tunnel didn't seem to work but it might if I try harder
Thank you!! Good question! The best way would be to use a reverse proxy like nginx to set up a domain on your machine so you aren't just using localhost. This is the kind of thing an LLM like Claude could help you do pretty easily so I would suggest that! haha
I solved it, and it's embarrasing, but just in case anyone is running into the same thing. First I tried NGINX, didn't work. Then I tried setting up a tunnel with NGROK, also didn't work. I kept on seeing people actually using the localhost url to connect with Google so I figured it HAD to work, also without NGINX or NGROK. Still Google just wouldn't let me connect, and give me no other error than "something went wrong". So as a final resort I thought, maybe it's about the browser. It was. Doesn't work in Firefox, but it does in Chrome.
I got a lot of information from this video, but I was hoping to see some example n8n workflows for LOCAL files. I don't need to use Google Drive. Thanks!
I'm glad you got a lot out of it! I totally get wanting to remain entirely local and not even use Google Drive. You can certainly do that in n8n by replacing the Google Drive triggers with local file triggers: docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
Great stuff. I find your videos and explanations very easy to follow. Can you put up a tutorial on how to get Ngrok or Traefik working on this install to get webhooks working properly? I can't seem to get either working in Docker Compose at the moment in the self-hosted environment.
Thank you Keith! I'm glad you find my videos easy to follow, that is one of my main goals! Yes, this is another one of the extensions I want to make with this package - getting a domain set up so this can be hosted in a cloud instance with a reverse proxy! Not set on the specific tool for that yet but Ngrok is one I was considering! Haven't used Traefik before myself actually. What is the issue you are running into with Docker Compose?
@@ColeMedin I just could not get Ngrok working on the free domain they offer on the free tier. I kept getting error messages and was trying to troubleshoot for a few hours before i just gave up. I have not had to use webhooks yet, but i know that would greatly improve the functionality. I also saw that someone else was asking n8n directly about how to get webhooks working in the self-hosted-ai-starter-kit and n8n said it was still something they have not quite worked out. Glad you put this tutorial together. i might uninstall by current self-hosted-ai-starter-kit installation and start off with your build. I did not know about the postres login that needed to be created, so i had followed one of your other tutorials and had a rag ai agent set up using Supabase for the chat history and vector store. I was using Ollama with llama 3.1 and i was having issues with the embeddings, so this newer tutorial is great. Once again, i really appreciate what you are doing here. Your tutorials are giving me some inspiration to build some real world tools.
Thank you for the kind words Keith, they mean a lot! That's strange n8n would say the webhooks wouldn't work in the self-hosted setup. I'm pretty sure I can get those to work 100%, I will be diving into that and will include that in my video I'll put up in the near future where I show how to host this whole setup remotely!
Cole, great video! Thank you for all the work. Quick question, @ 3:33 you said you already set up your credentials. I inserted a random string in the .env file and when attempting to set up docker I get a mismatched credentials error. Is there a file I need to save my keys to? Sorry, kind of new at this.
Thank you, Charles!! That is a great question. You are doing the right thing inserting a random string for the credentials in the .env file. The issue you are encountering is just when using the default credentials that the n8n team provides for this starter kit. But what you can do (and what I did) is just create my own credentials for each service and delete the default ones!
The credentials are created within n8n, not Docker Desktop! So when you click into a node in n8n that needs credentials, you can select the credential drop down and select "Create Credentials". Then n8n will show you what it needs, and there will also be a button to see the documentation to help you no matter what the credentials are for!
Also know that OLLAMA HOST variable has been set! I also customized the .env per your suggestion Cole, then deleted the old Ollama and Qdrant credentials made back on 2-23-2024 by the n8n team. The Ollama credential has only one variable "Base URL" which I left as "localhost:11434" and I was getting an ECONNREFUSED error. Then realized the dockercompose.yml file sets the OLLAMA HOST var to OLLAMA_HOST=ollama:11434. So if you replace localhost with ollama (like this ollama:11434) it works!
I installed both GitHub Desktop and Docker on Win11. When I run the self-hosted-ai-starter-kit.git in the terminal (assuming Windows Terminal), I get an error "The term 'git" is not recognized. 🤷♂
Dang that's really weird... Could you try downloading Git directly? git-scm.com/downloads/win It should come with GitHub desktop but this will for sure solve it for you!
Great question! The answer is yes! Tiny Llama is available in Ollama so you can pull it just like I did with the other models and use it within the n8n workflow instead of Llama 3.1: ollama.com/library/tinyllama
Thank you and great question! If your hardware isn't strong enough for Llama 3.1 8b, I would highly recommend trying Microsoft's phi3 mini model! ollama.com/library/phi3
Depends on the model! For a model like Microsoft's Phi3 Mini you don't need that good of hardware! But for something like Llama 3.1 70b you definitely do.
First of all thanks for the video, great work :) I am on a M3 Macbook and still running Ollama. I do not understand how to connect local Ollama to this?
Thank you very much! When you have Ollama up and running on your laptop, it is exposed through a port on your computer which defaults to port 11434. So to access your Ollama instance locally the address would be localhost:11434. That's the URL that n8n is looking for to make the Ollama connection as well. I hope that helps!
@@ColeMedin Yes did try but I cannot open all credentials. message: "Problem loading credential Credentials could not be decrypted. The likely reason is that a different "encryptionKey" was used to encrypt the data." I am lost ...
Oh I see! I believe this is because you are using the default credentials that come with the local AI starter kit. In the Ollama node in n8n, could you click on the "credentials" dropdown and create new credentials? Then all you have to specify there is the URL to your Ollama instance like I showed in the video. Let me know if that works for you!
@@ColeMedin couldn't update or create credentials for whatever reason I had to delete and reinstall. Now it runs , connection to google drive is ok doc in but get allways: "No data with the current filter could be found" for create & update node. Edit: seems that credentials are ok, testing the connection ok. but it does not find anything neither by list (nothing and I cannot type in a folder name) nor file id nor file url. allways states check credentials but there seems all to be ok.
Hmmm... so the connection is working entirely but it thinks the folder in Google Drive is empty? Are you sure the folder the Create/Update triggers are referencing has content in it?
Thanks for the video, amazing content Not sure on how I can use this in production, will I need a powerful VM with good GPU to run this? I have self-hosted the n8n on ec2 but I am not sure about adding ollama on that instance. Looking forward to the self-host on domain video, it will clear a lot of things
No this isn't heavy on anything except what llm you get locally , phi3 mini might be best if you can't do llama 3b, you don't want something that's to slow, also there is a small qwen2 and Mistral 7b I'm testing
Thank you very much! I'm glad you enjoyed it :) Fantastic question! I really appreciated what @jarad4621 said - it's really only the self-hosted LLM that is heavy. His suggestion of using phi3 mini is great if your EC2 isn't strong enough for a Llama 3.1 model. If you want to self-host Llama 3.1 8b, I would recommend a dedicated GPU on your cloud machine with at least 8GB of VRAM. Then at least 16GB of VRAM for Llama 3.1 70b.
Thank you so much for this. It's exactly what I was looking for. The n8n data flow is just so useful. I've been looking at how to make an AI for a finance customer, and they needed private data injested so retrieve information. Just need to combine with a voice assistant to have a completly internal ai assistant.
@@kilasat its so they can have a conversation with the ai while entering other data. This particular industry they are always on the phone (mostly on hold), so they can interact with it just like they currently do with their current colleagues. In the end most people dont want to be typing all day, if this can be running on the side while doing other things its just a huge time saver.
Great question! It depends a lot on the specific LLM you want to use, but with 16GB of VRAM you should be able to run a good number of models! I would try smaller ones like 7b/8b/11b/13b parameters and then go up from there if you are getting the speeds you want.
Good question! There is a local file trigger in n8n you can use to work with files on your machine instead of in Google Drive: docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
instead of "clear old vectors", i'd prefer a diff and then just delete the -- and then insert the ++, instead of deleting the old file and then uploading a new one. the reason for that is that using the diff method, the update of data would be on the reference of the data, rather than the reference on some file, because what if the filename changes (or if the ref to the vectors is not on the filename, then something else "static")? this would open up the amazing possibility to then add content via "stream" (perhaps via API calls, like via a chatbot that people just chat with, AKA your very own chatgpt). i know, yours is a rag solution, i'm just wondering if / how what i'm talking about is possible...
I love your thoughts here! The reference to the vectors is actually on the file ID not the file name, so it is static. But doing something closer to an actual vector "upsert" is still ideal so I'm with you there! My solution is what I came up with given the limitations of n8n since it doesn't support vector upserts without a custom approach, but I'm sure this could be made even better!
I'm excited. I just bought my first desktop PC and went all out with a 4080 Super and a Ryzen 7 7800X3D. I managed to pick up a mint condition use Samsung Odyssey CRG9 2x Quad HD monitor so gaming will be fun, but I'm equally excited about playing around with some AI stuff. I love the idea of fine tuning models, using RAG, oh and especially stable diffusion. I keep finding out things about my Nvidia GPU that I didn't know, like being able to watch everything in HDR which is fantastic because so little content is in HDR but it looks so good.
For someone doing it for the first time there is still a lot of information missing. I was unable to set it up in 3h of trying. I'll try again with some help of AI. Thanks for the video anyway!
![[LIVE] : ONE ลุมพินี 88 | คู่เอก "ป้อมเพชร vs อัสลามจอน"](http://i.ytimg.com/vi/1ZqTVVbMgbo/mqdefault.jpg)
Follow up video here for deploying this to the cloud!
th-cam.com/video/259KgP3GbdE/w-d-xo.htmlsi=nUt90VMv63iVMQMe
That timing though....Sweeeeet! Thank you!
Cole, you’ve done an outstanding job! Your videos consistently make complex topics clear and easy to follow, which is a rare talent. I appreciate how you break down each concept, making it accessible for viewers at all levels. Your teaching style not only engages but also inspires confidence in learning.
I’m eagerly anticipating your next iteration. It’s always exciting to see how you evolve your content and introduce new ideas. Keep up the fantastic work, and thank you for sharing your knowledge with us!
Thank you very much!! I really appreciate the level of detail in your kind words - it means a lot to me to know that my work is noticed in such a thoughtful way and that I'm hitting my target of breaking down complex topics well!
@@ColeMedin Yes indeed! You explain topics with such a good flow between ideas and concepts that rivals that of other popular tech youtubers such as Networkchuck, Linus Tech Tips and Christian Lempa
Wow that means a lot - thank you!!
I have to agree. Literally just getting started with local AI. Was about to skip past this video and thought, maybe it’s something I can use that I didn’t know existed. BAM! This video is going to be my beginning into what my vision is for my local AI. Really appreciate you made this understandable!
@@Hatch3dLabs the author tells well, and help me understand more clearly. I am getting started too, what tool are you using? I'm using xark-argo/argo on github, it's simple to install on Mac and Windows, what about you? I'd like to keep in touch for more learning.
This is a very good step-by-step tutorial. Following the instructions in this video will get you started with local AI. For people trying M1 and above, the ollama must be installed separately, and the rest are the same.
Thank you Dinesh, I appreciate it a lot!!
Could you clarify why Ollama needs to be installed separately for M1 and above?
@@ColeMedin If you want to use GPU/NPU-accelerated LLM rather than CPU on Apple Silicon (which doesn't have either AMD or Nvidia GPUs), you'll need the actual Ollama desktop app on your host Mac and pull the models from there rather than using the Ollama container in the docker compose file. That's why in the Quickstart they call for starting the Docker compose stack without a profile - it doesn't even start the Ollama container. Docker will still be able to reach Ollama using the docker internal hostname, but you'll get much faster processing using the Apple Silicon GPU and NPU (what pieces are accelerated depend on what the Ollama team have done with the app over time). It took me a few minutes to figure it out, but once I did it works just fine.
@@scottstillwell3150 Ok, but how does the rest have to be configured? I tried, but the whole n8n workflow seems to be broken.
@@scottstillwell3150 Since I could not open the credentials, I tried to setup new ones. They say they could connect, but I am not able to use the ollama node in the demo workflow. It can't fetch any models. This is super confusing.
I'm not a developer, so figuring this out still feels like a big step for me, but you've done an outstanding job here anyway!
Yeah I get it! I'll be continuing to put out content to make it even easier to digest! Thank you though 😃
By far the best tutorial and overview on Local RAG and also dropping gems on the little improvements you've made from the original repo. Workflow is amazing too!! One of my ideas is playing some of older rpg's back in the day on the steam deck but with less time that I have now for other priorities, its nice to just query the walkthrough docs and ask where to go next etc.
Thank you very much man, I appreciate it a ton!
And that's a super fun and unique use case - I like it!
Genius, this is like a “medior ai engineer” tutorial video if someone builds the same thing then tweaks it to make a unique llm app out of it. I think a lot of companies would appreciate their engineers to know all this
Thank you and yeah I agree! Definitely would take some tweaks to make this fit a specific use case, but it's a good start for sure!
man.. just dropping casual double entendres as hole references? that’s an instant sub
@@jordon7999 Haha I appreciate it Jordon! 😂
The removing of the vectors records, when reimporting and updated file fixed a lot of my problems. Thanks for the help. U da man!
Seriously glad I could help, thanks Luis!!
Local is a good start. As a practical application, I think a good project would be to distribute the containers and have a login system for businesses.
Yes I definitely agree! Wish I could cover that here without making the video way to long, but I will be making content on this kind of thing in the future!
You can use also Postgress with pgvector instead of Qdrant
Yes definitely!! I love using pgvector so I'm 100% behind you there.
I focused on Qdrant in this video just to show the entire package, but often times using less services (so using Postgres both for chat memory and RAG) can be the way to go if you find it works well for your use case.
@@ColeMedinthat was my question answered 😅 simplified the stack, if you get it to work with supabase you have all the db you need for different functions in this pipeline
Is pgvector still a couple orders of magnitude slower?
My point exactly
@@ColeMedin and don't forget about Apache AGE for PostgreSQL!
Outstanding work, Cole. Love it. I will implement it today. Looking forward to more of your excellent content. You are not verbose, just straight to the point and deliver value to boot. Thank you!
Thank you very much - your kind words mean a lot to me! 😃
15:20 this is truely the most important part of the logics. It's absolutely necessary to have a function in order to handle contingency regarding file duplicates
Indeed! This part of the workflow definitely took the longest but I wanted to include it because I totally agree it's super important to have.
This is the best example I have seen for the Local AI Agent and Rag
Thank you - that means a lot to me!
Nice video! As long as it's running locally and documents are safe
This looked great, right up to the point you try to access the web interface and you find you can't proceed until you have made an account with n8n. I must have missed where that was shown in the video.
Oh you don't have to make an account with n8n! That is just a local account for your n8n instance!
I'm excited to see you extend this! Working Supabase into this flow for authentication, etc would be incredible. Awesome video bro!
Thank you Alex, I appreciate it a lot!! I'm stoked to extend this, so that won't be happening too far in the future 😎
Thank you for putting lot of time to simplify for learners. Great work!
You bet, thank you!
Something I'd like to see is building in mixture of agent frame works and tool use and an actual chat interface. This is a great start and that is exactly what I'm going to start working on lol
I love it! Mixture of agents is definitely something I'm going to be diving more into in the near future.
Your doing a great job keep making content much appreciated ....i have some amazing ideas but unfortunate dealing a shitty computer phone n lack of the 5 months spent know i realize why i struggled so hard ...so i thank you for clear explanations.....if u can help be so grateful or anyone their as soon can enough or hit streets and hustle im buying anew computer cheers everybody
Excellent explanation🎉🎉 most answer "why" questions
Thank you very much!
This is absolutely awesome! Amazingly useful! Thank you so much bro! Amazing job! 🙌🙌
Thank you so much!! :D
Open Web UI is still the best and cleanest implementation I've seen.
Yes I am actually planning on potentially including Open Web UI in this stack as I expand it!
@@ColeMedin yes please :)
Outstanding Bro I was looking for this solution !!!! since long months
Awesome man, I'm glad I could help!!
Thank you sooo much.. I had to change the document import a bit to work with local network shares for my company but it works .. SO GOOD.
The deleting of documents already in the system before adding more is really important, ** I cant wait for your front end video **
My pleasure!!
I'm curious - what exactly did you have to change for local network shares? I'm glad it's working great for you!
I can't wait to add the frontend into this setup - I appreciate you calling that out :)
@@ColeMedin I am happy to do a short clip on YT showing the local file import change, we are a large mining house and have hundreds of policies "PDF" now you can ask the AI assistant to compare policies from different operation / counties and highlight inconsistencies, or to find the one relevant to you or just to summarize the content to get you up 2 speed.. will reply with a link to the clip :)
@@HermanRasthis would be great! Having the same Point here. Have to Observe a local unc for new and Updated pdfs and markdowns to Feed the rag
That sounds awesome, I look forward to it!
@@HermanRas Any news on your video? We have a local file share with thousands of documents and I wonder how they could be added.
I have been trying to make Llam3.1 work using llama-stack but felt it was too complicated or still unfinished. Docker and Postgres? Oh yeah, this one sounds more like it for me! Subbed.
Thank you for reminding me of this! Keep to this type of content for the people who want to benefit with our own offline AI ventures!
Of course! And I will certainly be sticking to this type of content!!
Excellent video ! Thanks you so much 😊 i’m wondering if multilanguage is ok for the embedding process with the selected model ?
Thank you! Yes, multilanguage should work great with most LLMs!
Great content, thanks!
What minimal computer specs do you recommend so the response feels natural. I want to connect home assistent to it, so it can ve controlled with vocal commands.
You are welcome!!
It all depends on the model you want to use. Something like a 3070 GPU will be good enough for smaller
what a brilliant delete chunks trick
hello do you have discord community or reddit? I have a lot question for my workflow :(:(
Thanks for sharing this - it is a good starting point for my needs.
My pleasure! I'm glad you can take this and run with it!
Thanks for video. A lot of web ui chat tools compatible with ollama nowadays can now do the RAG just right out of the box. Like Open Web UI. Auto triggers part with n8n is a good one, if you need to automatically process a lot of documents.
Thank you and fair point! I am actually looking into Open Web UI and will be doing a video on it in the near future. Awesome platform 🔥
Yes, would love to see setting like Redis caching, supabase auth, next.js project, stripe payment system to have some template for a Saas. God bless you
Thank you for the suggestion! It'll be a larger project to get a video for all of that but I am planning exactly that!
This is looking fantastic, I shall try this later, but I suspect maybe a little too much was skipped for my abilities. We shall see. Thank you either way for the video :)
Thank you and sounds good! Let me know if you have any questions as you implement it!
Great content. any chance you can create a video about how to make actual code changes to the n8n instance and redeploy it via github actions?
Thanks and yes! Maybe not GitHub actions specifically but I do want to integrate a more coding focused part of this ecosystem with CI/CD.
Amazing video Cole. Quick question, I downloaded the json file and try to use it but looks like the Clear Old Vectors node doesn't exist? Also, if i do this with pinecone and open AI how would i change the code? Thanks man
Thank you! That's really weird... it could be a difference in versions of n8n. There's just nothing there where the Clear Old Vectors node should be? For using Pinecone you would just have to swap out everything Qdrant with Pinecone!
Also check this out for clearing old vectors with Pinecone:
community.n8n.io/t/enabling-upserts-for-vectorstore-using-langchain-code-node/47633
Great video, thanks for sharing. Just one question, how do we get to docker-desktop? Was that installed as part of this?
Thank you!!
I have a link to install Docker Desktop in the description! I can put it here too:
www.docker.com/products/docker-desktop/
It's a very quick and easy install!
hey, thanks for the tutorial! very easy to follow. I seem to be getting a stuck with a UNAUTHORISED NULL on ``clear old vectors`` though?
Thank you - I'm glad it was easy to follow!
I haven't encountered this error before... did you make any custom changes to the workflow by chance? Otherwise, it seems like it could be an issue with your Supabase credentials since it's an authorization error. I'd check to make sure the API key you set for the creds there in n8n matches the one in your project
Great work Cole. I plan to set up RAG for my business as I’ve followed RAG developments for about a year. Things have come a long way. I plan to model your work and would like to connect to Supabase since I plan to use for some of my other App work.
Thank you and good luck! Are you planning on hosting Supabase yourself or using the cloud offering? Either works great!
Awesome video, but I think I am a little bit confused. When I follow the instructions in the video, it seems like I am basically installing the original n8n version, as this is what we are pulling. How do we actually get your version with the custom workflow?
Thank you and I understand the confusion! My version of n8n isn't anything special besides the custom workflow. But I have the custom workflow as JSON file you can download from the link in the description, and then you can import it into your n8n by creating a new workflow, clicking on the options in the top right, selecting "Import from file" and then selecting my JSON file once you download it to your machine.
There are two more additions that need to be added for this local dev environment. Open Web UI and ceph nano with S3 enabled
With this you have your own local dev cloud environment then you can build functions and tools in open web UI that call n8n workflows, and store files using S3 protocol
Amazing Cole! at 20:18 when you download files from the google drive, where are they downloaded too? If instead of getting files from google drive, I get it from my local computer (mirrored in the container in data/shared) with the Local File Trigger, do I need the step of downloading files? Thank you so much, I subscribed already!
Thank you very much! The Google Drive files are downloaded inside the container running n8n. So if you are using the local file trigger you don't need the download step!
Can you make the same tutorial with cloud installation?
My issue at the moment is I believe I followed all the steps, but I am unable to connect to the Postgres account. I get an error saying password for user "root" failed. I tried the default 'password' and also the one I set in Visual Code Studio whilst following along with your steps, but neither work.
Dang sorry you're running into that! What URL (host) are you using for Postgres?
@@ColeMedin Thank you for replying, the 'Host' is 'host.docker.internal'.
Thanks Cole! This is pretty amazing!
You bet! Thanks man! 😄
I found your video very helpful. Could you suggest a large language model that I can use for coding on my computer without sacrificing performance?
I'm glad you found it helpful!
For running an LLM locally to help with coding, you could use a really lightweight model like Microsoft's Phi3 Mini! It does depend a lot on your hardware though. If you have a really good GPU (8-16GB+ of VRAM), you could run any of the 8b parameter models to help you with coding without sacrificing performance too much.
IIRC you don't need to expose ports in docker compose if all services are on the same docker network and use their docker hostnames to communicate.
That's what I thought too but I wasn't able to connect to Postgres within n8n until I exposed the Postgres port! Maybe I was just setting up the URL incorrectly at that point, you could totally be right. But that's what ended up working for me!
Have you looked into how to extend the intelligence by using `o1-preview`, `o1-mini`, `claude-3.5-sonnet`, `4o` and so forth as high-level thinkers/orchestrators that manage many small agents to pull intelligence in and process?
You sir are speaking my language! haha
I have been experimenting with this a bit with some initial success. I'll almost certainly make a video on this kind of idea in the future!
@@ColeMedin very exciting. This kind of synergy is where real power lies.
That's right!!
Genio total desde argentina te mando un gran saludo y gracias por tu info
First off, amazing work, I'm puzzled why you chose to use google drive if the aim is to be local? I'll be using your workflow as inspiration for local files, not knowing n8n, can it work with local files/folders? lets hope I can get it working! Thanks for sharing.
Thank you very much!
And that is a really fair point! I have worked with businesses in the past where they wanted a local AI solution, but they were fine storing sensitive information in Google Drive. The biggest concern is more sending their data into a cloud LLM that will be trained with it. In the end, cloud storage is just so convenient that almost no one wants to store all their files locally (on something like a NAS box) even if it's sensitive data. So that is why I consider this setup fully local even though it uses Google Drive.
If you do want to work with local files, n8n can certainly do that! I would check out the local file trigger for n8n:
docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
@@ColeMedinthanks for your work ! Is it possible to add the local trigger on top of the google drive one ?
This is exactly what i need ❤ please, please, please do a short clip on how to do this with local or files stored on a share!
What was the comment by Cole? Why has it been deleted?
@@MustRunTonyo Honestly I'm not sure how my comment got removed! TH-cam says it is there still but when I try to view it I don't see anything.
All I said was Google Drive is generally considered "safe" to store company data. Numerous businesses I work with have sensitive data there even though they want to run LLMs/RAG locally for privacy. Of course it's up to each business to determine if this is really the case for them!
However, I am looking into using the local file triggers in n8n to make this setup entirely locally! I will probably be making content on that in the near future!
Thanks for doing all this work! Huge help!
You are so welcome!
Hi Cole, would love to see your tutorial on how to implement this with Digitalocean.
I will be making a guide in the near future on deploying this all to DigitalOcean! Thank you for mentioning that!
@@ColeMedin There are many existing self-hosted n8n users, and we don’t want to start from scratch. Hopefully, this idea can inspire you to create a tutorial on how to onboard the AI starter kit with an existing DigitalOcean self-hosted setup 🙂
@@ruellago22 Yeah great point! I'm on the same page as you 👍
Very nice video, thank you. Just a question, any ideas about the minimum configuration for a server running LLama3, RAG Stack and more, to have useful performance? On my Mac silicon LLama3 take minutes to answer. I'm talking about RAM size GPU type CPU numbers and so on. Thanks
You bet! And good question! Could you clarify which version of Llama 3 you are trying to run though? As in number of parameters (8b, 70b, etc.).
You could also try a tool like Air LLM to run larger models on not as powerful as hardware:
github.com/lyogavin/airllm
@@ColeMedin Hi Cole, thank you for the answer. I'm running llama3.2:latest with 3.2B parameters.
You bet! Yeah if even 3.2 billion parameters is taking a while I would try something like AirLLM!
For production setup, which one do you use? can you recommend better setup than this: n8n cloud, supabase, supabase vectorstore, openai.
That's what i would do on a vps for production except I feel it's missing something not sure what
Great question! Everything you mentioned here is fantastic. I've more recently liked Claude 3.5 Sonnet more than GPT-4o, but that would be the only thing I'd maybe change. Once o1 supports function calling in the API though, OpenAI will probably be the way to go again!
Jarad I appreciate you replying to so many comments on this video!
There's a few things that could be considered missing for production that maybe you're thinking of. Some sort of logging/tracing tool like LangSmith, or maybe caching using something like Redis? Also some sort of reverse proxy service like Nginx would probably be needed to host this in a VPS.
installed docker desktop and git desktop on W10 (12 cores, 48GB machine, Nvidia GT710) and it seems to keep blue screening on me. WTF? never did that before. Pretty sure it is docker desktop and WSL that is the problem, I don't know why.
Hello! Thank you for sharing this wonderful video! I am interested in changing the workflow, specifically the source of the knowledge base files. Can we upload from a local source like my system C drive? Perhaps we can try to upload to the Docker storage and use this as a source?
I created a folder on n8n container using docker, added it to the yaml file under the n8n service>volumes section , format .\Folder:path, then tested the imported using docker cp command.
I can import in the local file trigger but stumbling around the conversion and import to vector DB. Any recommendations? It looks like the google trigger takes in the data as a binary file(text), and sends it to an Extract trigger. I need to replace the steps where the Download File is completed and the Extract step is completed but using local path and PDF file.
Got it working by removing the google step and using the same Trigger to Extract Document Text🤩
My pleasure George! Great question and I see you figured it out yourself - thanks for following up on that! I'm sure others will benefit from what you've done here too.
Tell me guys, I didn't really understand where I can get the api for qdrant? And also, if I take the api, doesn't that mean it's not a local network?
Great questions! So the API key for Qdrant can actually be anything since it is running locally. That parameter isn't actually used (because it is local). Which also means it is fully local to your second question!
It is just set up this way for n8n to make the node compatible with the hosted version of Qdrant if you wanted to use that.
@@ColeMedinThank you ❤
You bet!
Nice video thanks! Is any way to create a permanet memory based in the conversations with the chat bot? I mean: I just with the bot and it keeps all the learnings from this conversation for ever. Thanks
Thank you, my pleasure! :)
This kind of thing you're describing where an LLM can learn from conversations over a long period of time is definitely possible, but it isn't a quick and easy implementation!
Essentially, you have to tell the LLM (usually in the system prompt) how to determine what is important to remember in conversations. Then, you would set up a tool for it to add parts of conversations to its knowledge base (for RAG) when it determines something is important.
I will be making content around this in the future so hopefully that will make this much clearer!
@@ColeMedin great thanks!! Please if you nake it do simple and easy to follow please. I am new with all this world :)
Of course! And yes, I will make it simple and easy to follow!
Nice video bro!
Is a Macbook Air M1 with 8GB enough for running it?
Thanks man!
It all depends on the LLM you are using. Your laptop could definitely run smaller models like Microsoft Phi3 Mini! And then you can certainly try the new Llama 3.2 models (I'd test out 3b and 11b) and see what kind of speeds you get!
@@ColeMedin thanks for the swift response brother!! I managed to set it up on my Ubuntu Linux 20.04 laptop with 20gb ram ddr4 and I am actually running some tests since my mac is out of storage lol
Of course and that's great!
Very useful content. I think if the camera is moved a little lower, your face will be closer to the middle of the frame, and this will create a more balanced angle. Thank you very much.
Thank you and I appreciate the suggestion a lot!
Great video mate ! Thank you for your effort
Thank you very much - my pleasure :)
Should be illegal to use github in lightmode
I've noticed a trend going on with that. There's not enough argument been made to convince me to go bright.
Haha touché 😂
I generally prefer dark mode in every app I use, honestly not sure why I'm using light mode for GitHub still!
@@ColeMedinheathen!
Light mode is objectively better if you're working during the day. The brain needs a bright day to work well, sitting in the dark is quite counter productive.
@@sCiphre well thats a bunch of nonsense. Im most productive at night when everyone is asleep.
Thank you so much. Love the videos. I am trying to setup a rag ai chat bot for work. I’d love to run this from a SharePoint site or teams channel. Would some like this be possible to do. Change the google drive to the SharePoint or teams channel? Also be cool to see a next.js chat interface.
Thank you so much!
n8n does have a Teams integration so you could easily swap out Google Drive for Teams! There isn't a direct SharePoint integration, but there is OneDrive and I know a lot of companies sync Sharepoint to OneDrive.
Including Next.js with this local AI package for a frontend is on my list of ways to extend it! So coming soon!!
Hi dude, thank you very much for share your knowledge. Talking about RAG, right now the Graph RAG looks like the best option for get better answer, I found projects about but every single one needs code, would you do a video building a flow with neo4j? Please
The idea is use Neo4J like vectorial data base for embebing
Great video! What about making this runnable in a digital ocean VPS, and having some authentication so an API could be exposed only to those authorized. That's what most companies need, you could sell it in Gumroad even.
Thank you and I appreciate the suggestion a lot! I am actually planning on making a video on this in the VERY near future!
How much storage does it require? And what are system requirements?
Great questions!
The whole local AI starter kit takes about 6GB (5.3GB of that is the Ollama container).
Then the system requirements all depend on the LLM you want to run locally. Almost any computer would be able to run an LLM like Microsoft's Phi3 Mini! Then for a model like Llama 3.1 8b, you'd want a graphics card with at least 8GB of VRAM. Something like the RTX 3090.
so unfair that cannot give you more than one thumb up
Haha I appreciate that a lot!! I sure wish you could too 😂
Error in sub-node ‘Embeddings Ollama1‘
fetch failed
Open node
Can anyone help?
Sorry you're hitting this error! What are you trying to embed exactly?
Hey there mate thanks heaps for this tutorial and everything else you do, your doing really well as your well aware of :)..
Was wondering if anyone had some advice please, if im running a i7 147000 and a RTX 4070 12G and plenty of RAM how much can I bump up the chunk size??
Thank you very much, it's my pleasure!
A good recommendation for a chunk size is somewhere between 1000 and 2000. Depends a lot on your use case so something to play around with. But your setup can definitely handle that kind of chunk size with a model like Llama 3.1 8b!
Thanks for the great videos, man. Got Ollama and AnythingLLM set up last night and I'm checking this out now
@@namegoeshere2805 Of course!! Let me know how it goes when you give it a shot!
Thanks for sharing.
Can you not send a pull request or whatever with your improvements on the help doco if you find it lacking in certain areas?
My pleasure!
And I appreciate the suggestion! I honestly didn't think about sending in a pull request to update their README but I will consider that! As long as they are open to it!
Oh man, this could really lower the barrier to entry for Linux's absurd documentation problem.
Haha I hope so!!
really good video. Thanks. Liked and subbed
Hello and thank you for the video.
I don't understand what the point of having n8n locally is, since if I switch off my computer, the workflows no longer work?
And how can I avoid losing my data?
You are welcome, and good question! You can host n8n locally but do it within a machine you rent in the cloud so you can have it running 24/7 and not lose any data!
thx for this awesome details content, once i get my hardware i am going to start \o/
You bet, sounds great!!
You made an amazing video, I learned a lot and I have subscribed to your channel.
May I ask a silly question: Why does conversational memory use a traditional database(postgresql)? Why not do the embedding and save it to a vector database?
Thank you very much, I'm glad you learned a lot from the video! :)
Good question! Conversational memory uses a traditional database like PostgreSQL because it’s simple and reliable for storing structured data like chat messages. Vector databases are meant more for unstructured text - using a vector database with embeddings is fantastic for complex searches or matching based on meaning, but for just saving and retrieving straightforward info, a regular database gets the job done more efficiently.
@@ColeMedin Thank you very much for your reply, I can understand now😊
Awesome, of course!!
thanks for this Cole... would be a pleasure to work with you
My pleasure, and I am always open to consulting/collaborations, so feel free to reach out to me at colemedin@yahoo.com
compete is a strong word, perhaps better to say they are usable, but they still hallucinate like crazy
Yes, a lot of local LLMs hallucinate a ton, but that can be helped with good prompting and techniques like mixture of agents, chain of thought, and multi-shot prompting! I agree though that they do have a long way to go. Thankfully progress for open source LLMs is insane!
Great content my man
Thank you very much!!
Such a cool idea. One question, I can't connect to the Google API when I run on localhost, what is your (or anyone's) workaround for that? Tunnel didn't seem to work but it might if I try harder
Thank you!! Good question!
The best way would be to use a reverse proxy like nginx to set up a domain on your machine so you aren't just using localhost. This is the kind of thing an LLM like Claude could help you do pretty easily so I would suggest that! haha
@@ColeMedin going to give that a go! Thanks, much appreciated
Sounds great, of course!
I solved it, and it's embarrasing, but just in case anyone is running into the same thing. First I tried NGINX, didn't work. Then I tried setting up a tunnel with NGROK, also didn't work. I kept on seeing people actually using the localhost url to connect with Google so I figured it HAD to work, also without NGINX or NGROK. Still Google just wouldn't let me connect, and give me no other error than "something went wrong". So as a final resort I thought, maybe it's about the browser. It was. Doesn't work in Firefox, but it does in Chrome.
I got a lot of information from this video, but I was hoping to see some example n8n workflows for LOCAL files. I don't need to use Google Drive. Thanks!
I'm glad you got a lot out of it!
I totally get wanting to remain entirely local and not even use Google Drive. You can certainly do that in n8n by replacing the Google Drive triggers with local file triggers:
docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
Thank you for this special episode. I subscribed because of this
@@acs2777 My pleasure, thank you very much!! 😊
Me too. Thank you. You help my battle with Boomer tendencys😊
Great stuff. I find your videos and explanations very easy to follow. Can you put up a tutorial on how to get Ngrok or Traefik working on this install to get webhooks working properly? I can't seem to get either working in Docker Compose at the moment in the self-hosted environment.
Thank you Keith! I'm glad you find my videos easy to follow, that is one of my main goals!
Yes, this is another one of the extensions I want to make with this package - getting a domain set up so this can be hosted in a cloud instance with a reverse proxy! Not set on the specific tool for that yet but Ngrok is one I was considering! Haven't used Traefik before myself actually.
What is the issue you are running into with Docker Compose?
@@ColeMedin I just could not get Ngrok working on the free domain they offer on the free tier. I kept getting error messages and was trying to troubleshoot for a few hours before i just gave up. I have not had to use webhooks yet, but i know that would greatly improve the functionality. I also saw that someone else was asking n8n directly about how to get webhooks working in the self-hosted-ai-starter-kit and n8n said it was still something they have not quite worked out. Glad you put this tutorial together. i might uninstall by current self-hosted-ai-starter-kit installation and start off with your build. I did not know about the postres login that needed to be created, so i had followed one of your other tutorials and had a rag ai agent set up using Supabase for the chat history and vector store. I was using Ollama with llama 3.1 and i was having issues with the embeddings, so this newer tutorial is great. Once again, i really appreciate what you are doing here. Your tutorials are giving me some inspiration to build some real world tools.
Thank you for the kind words Keith, they mean a lot!
That's strange n8n would say the webhooks wouldn't work in the self-hosted setup. I'm pretty sure I can get those to work 100%, I will be diving into that and will include that in my video I'll put up in the near future where I show how to host this whole setup remotely!
Cole, great video! Thank you for all the work. Quick question, @ 3:33 you said you already set up your credentials. I inserted a random string in the .env file and when attempting to set up docker I get a mismatched credentials error. Is there a file I need to save my keys to? Sorry, kind of new at this.
Thank you, Charles!!
That is a great question. You are doing the right thing inserting a random string for the credentials in the .env file. The issue you are encountering is just when using the default credentials that the n8n team provides for this starter kit. But what you can do (and what I did) is just create my own credentials for each service and delete the default ones!
@@ColeMedin I am also stuck here, and also a beginner. How do I create my own creds for each service? Is that done in Docker Desktop somehow?
The credentials are created within n8n, not Docker Desktop! So when you click into a node in n8n that needs credentials, you can select the credential drop down and select "Create Credentials". Then n8n will show you what it needs, and there will also be a button to see the documentation to help you no matter what the credentials are for!
Also know that OLLAMA HOST variable has been set! I also customized the .env per your suggestion Cole, then deleted the old Ollama and Qdrant credentials made back on 2-23-2024 by the n8n team. The Ollama credential has only one variable "Base URL" which I left as "localhost:11434" and I was getting an ECONNREFUSED error. Then realized the dockercompose.yml file sets the OLLAMA HOST var to OLLAMA_HOST=ollama:11434. So if you replace localhost with ollama (like this ollama:11434) it works!
I installed both GitHub Desktop and Docker on Win11. When I run the self-hosted-ai-starter-kit.git in the terminal (assuming Windows Terminal), I get an error "The term 'git" is not recognized. 🤷♂
Dang that's really weird... Could you try downloading Git directly?
git-scm.com/downloads/win
It should come with GitHub desktop but this will for sure solve it for you!
same
can this be done but using tiny llama instead?
Great question! The answer is yes!
Tiny Llama is available in Ollama so you can pull it just like I did with the other models and use it within the n8n workflow instead of Llama 3.1:
ollama.com/library/tinyllama
Good video, also what to do when your hardware can’t run these big models?
Thank you and great question! If your hardware isn't strong enough for Llama 3.1 8b, I would highly recommend trying Microsoft's phi3 mini model!
ollama.com/library/phi3
You need to have a really good Specs to work with this AI model comfortable )))
Depends on the model! For a model like Microsoft's Phi3 Mini you don't need that good of hardware! But for something like Llama 3.1 70b you definitely do.
First of all thanks for the video, great work :) I am on a M3 Macbook and still running Ollama. I do not understand how to connect local Ollama to this?
Thank you very much!
When you have Ollama up and running on your laptop, it is exposed through a port on your computer which defaults to port 11434. So to access your Ollama instance locally the address would be localhost:11434. That's the URL that n8n is looking for to make the Ollama connection as well. I hope that helps!
@@ColeMedin Yes did try but I cannot open all credentials. message:
"Problem loading credential Credentials could not be decrypted. The likely reason is that a different "encryptionKey" was used to encrypt the data."
I am lost ...
Oh I see! I believe this is because you are using the default credentials that come with the local AI starter kit. In the Ollama node in n8n, could you click on the "credentials" dropdown and create new credentials? Then all you have to specify there is the URL to your Ollama instance like I showed in the video. Let me know if that works for you!
@@ColeMedin couldn't update or create credentials for whatever reason I had to delete and reinstall. Now it runs , connection to google drive is ok doc in but get allways: "No data with the current filter could be found" for create & update node.
Edit: seems that credentials are ok, testing the connection ok. but it does not find anything neither by list (nothing and I cannot type in a folder name) nor file id nor file url. allways states check credentials but there seems all to be ok.
Hmmm... so the connection is working entirely but it thinks the folder in Google Drive is empty? Are you sure the folder the Create/Update triggers are referencing has content in it?
Would be cool to also have cronjobs fine tuning for like large documents instead of storing them in a vector database
Sounds really interesting! Could you expand a bit more on this idea? :)
Thanks for the video, amazing content
Not sure on how I can use this in production, will I need a powerful VM with good GPU to run this? I have self-hosted the n8n on ec2 but I am not sure about adding ollama on that instance.
Looking forward to the self-host on domain video, it will clear a lot of things
No this isn't heavy on anything except what llm you get locally , phi3 mini might be best if you can't do llama 3b, you don't want something that's to slow, also there is a small qwen2 and Mistral 7b I'm testing
Thank you very much! I'm glad you enjoyed it :)
Fantastic question! I really appreciated what @jarad4621 said - it's really only the self-hosted LLM that is heavy. His suggestion of using phi3 mini is great if your EC2 isn't strong enough for a Llama 3.1 model.
If you want to self-host Llama 3.1 8b, I would recommend a dedicated GPU on your cloud machine with at least 8GB of VRAM. Then at least 16GB of VRAM for Llama 3.1 70b.
Awesome content !!!
Thank you very much!!
Awesome sauce. Thank you!
Thank you! My pleasure! 😃
Thank you so much for this. It's exactly what I was looking for. The n8n data flow is just so useful. I've been looking at how to make an AI for a finance customer, and they needed private data injested so retrieve information. Just need to combine with a voice assistant to have a completly internal ai assistant.
Why they need voice? What's the use case?
My pleasure, I'm glad you found it helpful!!
A voice assistant would be super cool...
@@kilasat its so they can have a conversation with the ai while entering other data. This particular industry they are always on the phone (mostly on hold), so they can interact with it just like they currently do with their current colleagues. In the end most people dont want to be typing all day, if this can be running on the side while doing other things its just a huge time saver.
@@warfy05like the chatgpt advanced audio?
@@kilasat yes, similar to that. but fully integrated into google docs or other file platform / data collection source / database.
Hi, is it possible to run the ai onto an Intel Arc A770 GPU (16GB VRAM) ? Most KI Systems only have integrated support for Nvidia / AMD GPUs. I
Great question! It depends a lot on the specific LLM you want to use, but with 16GB of VRAM you should be able to run a good number of models! I would try smaller ones like 7b/8b/11b/13b parameters and then go up from there if you are getting the speeds you want.
This sounds so interesting
I'm glad it does!
How would this work without google and with local folders instead?
Good question! There is a local file trigger in n8n you can use to work with files on your machine instead of in Google Drive:
docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/
instead of "clear old vectors", i'd prefer a diff and then just delete the -- and then insert the ++, instead of deleting the old file and then uploading a new one. the reason for that is that using the diff method, the update of data would be on the reference of the data, rather than the reference on some file, because what if the filename changes (or if the ref to the vectors is not on the filename, then something else "static")? this would open up the amazing possibility to then add content via "stream" (perhaps via API calls, like via a chatbot that people just chat with, AKA your very own chatgpt). i know, yours is a rag solution, i'm just wondering if / how what i'm talking about is possible...
I love your thoughts here! The reference to the vectors is actually on the file ID not the file name, so it is static. But doing something closer to an actual vector "upsert" is still ideal so I'm with you there! My solution is what I came up with given the limitations of n8n since it doesn't support vector upserts without a custom approach, but I'm sure this could be made even better!
I'm excited. I just bought my first desktop PC and went all out with a 4080 Super and a Ryzen 7 7800X3D. I managed to pick up a mint condition use Samsung Odyssey CRG9 2x Quad HD monitor so gaming will be fun, but I'm equally excited about playing around with some AI stuff. I love the idea of fine tuning models, using RAG, oh and especially stable diffusion. I keep finding out things about my Nvidia GPU that I didn't know, like being able to watch everything in HDR which is fantastic because so little content is in HDR but it looks so good.
That's so cool man, I love it! That's an awesome setup you've got :)
What is the CPU or HW requirements for this? Can any XEON running win 10 run this?
For someone doing it for the first time there is still a lot of information missing.
I was unable to set it up in 3h of trying. I'll try again with some help of AI.
Thanks for the video anyway!
You are welcome! I'm sorry it's taking you a while though! What information would you say is missing? I would love to improve my walkthrough.