Effortless RAG in n8n - Use ALL Your Files (PDFs, Excel, and More)

แชร์
ฝัง
  • เผยแพร่เมื่อ 24 ธ.ค. 2024

ความคิดเห็น • 188

  • @BuddhaMedam
    @BuddhaMedam 2 หลายเดือนก่อน +12

    Keep feeding awesome n8n content buddy. This tool has immense potential and I believe you can be the the right person to show that ;-))

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Thank you very much - that means a lot!! And I certainly will be putting out more n8n content!

    • @PatentFinance
      @PatentFinance 2 หลายเดือนก่อน +1

      @@ColeMedin woot!

  • @user-uv3nv2bc6v
    @user-uv3nv2bc6v 2 หลายเดือนก่อน +5

    Yes please, more n8n and AI => you explain all perfect Cole!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Thank you very much! I'll be doing a lot more with n8n and AI!

  • @jonathanbarber3004
    @jonathanbarber3004 2 หลายเดือนก่อน +2

    Another first class tutorial. Very generous of you to share your knowledge with us all. Im sure you have a big list of turorials that you have planned, but thought id ask whether you were going to explore vision? It would be fantastic to upload images to a RAG system. It would also be amazing to upload videos to then question and retreive sections from... great tool for teaching others with extractions from videos. Again, thank you so much. You tutorials are accessible by so many and you'll be hitting millions of subscribers in no time!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you so much Jonathan, I appreciate the kind words a lot!
      I love your suggestion too and vision is definitely something I need to be covering for my content as well. I am planning on creating content around multi-modal RAG and things like that in the future!

  • @alvaroaraujo7945
    @alvaroaraujo7945 2 หลายเดือนก่อน +3

    This is golden! Btw, I vote for a follow-up video on a front-end recommendation to go with this workflow

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +2

      Thank you man! And I appreciate the suggestion a lot! I have a video on my channel that goes over frontends for n8n agents but I can certainly put out more content on it.
      th-cam.com/video/JyolNYRbAcs/w-d-xo.html

  • @jarad4621
    @jarad4621 2 หลายเดือนก่อน +2

    Thanks cole awesome stuff
    One of my favorite channels, I realized to myself what makes a great channel you watch every video from? On that focuses on one very specific great thing of value, note to self.
    It seems like how the different file type or complex ones like excel or sheets is ingested is really important based on output results you want?
    Would love to see a video on that for sheet or xlsx specifically, to understand the different ways to process these and reasons and how to be thinking about this based on your goals.
    Honestly rag is still really confusing based on it's limitations and when and how to use it or not specifically for files like sheets like if I wanted to later extract and process a larger Excel and multiple rows at once based on a column filter for example perhaps to do an overall sheet analysis how that would work.
    More complex sheets rag workflows would be really interesting as it has a lot of value to many people who use sheets so how to ensure the best results for maybe a bigger complex queries.
    Thanks appreciate you.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much Jarad, that means a lot!
      I love your thoughts here! This is exactly what I was getting at in the video when I said there are a million ways to handle CSVs/Excels especially and that I'll be covering that in a later video.
      What you mentioned about wanting to process a whole file based on a column filter to do overall sheet analysis is definitely something RAG wouldn't be good at since it does smaller lookups, not analysis over many chunks at once. But what you can do is have RAG combined with other tools (usually with Python code generation) to do this kind of thing to make a powerhouse of an agent! That's what I'll be making a video on in the future.

  • @nicholasburdick6086
    @nicholasburdick6086 2 หลายเดือนก่อน +4

    Awesome video! I would love to see the video specifically on csv and excel files, or even hooking into sql databases…. That is something I have been struggling with. Keep up the awesome content!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +3

      Thank you very much and yeah it's pretty clear to me I have to make a follow up for CSVs/Excels for RAG - I certainly will!

    • @BirdManPhil
      @BirdManPhil 2 หลายเดือนก่อน

      @ColeMedin what about using sql to drive the accuracy up when searching your documents, to make the ai follow directions basically, then having pinecone for few shot vectors for the conversational part

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      @@BirdManPhil I like where your head is at with this! That is certainly possible depending on the use case!
      One use case I am working on right now actually involves having the agent write SQL to perform calculations across a large amount of records that wouldn't work well with RAG. But the table structure is also simple enough where I know the agent won't mess up on the SQL generation.

  • @mharding27
    @mharding27 2 หลายเดือนก่อน +6

    Hello Cole, thank you for keeping up on these rag videos. I am trying to do something similar for my work. But we have a lot of files on local shares and SharePoint.
    I am wondering if you know that this would work for local files instead of cloud serivces like Google drive or SharePoint?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Of course! And fantastic question!
      This would indeed work for local files. You can trigger the workflow when files are created or updated locally just like with Google Drive, and then processing them would be exactly the same. Check this n8n documentation out!
      docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/

    • @mharding27
      @mharding27 2 หลายเดือนก่อน

      @@ColeMedin Hello Cole, thank you and I get pdf, excel and text working. But I have not found a way to get doc or docx extracted. Do you know how this can be done

  • @VB_Guron1299
    @VB_Guron1299 2 หลายเดือนก่อน +3

    Hi! I just discovered n8n due to great interest in self hosting. May I ask is there a way to connect n8n to a front-end? Like I don't need to go to n8n to access that chat and do things wit in but instead have a front end while my n8n will be the back end if that makes sense? If possible, is there such a thing that is already build and I just need to debloy?
    Thank you!! Super fun to learn about it and thanks to Cole for the majority on n8n! Looking forward for more!

    • @NishanthA32
      @NishanthA32 2 หลายเดือนก่อน +1

      If you want to just use the chat feature, n8n gives the option to embed the chat widget on any website or use the chat directly on a publicly available url too

    • @NishanthA32
      @NishanthA32 2 หลายเดือนก่อน

      when you open the chat node, it gives you the chat url that you can access on the browser directly

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much and that is a great question!
      As @NishanthA32 pointed out, n8n gives you the ability to embed the chat widget on your website. Not the most customization but for basic use cases it works well.
      Otherwise, I made a video on my channel recently for how to use an n8n agent as an API endpoint for a frontend built with something like Streamlit or Next.js!
      th-cam.com/video/JyolNYRbAcs/w-d-xo.html

  • @Strupido
    @Strupido 21 วันที่ผ่านมา +1

    Great tutorial! Thanks a lot Cole.
    Is there any way for the Assistant to additionally return links to the files it got the information from? Maybe material for another video :)

    • @ColeMedin
      @ColeMedin  18 วันที่ผ่านมา

      Thank you! Yes you can, though that is a more custom approach than using the built in vector DB nodes from n8n. Certainly would make another good video!

  • @jamesturner246
    @jamesturner246 2 หลายเดือนก่อน +2

    Amazing video, if I may ask, on the previous version of this you used Qdrant but on this version you use Supabase, is there a reason ?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you and good question! Both are great, I just used Qdrant in one of my other videos because it was a part of the local AI starter kit so I wanted to leverage it for RAG to use the whole package.
      I'm a big fan of Supabase since I can use it both for the SQL DB for conversation history and for RAG, so it's a double wammy. But again, Qdrant is great for RAG too!

    • @dleitch01
      @dleitch01 หลายเดือนก่อน

      @@ColeMedin Hi Cole, for Supabase? Did you just install it into the self-hosted-ai-starter-kit container in Docker?

  • @SergeyNeskhodovskiy
    @SergeyNeskhodovskiy 2 หลายเดือนก่อน +2

    Thank Cole, always learning something from you! How would you handle this challenge: having a google spreadsheet (not excel), feed it to the agent and ask it questions that depend on the data, like what is the sum of the hours worked etc. I guess since LLMs are awful as calculator, you'd have it convert the spreadsheet to an SQL database and make the AI write and execute the query, but I'd appreciate if you could show the exact steps. Thanks!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +2

      Thanks Sergey, I'm really glad to hear that!
      I appreciate this question a lot because you're definitely hitting on one of the limitations of RAG. Since RAG is meant more for specific lookups, it doesn't handle larger file analysis well like determining averages, sums, maximums, minimums, etc. over a large CSV/Excel file.
      However, you can create a BEAST of an agent by combining RAG with other tools (like Python code generation or SQL generation if you table is in a database like you mentioned) in an agent so it can handle both lookups and calculating values across your files.
      I will be making a video on exactly that in the near future!

  • @martintaborda_uy
    @martintaborda_uy 15 วันที่ผ่านมา

    Excellent tutorial, here's a question for you: Where is the collection name defined when you're inserting a file into Qdrant? I don't understand where that definition is. How could I later search for that collection if I have multiple Qdrant vector stores for different agents?
    Super thanks in advance!

  • @rodillo
    @rodillo หลายเดือนก่อน

    Hi Cole, great video! I’d like to know how you could process Excel files so that each row in an Excel file gets converted into plain text with the column fields, and embed it. The issue is that when Excel files are processed into plain text, sometimes the data ends up split between two embeddings, causing problems when retrieving the data. A video explaining how to handle Excel files would be really interesting. Great job!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Yes I'm actually going to be making more content soon on RAG specifically with Excel files covering this kind of thing! But in n8n you can split the Excel file into separate items for each row and then ingest them one at a time into the vector DB, our batch multiple together if you want to have let's say 10 rows per chunk.
      Thanks!

  • @kitlee888
    @kitlee888 2 หลายเดือนก่อน +1

    Amazing tutorial...really helpful...thanks thanks thanks

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Glad you found it helpful - my pleasure! 😃

  • @McAko
    @McAko 2 หลายเดือนก่อน +1

    which LLM did you use in this example? in previous videos you said that Llama 3.1 8B was not good enough for RAG 🤔

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      Fantastic question! I'm using Claude 3.5 Sonnet in this video (not sticking to local in this case). Llama 3.1 8b doesn't do well with RAG specifically in n8n I should mention, since n8n does RAG as a tool and Llama 3.1 8b doesn't handle tool calling well.

  • @johnsaxxon
    @johnsaxxon 2 หลายเดือนก่อน +2

    Thank you so much! I was banging my head against the wall from some other tutorials and your video helped me get my first bot up and running. RAG, Asana, Atlassian Confluence, oh my! I'd love to see a slack video where you hook n8n up directly to slack without runbear. Or is that a terrible idea? :)

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      I'm so glad I could help!! My pleasure :)
      I am going to be doing more Slack integrations in the future! Runbear is great but there is always a time and place for custom implementations so I'll certainly cover that still.

  • @choijungho1
    @choijungho1 2 หลายเดือนก่อน +2

    Perfect!❤

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you!!

  • @digitalmole8236
    @digitalmole8236 หลายเดือนก่อน +1

    Awesome tut. Can you clarify if the extract text from pdf node is using any kind or ocr or just extracting text? Is there a way to use an ocr tool for pdfs that aren't just text?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      I believe it is actually just extracting text but I'm not totally sure since those implementation details I haven't looked into for that node. You could always use the HTTP node with an OCR API if you want to extract with something more custom!

  • @alvaroaraujo7945
    @alvaroaraujo7945 2 หลายเดือนก่อน +2

    Interestingly, I got accurate/complete results with less hallucination (AND saved prompts) after setting the Supabase Vector Store node to 'Get Many' instead of Retrieve Documents (thus having to switch agent to 'conversational agent')
    Wondering if that's because the task I hand to the Agent is quite complex (had to to split it into more than one agent and token usage was skyrocketingwhile output quality plummeted)

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Wow that is super interesting... thanks for sharing!! That's definitely counterintuitive that making that switch would help, but I'm glad it worked for you!
      Did you do anything custom with the prompt for the 'Get Many' node? Maybe something you did to set it up more custom (since the Retrieve Documents gives less control and 'Get Many' gives more) made the results better for your more complex use case.

    • @alvaroaraujo7945
      @alvaroaraujo7945 2 หลายเดือนก่อน

      @@ColeMedin Just "search for the docs on, or most related to this query: {{ $json.chatInput }}
      I wish I knew why Get Many is performing better lol... Since we're here: do you happen to know how can we 'filter' queries? I wish to add similar data but with different contexts inside the same supabase table

  • @dleitch01
    @dleitch01 13 วันที่ผ่านมา +1

    @colemedin really awesome video. Your channel is super informative. A really great add-on to this might be to show how/if documents in other languages could be translated to English before being inserted into the vector database. As an example a Word document or PDF that is written in French is loaded into Google Drive, the file should be downloaded, the text extracted, the text converted into English, and then the information gets inserted into the vector database. Thoughts?

    • @ColeMedin
      @ColeMedin  8 วันที่ผ่านมา

      Yeah I'd love to add that in the future. Translation is a really great idea and it would make this so much more useful!

  • @PrakashBhat-w7t
    @PrakashBhat-w7t 9 วันที่ผ่านมา +1

    Hi, Thanks for the video, I have one question, how do I use data in my local drive instead of google drive ?

    • @ColeMedin
      @ColeMedin  8 วันที่ผ่านมา

      You are welcome and great question! Take a look at the "Local file trigger" in n8n, that will get you started!

  • @clarkzara15
    @clarkzara15 หลายเดือนก่อน +1

    Thanks for the great video! I'm considering switching to Pinecone for the vector store to see if it improves similarity search accuracy, while keeping Supabase for PostgreSQL to manage chat memory. I’m also encountering out-of-memory errors on my self-hosted n8n whenever I try inserting large PDF files into Supabase’s vector store. Any thoughts or tips would be much appreciated!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Yeah that is certainly worth a try, especially if your knowledgebase is huge. Pinecone is fantastic, though PGVector should work just as well as long as you don't have millions of records.
      Out of memory errors probably means you'll need to upgrade your instance that is hosting n8n! How large are your PDFs?

    • @clarkzara15
      @clarkzara15 หลายเดือนก่อน +1

      @@ColeMedin Thanks for the insight! The PDFs I’m uploading to the vector DB are legal documents and agreements, typically around 3.5MB each. I ran into out-of-memory issues trying to handle everything on my self-hosted n8n instance, so as a workaround, I split the workflows: one instance on Render purely for chat functions, and another on my local PC for inserting PDFs into the vector store. This approach has helped manage the load better, but I'm always open to new ideas if you have any other tips. Thanks again!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      That's definitely a good approach! Nice!

  • @charlleskleber9771
    @charlleskleber9771 2 หลายเดือนก่อน +1

    Thanks for this, you are awesome!
    Do you know any content in the web that follows up on pdf analysis agents? The results are very bad for big documents with tables, like financial statements

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Of course, thank you very much!
      I will be making more content specifically on processing PDFs (and Excel docs) in the future. But off the top of my head what I can think of is there are tools out there like AWS Textract that are meant for working with a lot of complicated data in PDFs. I would check that out to start!

  • @itsdubc8550
    @itsdubc8550 หลายเดือนก่อน

    Great video! I always seem to have reliability issues w/ Google Drive triggers that watch for File Created tho. With the workflow activated and polling every minute, uploading a PDF rarely triggers the File Created event for me. Have you experienced similar issues?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      Thanks! I have actually noticed this myself before, it seems to happen randomly once and a while where it doesn't trigger. But that's pretty rare. If you upload multiple files at once it'll actually only trigger the workflow once but with multiple items, keep that in mind!

  • @nusquama
    @nusquama หลายเดือนก่อน +1

    this is great ! but vector wiht xls files is always difficult ! Do you have better way to vector xls ?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      Thanks and I have a video coming out for this next month!

    • @nusquama
      @nusquama หลายเดือนก่อน +1

      @@ColeMedin this guy on youtube like your video GucciDGIxOption !!!

  • @DanielBowne
    @DanielBowne 2 หลายเดือนก่อน +1

    Would love to see this done with the new pgvector node.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      That's news to me - I will definitely look into this!!

  • @johnfreddyvalencia
    @johnfreddyvalencia 2 หลายเดือนก่อน +2

    Gracias desde Colombia. Great Job

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much!!

  • @misheltal692
    @misheltal692 2 หลายเดือนก่อน +1

    Great job!
    Can you make a video of an CRM agent that can manage clients. e.g update status, search records and get information, assign task based on criteria.
    That would be amazing 😊

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you and I love the suggestion! I've got it added to my list of content ideas :)

  • @hajduadam
    @hajduadam หลายเดือนก่อน

    This is great, I could build a POC with this. Can you make a video, 9if you haven't yet) who to built in the Chat window in a live website? (hosting N8N locally?)

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Thanks! I have a video on using N8N within a website on my channel (using Streamlit)!
      th-cam.com/video/JyolNYRbAcs/w-d-xo.html

  • @jaumerrr
    @jaumerrr 15 วันที่ผ่านมา

    Thanks for this tutorial !!!
    Would there be a way that it can process files if you Bulk add to the Drive folder?
    Also would there be a way so if a file is deleted from the Drive folder its also delted from the vectors database?
    Thanks !!!

  • @LePsyclone
    @LePsyclone 2 หลายเดือนก่อน +1

    Great content. Can you create a tutorial to ingest detailed sales data excel sheets ? That is a common real world need! Sales by period, by source ...

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you and I appreciate the suggestion! I will be doing advanced RAG tutorials in the future where I will cover these kind of documents!

  • @FarisandFarida
    @FarisandFarida 2 หลายเดือนก่อน +2

    Thanks you so much for this ❤. Question, how would you work with this if you are using something like a notion database?
    And, if for instance, you are running a company and have multiple departments with different knowledge bases. Are you adding all in the same table? or can you add them in different tables in supabase? will adding them in the same table mess with the quality of retrieval?

    • @clementgirard2045
      @clementgirard2045 2 หลายเดือนก่อน

      +1 Thanks for this questions and thanks to Cole for his tutorials ❤.
      I would love to be able to use the same RAG Agent, but from a Notion database with the ability to be able to update properties AND content blocks within database pages, all in sync with Supabase. I tried to create this workflow from Cole's previous tutorial, but it's much more complicated with Notion than with Google Docs.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      My pleasure and great questions!
      To your first question, Notion has triggers in n8n for when a page is added or updated. So it should be very similar to Google Drive! I haven't used Notion much before so I'm not sure how exactly you would extract the text from a page to put it into Supabase, but I believe you could use the "Get Many Child Blocks" node to break the created/updated page into blocks and insert each of those into the vector database in a loop.
      For your second question - when you have multiple companies/departments you want to manage knowledge for, one easy solution is to use a separate table for each. That doesn't scale very well though. The better option that I typically go with is to set a piece of metadata on each record for the company/department the record is tied to. Then, when I query the knowledge base (assuming I know the company/department) I can do a metadata filter on that first and then only query on the subset of documents! So I would just take a dive into metadata filtering in general for this kind of use case!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      My pleasure @clementgirand2045!
      I'm really curious! What were the challenges that you ran into when trying to set this up to work with Notion?

    • @FarisandFarida
      @FarisandFarida 2 หลายเดือนก่อน

      @@ColeMedin Thank you for this response. i am not sure i get the answer though lol on the supabase one😆. i will add you response to clause to get more clarification

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      @@FarisandFarida Of course and I'm sorry! Hopefully Claude can help haha but yeah lmk if you want more clarification! I'd ask Claude about metadata filtering and it could give a really good breakdown for you!

  • @jackmermigas9465
    @jackmermigas9465 2 หลายเดือนก่อน +1

    ah this is what I was stuck on last time was the different file types in the data loader thanks so much for clarifying that! About pdf's, let's say we are putting through appointment or meeting reminders, is it better to summarize with an ai and convert to text or can it handle just dumping the whole pdf in there in this way you're showing on video?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      You're so welcome! The point of RAG is to make it possible to dump huge documents and still make it possible to look up specific pieces of information within the documents. So to answer your question I would just put the whole PDF in the vector DB instead of summarizing it first!

    • @jackmermigas9465
      @jackmermigas9465 2 หลายเดือนก่อน +1

      @@ColeMedin brilliant thanks!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      You bet!

  • @agsvk-com
    @agsvk-com 2 หลายเดือนก่อน +1

    Thanks, Cole. Really great videos that you do! 😊 @16:00 you mentioned about ingesting documents with Excel or Google sheets in different ways of doing it. Do you have a video on it yet? God bless you

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much - my pleasure :)
      I do not yet but I am planning to do this video pretty soon! Lot of experimentation I am doing behind the scenes for this actually!

  • @photize
    @photize 2 หลายเดือนก่อน

    Great vid. The pdf scanner does it extract the txt from flattened PDF files as that was always the difficulty. In flattened the text being embedded in an image so some OCR is involved . Thx

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thanks and yeah that makes sense! It's always nice to have a simpler solution than using OCR to extract text from PDFs.

  • @hajduadam
    @hajduadam หลายเดือนก่อน +1

    Can you continue this wideo with finishing the chat node? I activated it but the hosted and embedded ones are responding in JSON instead of plain text in the chat window something like:
    { "response": { "text": "The emergency contact is Jack Daniels." } }
    In the ChatModel node I have the response set up as text. Would be nice to see a tutorial how to get rid off this "bug" and embed it in a website. THANKS

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      That's weird! Which LLM are you using? My guess is it is getting confused with the N8N prompting under the hood.

    • @hajduadam
      @hajduadam หลายเดือนก่อน +1

      ​@@ColeMedin it is. Using your workflow as a base but with Pinecone and cant find out the reason.

  • @Minotaurus007
    @Minotaurus007 หลายเดือนก่อน

    I cannot find the field "file_type" to drag it into the set file id node. I only see mimetype. Is this equivalent or changed N8N the functionality?
    I also cannot open your linked workflow: "not a json file". Helping hint would be appreciated.

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      You can use mimeType, it is equivalent! Make sure when you import the JSON file you do import from file not URL! I checked and it is a valid JSON file I can import!

  • @djagryn
    @djagryn 2 หลายเดือนก่อน +1

    nice job !

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you!!

  • @robgonda3467
    @robgonda3467 2 หลายเดือนก่อน

    @Cole, it seems like the workflow breaks when you upload multiple files to the google drive. It can only pull one at a time.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Interesting... it has worked for me before but I know once and a while it can have issues! Could be some sort of limitation with the Google Drive API as well or a short term rate limit being hit.

  • @gtrusler
    @gtrusler 2 หลายเดือนก่อน

    Do you have any issues with the Google Drive trigger having trouble when you add a lot of documents to the watched folder at once? I find that it sometime submits the same file that was triggered earlier or it misses a lot of them. I see this trigger used in tutorials all over youtube so I'm either doing something wrong or most people only feed it a small number of documents at once. I'm curious if you've notice anything similar.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +2

      I have had some issues in the past!
      I believe what it comes down to is if you upload many documents at once, the workflow will only trigger once but with multiple items going through the workflow at once. So you just have to modify what I have here to handle items in a loop within the workflow.
      Otherwise, if you're actually seeing the workflow entirely miss a trigger, it could potentially be an issue with rate limits for the actual Google Drive API and not because of n8n.

  • @BirdManPhil
    @BirdManPhil 2 หลายเดือนก่อน +1

    does this allow multiple users to use this chat at once? ie a small team or even a large customer base?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Great question - it sure does! You just have to make sure the instance you are hosting n8n on is powerful enough to handle the number of concurrent requests you are looking for. But it doesn't take that much compute power so you should be good regardless.

    • @BirdManPhil
      @BirdManPhil 2 หลายเดือนก่อน

      @ColeMedin would you be available to help me with a chatbot I'm building if I pay you

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      @@BirdManPhil Yes I am! Feel free to reach out to cole@dynamous.ai

    • @BirdManPhil
      @BirdManPhil หลายเดือนก่อน

      @ColeMedin I'm ended up spinning up a 3 node kubernetes cluster with a node balancer and persistent volumes. It's not the best at all but it will scale and that's what matters. I've got treafik and let's encrypt all set up woth postgresql and pgvector for hybrid semantic search, n8n is in que mode with the main node set to 2 replicas for now, a dedicated worker node that can spawn up to 5 replicas, and a webhook node that can spawn 2 with redis handking all the it er njde cimmunication cache and queueing . So all that works perfectly, but I can't figure out now how get other tools to properly ingress and I still need at least flowise and langflow and a purple monitoring services working before I can really dive in. It's hard I'm barely ok with docker, and kubernetes is like docker on steroids from thanos

  • @simonsaysboo
    @simonsaysboo หลายเดือนก่อน

    My node that should be inserting into the Supabase Vector store isn't working... I'm getting no error from the node, the extracted input data is waiting at the input stage, the node settings are per the video, the SB project is setup, 'documents' table exists, credentials are right... but nothing gets inserted. It just says 'nde successfully executed' without doing anything. Any ideas? Thanks :)

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      That's really weird! It must be inserting somewhere if you aren't getting an error. Do you have any other tables in your Supabase? Also I've had this happen a couple times where it seems nothing is getting inserted but then when I refresh my page the records are there.

  • @thomasstavrotheodoros255
    @thomasstavrotheodoros255 หลายเดือนก่อน

    great video! How much does querying this model cost per month?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Thanks! This is using GPT-4o-mini, so the pricing is $0.150 / 1M input tokens and
      $0.600 / 1M output tokens

  • @jaggyjut
    @jaggyjut 2 หลายเดือนก่อน

    Great video. Any chance to create a tutorial with open source framework like langchain. Thank you

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Thank you! n8n is indeed open source itself, but yes I have a lot of content on my channel for LangChain already and will continue to put out more!
      As an example, I created a video recently for how to use LangChain and n8n together to create some really neat agents:
      th-cam.com/video/8hAMASB-RpM/w-d-xo.html

  • @frascenzo
    @frascenzo หลายเดือนก่อน

    Before anything, thanks for the video, it helped me a lot! I would like to ask two questions:
    1. When I delete a file from the folder, the it keeps existing on supabase, and is that supposed to happen?
    2. I'm getting "Error inserting: unsupported Unicode escape sequence 400 Bad Request", when uploading docx files, do you have any insight about it?
    Again, thank you for the video, you explain things very well.

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      You are so welcome! Thank you for the kind words!
      1. Yes, right now there isn't a way in n8n to remove a file from the knowledgebase when it is removed from Google Drive. You would have to set up a custom API integration with Google Drive to watch for file deletions.
      2. You might have to convert docx files to txt files, either before uploading them or within the N8N workflow.

  • @alvaroaraujo7945
    @alvaroaraujo7945 2 หลายเดือนก่อน

    Still working on this flow. may I ask: did you ever encounter a problem where 'document retrieval' output/completion goes only up to only around 300-500 tokens? I've tested it in all rag token use range, from 8k to 15k to 20k+ (by tweaking file limits)... my output seems to always be 'truncated' or limited to 300-400 tk... any clue? thanks already

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Hmmm... well the documents are split into chunks when inserted into the vector DB, so maybe you are just seeing a single chunk and that is why it appears to be truncated? Or maybe I'm not quite understanding the issue you are encountering so if you could clarify that would be sweet!

    • @alvaroaraujo7945
      @alvaroaraujo7945 2 หลายเดือนก่อน

      @@ColeMedin so… It’s not that the retrieval was truncated (actually was pretty extensive/complete). But that my output tokens were consistently too short. Well, for now, breaking 1 big task into 2-3 smaller ones did the trick for me

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Ohh so the actual response from the LLM was cut off? That's super weird! I haven't seen that happen before. I'm glad breaking it up worked well for you though! Maybe you were actually hitting the context limit with whatever LLM you were using.

  • @GrecoFPV
    @GrecoFPV 2 หลายเดือนก่อน

    will it work with images also ? and be able to use openai api to analyze context of it ?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Good questions! I believe RAG with n8n doesn't support images by default without creating a more custom workflow to do something like extract text from an image and then store that. You could certainly do that though! Multi-modal RAG is something I'd have to look into more with n8n.
      Yes, you can use the OpenAI API in n8n!

  • @navindergill3207
    @navindergill3207 2 หลายเดือนก่อน +1

    How to create a multimodal RAG system meaning not just limited to pdfs and excels but even images etc

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Great question! I don't think n8n supports images for RAG by default, but there are multimodal vector DBs like Chroma you could look into using!

  • @mohamedabdulla8097
    @mohamedabdulla8097 หลายเดือนก่อน

    Hey it's awesome. I've a quick question, i have csv file, each row has title, summary, and link. I need to return link Based on the question. Is it possible? I'm New to n8n. Appreciate ur help

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      Thanks! Yes you could set this up but it would take a good amount of customization!

    • @mohamedabdulla8097
      @mohamedabdulla8097 หลายเดือนก่อน

      @ColeMedin can u suggest any doc or make a video about it? It would be really helpful. If you can help, can i have ur email?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      You know I'm pretty at capacity with everything going on now, but what I will say is I am building out a community for N8N soon and this would be a fantastic thing for you to post about!

  • @funnydogfargo1026
    @funnydogfargo1026 2 หลายเดือนก่อน

    Hi Cole, another great video! You just solved my biggest struggle of the last 3 days. But now I have a new problem. How can I upload files from a specific folder in bulk? I have a folder in Drive with 122 PDF files. The current automation only allows me to upload one file at a time, and it's the same file every time. What I'm missing?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much - I'm glad I could help!
      If you want to upload an entire folder for RAG at once, you would want to create a separate folder for that. This workflow is meant to ingest files that are created/updated after the workflow is made active, not to go retroactively through a folder.
      You can create a workflow that uses the Google Drive "List files and folders" to get your list of your 122 PDF files and then loop through each one to extract the text and add into the RAG knowledge base similar to what I do in my workflow.

    • @robgonda3467
      @robgonda3467 2 หลายเดือนก่อน +1

      @@ColeMedin the workflow breaks when uploading multiple files - says it gets confused about which ID to use

    • @robgonda3467
      @robgonda3467 2 หลายเดือนก่อน +1

      BTW, doesn't seem to be working with folders either, only files at the top level

  • @hajduadam
    @hajduadam หลายเดือนก่อน

    And one more :) I have been testing it with csv, and pdf. I have a user list (133 users), it looks like it only pulls in half of it randomly and never the full list. I also have the dates in yyyy-mm-dd format and it always tells me there are events from august however the first even is from October. Do you have some tips how can we make it more accurate?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      Yeah RAG is meant more for lookups and not taking in an entire CSV! But I will be making content this month actually on how to handle CSVs better for AI Agents with other things combined with RAG!

    • @hajduadam
      @hajduadam หลายเดือนก่อน +1

      @@ColeMedin can't wait. Keep up the good work!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Thank you!

    • @hajduadam
      @hajduadam หลายเดือนก่อน

      ​@@ColeMedinif i use Pinecone and export xls or csv to pdf it works brilliantly and I can enquire anything from the list and the counts are correct to. You might wanna explore it in a video, could be an interesting topic.

    • @randylust7566
      @randylust7566 หลายเดือนก่อน

      @@ColeMedin Looking forward to CSV inputs! Well done!

  • @charlleskleber9771
    @charlleskleber9771 2 หลายเดือนก่อน

    Is it possible to call a specific docuemnt name for it to analyze?
    Like analyze document "..." and give me xyz

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Yes you can! But you might want to include the document title in each chunk so it will be retrieved.

  • @T33KS
    @T33KS 2 หลายเดือนก่อน

    I would love to know your opinion about something I've been researching.
    What do you consider is the best tool for the production stage when it comes to n8n automation flows. In other words, what would be the best approach to "orchestrating" (publishing/cloning/managing/monetizing) n8n flows that you create for clients.
    BTW thanks for putting out one of the top channels for straight forward and hands on training in this field.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thanks for the kind words! And great question!
      I've been thinking about this myself actually and I'm planning on just creating something custom to manage all of my n8n workflows for production. Just some sort of frontend that I can build really fast with tools like Bolt.new/Claude dev/v0 and then managing the orchestration of n8n workflows with FastAPI endpoints.

    • @T33KS
      @T33KS 2 หลายเดือนก่อน

      @@ColeMedin that's a good approach. It's weird how no one is building an all-in-one solution. I mean that's a pain point for many AI devs.
      Good luck in that project. And thanks for the great content.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      I'm glad it sounds good to you! I've been putting a lot of thought recently into how to make it work super well. An end to end solution is definitely a big pain point for a lot of devs right now!
      Thank you and my pleasure!

  • @matheuslyncoln6071
    @matheuslyncoln6071 หลายเดือนก่อน

    My superbase is getting the error "Document loader is not initialized" but I think I did everything right, can you help me?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      If you click into the "Document loader" node do you see a more specific error?

    • @matheuslyncoln6071
      @matheuslyncoln6071 หลายเดือนก่อน +1

      @ColeMedin I switched to qdrant and it's working great!

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Okay perfect, nice!

  • @hajduadam
    @hajduadam หลายเดือนก่อน

    It was working great at the bigining and started to test it with modifying the files or adding pdf. and now getting error from the Supabase Vector Store node :Error searching for documents: 57014 canceling statement due to statement timeout null
    Error searching for documents: 57014 canceling statement due to statement timeout null
    In supabase I can see the uploaded file and all the rest of the flow shows the files. Also the chat says has no access to specific files.

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      I actually have not seen this before... it might be a memory issue if you are hosting N8N on a smaller instance.

  • @5capeg0at
    @5capeg0at หลายเดือนก่อน

    Why is installing Supabase so hard though? So many different parts, I’m stuck on launching the supabase-vector container, it just exits with code 78 and I’ve spent a night on it with no luck

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      It's really just because there is the vector DB that needs to be enabled and created and then there is the regular SQL DB for conversation history. In this video I go into more detail on getting it set up!
      th-cam.com/video/PEI_ePNNfJQ/w-d-xo.html

    • @5capeg0at
      @5capeg0at หลายเดือนก่อน

      @ thanks for replying Cole, have you used R2R?

  • @supergpt_tv
    @supergpt_tv 2 หลายเดือนก่อน

    hey cole, having an issue with my supabase vector database, I have created the DB in supabase with the basic script that they provide in the docs, with a different name from "documents", i have adapted everything else so I can call the new vector store. The retrieval works only sometimes, very awkward when I change the model from the vector store tool to a different one, it starts working again! But then, stops working again. Super weird, any thoughts why this is happening?
    Keep up the good content!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you and wow that is super weird!
      I'm curious if restarting the conversation would also help the retrieval just like resetting the vector store. Because one thing I see happen a lot is when the AI thinks it doesn't know the answer to something, it gets stuck in the mindset within the same conversation that it doesn't know, even if you ask a different question. Happens a lot with the weaker models especially.
      Though I guess some clarification would help too! When you say it doesn't work what does that mean exactly?

    • @supergpt_tv
      @supergpt_tv 2 หลายเดือนก่อน

      ​@@ColeMedin thanks for the reply mate!
      so basically the I got an error with the embedding node, it throws me an error "Cannot read properties of undefined (reading 'replace')" and there is no more info on the error at all, just some generic rubbish.
      If i go to to the logs in the "embeddings" part, there is no input for the LLM, so maybe the model is not being able query the vector? but once I change the model, it works again for 3-4 times, but eventually breaks again.
      i have changed the formatting, and using split code to markdown, and using 4o instead of 4omini now and it's been working... but how can I put something in production with an error like this lol, and 4o is not cost effective and I would like to keep using 4omini hehe
      i have opened a bug report in github, but no response still, hopefully it will be soon resolved
      link - github.com/n8n-io/n8n/issues/11173
      cheers!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      @@supergpt_tv That's really weird... I hope you get a reply with the bug report! I haven't seen this before so I'm not totally sure what it could be either.

  • @edyzakaria9522
    @edyzakaria9522 2 หลายเดือนก่อน

    Hi just want to check, what happen if i simply upload 2 (or more) files simultaneously, will the workflow will loop per file?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Good question! There will be a separate execution of the workflow for each file!

  • @tecnopadre
    @tecnopadre 2 หลายเดือนก่อน

    What's the difference between this an using the OpenAI API realtime?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      @@tecnopadre Great question! The main difference is you can take this n8n agent and extend it/use it with other tools/frontends in a way you couldn't do with OpenAI. The extra customization is the main plus!

  • @MustRunTonyo
    @MustRunTonyo 2 หลายเดือนก่อน

    Supabase vs Postgres vs Qdrant: what's best used in these ai apps?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน +1

      Great question!! I assume you meant Pinecone instead of Postgres since we are talking about vector DBs. Supabase with PGVector is the simplest and will perform just as well as Pinecone and Qdrant until you get to millions of records. Once your vector DB is absolutely massive, that's when it makes sense to go with a dedicated vector datastore like Pinecone or Qdrant. I'd choose Pinecone if you want something set up fast and Qdrant if you want to self-host since it is open source!

    • @MustRunTonyo
      @MustRunTonyo 2 หลายเดือนก่อน +1

      @@ColeMedin thanks for the explanation!!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Of course, glad to help!!

  • @otikaID
    @otikaID 6 ชั่วโมงที่ผ่านมา

    why my excel became 3 item records in supabase?

  • @parthchandak7944
    @parthchandak7944 หลายเดือนก่อน

    How do you deal with multiple google drive files uploaded at once?

    • @ColeMedin
      @ColeMedin  28 วันที่ผ่านมา

      Great question! They'll be combined together in a single workflow execution (so multiple input items in a list), so you just have to loop through each one of them and the rest of the workflow can be the same.

  • @yhojraj
    @yhojraj หลายเดือนก่อน

    Msword files getting all Gibberish when extract. Only extension .doc work a little but still have a lot gibberish words. Can you check this with some proper document file please?

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน +1

      This setup just works with Google Doc files right now! For MSWord you might need a custom extract step to parse the format for those files specifically.

  • @theuniversityofthemind6347
    @theuniversityofthemind6347 2 หลายเดือนก่อน

    Hi Cole, love your content! I have an Alienware m18 R2 with an Intel i9-14900HX, NVIDIA RTX 4090 (24GB), 64GB RAM, and 8TB storage, but I struggle to run LLaMA 70B models. If you ever find the time, could you create a video for users like me on optimizing setups (8-bit quantization, mixed precision, etc.) to run large models efficiently? Your help would be greatly appreciated. Many Thanks!

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      Thank you very much! Yes even for an awesome computer like yours you'll have to do 4/8-bit quantization to run 70b models efficiently. Running local LLMs and optimizing those setups (with quantization, fine-tuning, etc.) is something like will be putting a LOT more content out around in the near future! Actually working on building my next PC next week to run some more models!

    • @theuniversityofthemind6347
      @theuniversityofthemind6347 2 หลายเดือนก่อน

      @@ColeMedin Hey Cole! Thanks so much for your kind reply! More content in the near future would be awesome! But would it be safe to say that with the suggested fine tuning it would be possible to get it running efficiently? If you don't mind what type of performance could i expect when doing so?

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      @@theuniversityofthemind6347 Of course! And fine tuning won't make the model run faster, but fine tuning makes any given LLM work better for your use case so I meant more you can use fine tuning to make a smaller model (like 11b or 32b) work just as well for your use case as a 70b model! Not guaranteed of course but with good data you certainly can

  • @docdailey
    @docdailey 10 วันที่ผ่านมา

    Is there a difference between an image PDF and a tax PDF?

    • @ColeMedin
      @ColeMedin  8 วันที่ผ่านมา

      This n8n workflow extracts text from PDFs so it won't get images from PDFs if that is what you are asking

    • @docdailey
      @docdailey 7 วันที่ผ่านมา

      @@ColeMedin yeah.. the ocr part is my point. THe frontier models tend to do pretty well even compared to tesseract.

  • @jjolla6391
    @jjolla6391 2 หลายเดือนก่อน +1

    i'm struggling to understand the value of n8n over simple, well-proven utilities like pdftotext, pdfimages, xlsx2csv, csvtool, tesseract-ocr, goocr, rclone, syncthing, etc ... glued together with simple scripts. I was hoping to find some AI that did all this with minimal building, but n8n still needs you to do the same heavy work .. just with other tools.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      You raise a fair point! You're definitely right that n8n is really just a way to piece a lot of tools together that you could do yourself with libraries and simple scripting. But for a lot of use cases and for a lot of people, n8n just makes that easier. Especially for those who don't know how to do that kind of scripting to piece things together themselves. I will say though that I quite enjoy piecing tools together with Python code myself!

  • @merzougahmed4248
    @merzougahmed4248 12 วันที่ผ่านมา

    why do you need two LLM models ?

    • @ColeMedin
      @ColeMedin  8 วันที่ผ่านมา

      One LLM is the main one responding to the user and the other one handles the text retrieved from RAG!

  • @PrashadDey
    @PrashadDey 2 หลายเดือนก่อน +1

  • @Larimuss
    @Larimuss หลายเดือนก่อน

    The problem with these cloud workflow services is not being able to edit the code.

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Yeah that is fair! Though you can build custom nodes with code so you can extend the functionality in any way you want through that!

  • @digitalmole8236
    @digitalmole8236 หลายเดือนก่อน

    mate - this doens't work. every time you update a google doc it just created duplicate records - it doesn't delete the old ones

    • @ColeMedin
      @ColeMedin  หลายเดือนก่อน

      Sorry that's happening! I'm guessing the file_id addition isn't included in the document loader node in the workflow. I would download the workflow JSON I have and take a look at that! That's the node connected to and right below the Supabase inserter node.

  • @kushis4ever
    @kushis4ever 2 หลายเดือนก่อน

    I have been breaking my head over using n8n with free api from groq etc. None of these workflows have acceptable run times.

    • @webscrapeco
      @webscrapeco 2 หลายเดือนก่อน

      I just got groq premium and it works flawlessly! The groq api that is free is garbage.
      n8n works great with groq don't be fooled by this comment.

    • @ColeMedin
      @ColeMedin  2 หลายเดือนก่อน

      I'm surprised - I'm sorry things are fast enough for you! What kind of use case do you have?