"I want Llama3 to perform 10x with my private knowledge" - Local Agentic RAG w/ llama3
ฝัง
- เผยแพร่เมื่อ 18 พ.ค. 2024
- Advanced RAG 101 - build agentic RAG with llama3
Get free HubSpot report of how AI is redefining startup GTM strategy: clickhubspot.com/4hx
🔗 Links
- Follow me on twitter: / jasonzhou1993
- Join my AI email list: www.ai-jason.com/
- My discord: / discord
- Corrective RAG agent: github.com/langchain-ai/langg...
- LlamaParse: github.com/run-llama/llama_parse
- Firecrawl: www.firecrawl.dev/
- Jerry Liu build production-ready RAG: • Building Production-Re...
⏱️ Timestamps
0:00 Intro
1:33 How to give LLM knowledge
3:05 Problem with simple RAG
5:55 Better Parser
9:01 Chunk size
11:40 Rerank
12:39 Hybrid search
13:10 Agentic RAG - Query translation
14:35 Agentic RAG - metadata filtering
15:52 Agentic RAG - Corrective RAG agent
17:33 Install LLama3
18:00 Code walkthrough
👋🏻 About Me
My name is Jason Zhou, a product designer who shares interesting AI experiments & products. Email me if you need help building AI apps! ask@ai-jason.com
#llama3 #rag #llamaparse #llamaindex #gpt5 #autogen #gpt4 #autogpt #ai #artificialintelligence #tutorial #stepbystep #openai #llm #chatgpt #largelanguagemodels #largelanguagemodel #bestaiagent #chatgpt #agentgpt #agent #babyagi - วิทยาศาสตร์และเทคโนโลยี
This is prob one of the best RAG video I've seen, so many learnings in 20 mins
00:05 AI can revolutionize Knowledge Management
01:46 Llama3 can process precise knowledge with fast inference
05:27 Market strategy for AI startups
07:16 Convert PDF files to markdown format for enhanced accuracy and control
10:47 Finding the optimal chunk size through experiments
12:34 Hybrid search combines Vector search and keyword search for better results
16:12 Building a local agentic RAG with llama3
17:48 Running Llama3 model on local machine and using Visual Studio Code
20:53 Setting up key components for Llama3 performance
22:20 Creating a complex agentic RAG workflow for document retrieval and answering
Yet again an amazing tutorial, thanks so much Jason!
Man, your videos keep getting better every time I look. You have a great mind and your presentation is excellent. Thank you very much, again, for sharing!
he is much better than 99.9% wanna be over hyped ai gurus on youtubu, twitter and linkedin!
Jason, I watch a lot of AI videos but I learn the most from yours. I am actually excited everytime i see you have put another one out. Keep up the great work!
One of the most informative RAG videos I’ve seen. Can’t wait to see more from your channel.
Great content! Thanks for putting in the effort. Will use this.
This is the best RAG video on the internet, awesome job, no fluff, high complexity but easy to understand, nice work
What a great video! Thanks for sharing your knowledge
Really great tutorial, teaches a lot in very short time! Thanks!
dude... great video! Thanks for the knowledge!
You always amaze me by the amount of knowledge I get from your videos
Holy crap! This gave me such amazing background knowledge, love it! Now, what would be extra cool, would be if you could do a real "hands-on" type of workshop to go through it all by setting up the environment completely, including the actual training/RAG implementation of a set of various document types (PDF, excel, website etc..) to extend a locally running llama 3 instance 😊
1) The link for the corrective RAG agent had an extra URL attached at the end which caused it to fail; manually tracing the link got me to the proper location
2) LlamaParse looks like a wonderful tool, since I have a lot of documents with equations, and I really need it to grab equations, if for no other reason than to return them. Unfortunately, LlamaParse requires an API key and seems to send PDFs off for processing, something that others have noted and there is an open issue from 2 weeks ago. As of 3 hours ago, it's still an open issue - clearly most companies don't want to send internal docs out of house. Hopefully this gets resolved soon.
3) Really liked your presentation - easy to follow every step with the provided materials.
Hopefully we will have more better options for local use - shame it's not a local only pipeline yet
yes - I have found this issue too. LlamaParse seems use OpenAI llm to process the pdf and it leads to the privacy concerns.
You're the man 💯👏
Awesome content!
This is extremely helpful! Awesome!
Didn't know about the Agentic RAG techniques, thanks for sharing!! That's definitely a trade off between speed & quality, but good to have the option
Amazing tutorial! Thank you
Amazing info shared -. Thank you!
Solid video Jason
right when i needed it, thank you man!
also, just finished watching and i understood the theory behind it but kinda got lost during the code explanation, i might watching again and again
Great tutorial! Thank you
many many thanks, bro!
Firecrawl boosted our RAG accuracy at our company. fast + provided good markdown format.
Llama parse also super helpful too! Amazing video Jason! This is gold!
Edit: thanks for the likes :)
The search api is just insane on firecrawl
Thanks!
awesome jason thank you
Keep this up. This answered to loads of questions I have had previously, and were not answered in any of the HuggingFace tutorials!
Great video, thanks
wow nice work thanks!
Thanks... Awesome video
I prefer finetuning to RAG first then RAG on top of the finetuned model. Just a simple QLORA is all you need. It really helps a ton.
How would you go about doing that, as in just do it backwards from the video?
Best video💯
Great timing! Why do you always read my mind JASON!!?! lol
I am literally using this technique now in my internship for a project. I went through so many approaches and ended up on my version of this one. Wish you released this video about 2 months ago lol
This is epic! keep up...
Subscribed, dont have an AI company since I'm still a poor student... this video was very informative, the man speaks at two times speed just like my professor. I respect it 😁
Thanks! It's so fascinating how these programs 'think.' Even if I don't install one, concepts like chunking seem to translate to humans as well.
OG Jin Yang from Silicon Valley.. Amazing video 🎉
You are relevant, Subscribing to your channel!
I have to say, it is great :D
Clicked that BELL too! 🔔
Amazing lesson! I learned a lot in just 20 min!
Platform agnostic LLM space overview videos from Jason are the best on AI YT
Amazinnnnggggg🎉🎉🎉🎉
Such a bait and switch. Thumbnail promises fine tuning tutorial. Delivers best improve-your-RAG video on the internet. Excellent work.
thx
The corrective RAG schema explains why AI often tries to bring results from the web even when you tell them not to in prompt. If it doesn't understand the source properly it will look elsewhere. This was insightful, thank you.
great video keep making these please.. only "criticism" / advice if you can call if that is to keep things focused on local / open source solutions as much as possible.. love the use of Ollama here for example.. things that perhaps don't require API keys, subscriptions, external integrations / dependencies help people like me understand more of what's going on in a workflow like this! thanks again!
here come dat boi!!!!!!
The speaker in the transcript discusses the use of AI, particularly large language models, in knowledge management. They highlight that AI can provide value in managing vast amounts of documentation and meeting notes, which can be overwhelming for humans to process. The speaker also mentions the potential disruption of traditional search engines like Google by large language models, which can provide hyper-personalized answers based on their extensive knowledge.
The speaker then introduces the concept of a retrieval augmented generation (RAG) pipeline, which involves extracting information from real data sources, converting them into a vector database, and retrieving relevant information to answer user queries. However, they also note the challenges in building a production-ready RAG application, including dealing with messy real-world data, accurately retrieving relevant information, and handling complex queries that may involve multiple data sources.
The speaker also discusses various tactics to mitigate these challenges, such as better data preprocessing, optimal chunk size, relevance-based retrieval, and hybrid search methods. They also mention the use of agentic RAG, which utilizes agents' dynamic and reasoning abilities to decide the optimal RAG pipeline and improve the answer quality.
The speaker concludes by expressing their curiosity about how AI-native startups operate and embed AI into their business processes. They recommend a research document on the subject for those interested.
In summary, the speaker's points are:
1. AI, particularly large language models, can provide significant value in knowledge management.
2. Traditional search engines could potentially be disrupted by large language models.
3. Retrieval augmented generation (RAG) pipelines can be used to answer user queries based on private knowledge.
4. Building a production-ready RAG application is complex due to challenges like messy real-world data, accurate retrieval of relevant information, and handling complex queries.
5. Various tactics can mitigate these challenges, including better data preprocessing, optimal chunk size, relevance-based retrieval, and hybrid search methods.
6. Agentic RAG can further improve answer quality by utilizing agents' dynamic and reasoning abilities.
7. The speaker is interested in how AI-native startups operate and embed AI into their business processes, and recommends a research document on the subject.
Dead internet thory is getting closer and closer every day
great video jason! quick question, im wondering if a knowledge graph in place of vector database would be better since it mitigates the lost in the middle problem?
Awesome content Jason. A Question. I need to create an AI psychologist and store college data, but this college data is a guide of what to speak, not the content itself.
In that case, what is the best approach, RAG or Fine-tuning?
You got a sub. Finally, an AI channel that actually teaches.
I thought we were gonna fine tune llama3 😢 but the fire crawl implementation looks unreal I’ll have to check that out and add it to my rags.
I don’t know how well it’ll work for RAGs but people have extended the context window like crazy and still can do the needle in haystack to around 130k.
If you have 64gb on the Mac you can try out the 256k context window Llama 3 released by Eric Hartford. Would love to see a side by side with both of them using the same embeddings.
Thanks, Jason, incredible as always! Would you consider sharing the code from the walkthrough? 🙏
Thanks mate, appreciate it! Code is in the description link!
@@AIJasonZ Link is not there
Hey Jason thanks for the video, I think it helps a lot. Can I apply on GPT as well?
Thanks Jason, great video, this explains RAG pretty well. Subscribed!
Great video Jason, I only missed routing as a technique to determine if your question should really go through the RAG. James Briggs has done a few good videos on “semantic routing”.
Is your example notebook available somewhere?
I'm wondering the same thing. Don't see a link to a github repo
Hi brilliant session , do you have a link for the notebook ?
too many api calls here - do it local with no api calls - better and the model has to be able to crawl more doc formats - people will probably do p2p, real time and uncensored models for 'real' open source ai that has no limiting factors like api calls or tokens - this is where things need to go in order to take off, gain relevance and leverage economies of scale, of course cxl and better i/o will help but those are on the way. real open source ai will hit smb mkt in about 4-5 years and there will be more innovation and discovery - exciting times as we all watch the development curve
This answer a lot of questions why my chat with PDF doesn't work, llama parser & firecrawl looks so freaking good!
would be great to get a video on best methods for data extraction from these pdfs
Thank you. Can you say a little about your hardware setup for this work? This information is missing from a lot of online sources.
Very usefull, thank you! Is it posible for the model to retrieve images or graphs from a PDF, or it's only text?
Interesting. Someone needs to create a wrapper which works out the best way to answer questions / queries, based on the input and question/query. I think intelligence of system could then be increased.
I'm a simple man. I see a new AI Jason video, I click.
What about preparing data, for exemple as question / response, the response would be used to generate embedding and the response would be the data retrieved ?
I don’t understand everything but I can feel the gold penetrating my ears
thanks Jason, can i use llama on API and train PDf files in a specify directory train to respond
amazing as always. could you share the notebook please
We been trying to build a middleware that connects with any inventory ERP to be able to have real time data information about inventory data for the chatbot
Good rag video, the thumbnail taking about "training llama3" is hurting my brain tho
Can you share the code in the video?
I was hoping that was the case since it's a "simple" workflow
The code is personal you need to apply for a download link with meta and it will provide the code to copy / paste
@@pollywops9242 apply where? I don’t see it
Great video. How do I add PDF documents and llama_parse to the python notebook?
👍👍
Hi, what are the areas current LLMs excel at?
I am new to this world of AI, but not IT (familiar with infra). It is good that people are trying out things to see what it can do. But my naïve thoughts are that as a language tool, it just looks for patterns of words that appear close together, and knows enough of the formation of language that it produces text that is not only readable, but also relevant. But this surely must have limits, if it does not actually understand?
Would it be serving up answers from a well vetted and written sources such as internal KMS by using this RAG method? Our team was thinking about it use for education / learning - perhaps tied into custom flashcard and evaluation of human provided answers. Alongside the still very useful text summarisation, alternative wording suggestions.
Great thanks. Can we get the repo and link to the colab notebook?
I watch lots of AI videos and 99% of them are just a waste of time. As an AI engineer, this channel is hands down the BEST yet
KEEP UP👏🏼
Goddamn it Jian Yang
I like using gemini for getting quick up to date answers, and chat gpt for stuff that doesn't require up to date stuff
Does this code require a good GPU as a must? I am using my 32 Gp cpu and it is super super slow to generate the answer. If the GPU is a must, any commandation for GPU model? I am seeing Jason in the video generate the answer in seconds and I know he is using a mac. Thanks in advanced!
Would you have plans to create a tutorial that connects what ur teaching here and running thing on something like AnythingLLM that allows document reading to create embeddings.
Is those steps and advices are explained on your website ? It would be amazing if you could share the code 😮
You said that you can fine tune a model to teach it new knowledge. But is it really correct? A decoder based models are fined tuned for aligment.
Curious how this workflow changes with bigger context length. Gradient just released Llama-3 8B with a 1M context length
Can we also finetune the 70B model? Even if its not local
Would there be a way to automate this with Obsidian? I sporadically log everything in Obsidian and it would be amazing to find a way to do this with Obsidian
is there a good parser for powerpoint?
Hey, Jason. This video is 🔥🔥! Congrats, I was wondering if there is a chance to reach out to you? I might have an interesting offer for you.
Great video. Thanks! A lot of very good tips!
4:36 Someone walks into the void and disappears
Damn it Jin yiang 😂
Fucking dope bra
There is a problem with the "Corrective RAG agent" URL in the description.
Hi Jason, Amazing stuff, can u please share the code?
Can u create end to end custome fine tuning LLM LLAMA with API
The Corrective RAG agent: is not working for me. Also do you have a github project for this tutorial? Thanks!
GPT-5 has been released. Shall we explore it together?