Nice! Question for you, do you prefer if I do: * Explanation #1, Code #1 * Explanation #2, Code #2 or * Code #1, Code #2 * Explanation #1, Explanation #2 Unsure which method is better for everyone.
@@DataIndependent The former method would be better, will be able to connect with the theoretical explanation well if the practical is done along with it.
Absolutely amazed. One thing you should have done is => Explain each chain type and immediately show the demo rather than do it at the end. I forgot what refine and map-reduce does as I went towards the demo. Other than that, hats off dude.
I actually thought back and forth on this which would be better. I chose the method in the video (obvi) but I like the method you're mentioning as well.
Wooow! Thank you so much , I was really thinking about this the other day when I saw your previous Video , This is so helpful , I am not a coder I used to use excel to do the refining method(I didn't actually have a name for it ) with GPT-3 API , but your way is more efficient and I can easily implement in my work flow , I so much appreciate it
Cheers, always learning new stuff watching your videos! def gonna try the last two methods, although what is concerning me is the cost of using openAI. sure, it can do the summary of a 300 page document doing the refine method.. but at what cost? would be interested to see what the cost is for the different solutions what are the diferences in cost, which way is more effective to run.
No. Use the langchain.schema.Document object. Example python code for turning list of strings into python code from langchain.schema import Document list_of_strings = YOUR LIST OF STRINGS list_of_documents = [Document(page_content=string) for string in list_of_strings] #After initializing chain and llm chain({"input_documents": list_of_documents, "question": YOUR_PROMPT})
i used japanese text to extract data by chunking but problem is even if i use smaller texts i get token limit error for example you requested 4103 tokens (103 in the messages, 4000 in the completion).
Aren't there programs that automatically cut the files/docs into batches then it does it by itself? I'm trying to search my entire Twitter History and have to split up the data in order to feed it to an LLM.
The question in the end read who was the friend that he got permission from, but the text you searched and showed stated that both him and his friend got permission - based on that excerpt, the answer seemed incorrect - though you probably just searched for the wrong snippet, I guess.
Excellent explanation using langchain methods to split a large document! Like your Langchain videos. 👍 A small question, in your rerank example for Q&A. Where are the loaded document(s) stored? Because it would be not efficient if u need to do the reload the docs every time u asks a question or if you create a chatbot where multiple users are asking questions..
The documents are stored on your local machine when you run langchain like that. Langchain will only send up the pieces of information it needs to your LLM
@@DataIndependent That is what I thought, but just to be sure..😀 A follow up question: Which "InstructGPT" model is used If the question is submitted to OpenAi? Davinci I assume? Can Langchain also use the new turbo 3.5 Chatgpt API chat model which is much cheaper?
Your videos are great helpful. Much appreciated. I've a lament question. We can acheive this by reading the doc using python script and we can get the output right. I know AI gives the result without writing many code. But what is the major difference between these two? Thanks in advance.
Another con from re rank would be the fact that the LLM will have to parse the entire document for every question right? I guess this scales from a paralellization standpoint but it could also cost a lot.
thanks for the clear explanation. So what would be a good method for questions and answers across multiple docs? Seems map re-rank is most performant but restricted to one doc
Depends how many documents you have. If you have a ton, then you'll likely want to do embeddings and store them in a vectorstore so you can get the similar ones back. Check out my "question a book" video for more on how to do that.
How do u reduce if ur also passing ur memory in the agent bcs I am getting that error bcs of the conversation buffer memory that is mentioned in my prompt template
I can see video was posted a year ago. There is one better approach, which I'm not sure was available year ago. You can use RAG + Data Embedings. Each document chunk you can convert to vectors and store in vector store/database. Then also prompt can be converted into vector and matched against vector store/database, which will give you closest match. Then just use GPT model on best match found earlier.
Thanks for the videos man. One question - how can we increase the length of the final summary? I tried a custom prompt with something like "Write a summary of a minimum of 1000 words". But it seems to cut off the returned summary.
@@DataIndependent _Thats_ for sure. Just wait till someone gets joining up YT comments with AI right...."Hey, wait a minute...you can't have that AI idea...That's *my* intellectual property."😆
I tried and it works perfectly, thanks! I only have one problem: responses are in general quite short and general, 5 lines at most. How can I obtain more lenghty answers?
How to implement this on a CSV data which includes the tasks to aggregate ? For example, I've a supply chain data and the task is to retrieve the total amount spent by a customer.
Chunk size depends on your use case. I've done 400-2000 and have had good success. Overlap, though I've used it, I haven't tested it enough to have an opinion
Can this be done with a large CSV or only text? Here my problem: loaded a large amount of CSV data in Pinecone. Now my prompt is generating a response that is tool long. Thanks!
@@DataIndependent which method is more accurate between this video or embedding in vectors? I tried this one for a book with 120K words and took 10 mins to run. Embedding in vectors would make it faster (hopefully down to 3 mins)? I just started using all this so just learning from the pros! :D thanks in advance but also thanks for your content. Top!
Not an out of the box chain. But you could do a custom map reduce chain with custom prompts for your purpose. Check out my latest video on AI generated emails. You’d do the same thing but with different prompts for your use case
Question for anyone here: What is your approach if you're scanning say 100 new web sites and you want openai to summarize the news articles and categorize them. I can see setting up a loop and get openai to create a summary for 1 site at a time. I can also see myself using langchain with prompts and memory to store all the results in one place and then generating the output? Any suggestions on how a 'research script' would scale is appreciated. Thank you.
If you want to generate summaries, I would keep it at one summary per article per openai call So you'll eat a lot of tokens but the process will be straight forward
@@DataIndependent so are you saying I would use openai to provide a 'category' for the news article (one per call as you said) and then just use python to group/summarize those categories?
would you please make a video about your environment or provide your python enviroenment as well? By loading the documents, I am getting this error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 10160: character maps to Edit. I had to make them pdf and use: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
There are tutorials around that explain how to use "pipenv" on other channels python channels. I'm still getting started - different channels use "pipenv" or Docker or Anaconda. I suppose it's good getting comfortable with various environments - I haven't been programming for a while - also learning panda.
Also this the only channel that a playlist of 24 videos breaking down langchain extensively - a lot of other videos are good introductory videos but this "cookbook" approach is helping me get going in programming again.
it would be really really helpful if could make a video on classification. say subject classification , topic classification or chain of classifications
Taking a large document (100s pages of technical specification) and developing a classification language for content based on layout or based on conjunction of 2+ things in the document. Validating AI has a clear understanding of this new classification vocabulary. Then using that vocabulary to to query and allowing AI to use that vocabulary in its response.
Yes, I was curious about these LLMs and quickly realized they are tyring to squeeze a lot of money before free opensource APIs show up. Not paying for no tokens on something that may be free in 6 months. These corporations are truly greedy. May try a month subcrisption of chatpdg and spent only $5.
Apparently we are going to have our own trained LLM's - even on mobile devices. I suppose today's LLM's will become commoditized but way more sophisticated "supermodel LLM's" will keep everyone glued to their screens.
This is amazing work. Has anyone developed any technique that can hold memory with LLMs? i.e. an LLM that can save the context (the complete knowledge in the prompt) in some format to a local disk (memory). The memory is attached to the LLM so it can look up in the memory if required. The memory would work like a knowledge base. Let me know if anyone is working on this or already worked on it. I need to dig into that.
This is nice, but I don't think any of these work for code. For example, I have a long Stored Proc and I want to generate Documentation for it, breaking it up will lose context and get all confused. Code can be self referential, ie, a variable in the first chunk might get referenced in the last chunk, but by this point context is gone.
Aligned w/ you, you'll need to chunk up another way or go graph to keep the connections alive. Check out what www.mendable.ai/ is doing, they may have a chunking/retrieval technique that works for you
I appreciate that your videos always have two phases the explanation part and then you go the extra mile to explain the details, great work!
Nice! Question for you, do you prefer if I do:
* Explanation #1, Code #1
* Explanation #2, Code #2
or
* Code #1, Code #2
* Explanation #1, Explanation #2
Unsure which method is better for everyone.
@@DataIndependent explanations first then code. Option #1 I think is best
@@DataIndependent The former method would be better, will be able to connect with the theoretical explanation well if the practical is done along with it.
This tutorial is really well-structured. I really like how you connect the current video with previous ones. Thanks so much!
Amazing tutorial on this subject. Really appreciate your passion into detailing it in so much depths. Thank you !!!
Appreciate the clear explanation of Token Limit
Correction for the refine method: the calls are *dependent*, not independent. Each call depends on the results of the previous call.
You just saved me hours of trial and error on a task I was about to start working on this week.
Subbed immediately, thank you!
Nice!!! Love it - thanks Butter
Absolutely amazed. One thing you should have done is => Explain each chain type and immediately show the demo rather than do it at the end. I forgot what refine and map-reduce does as I went towards the demo.
Other than that, hats off dude.
I actually thought back and forth on this which would be better. I chose the method in the video (obvi) but I like the method you're mentioning as well.
Why would we use the summarization method over the vector embedding and retrieval method?
I think the answer is simple😂, the vector embeddings and retrieval method doesn't solve the summarization problem.
One word- Simply Great ! Thank You for this.
Nice! Thank you very much
Wow, I am just stunned. This video is so helpful and informative. Thank you so so much!
Nice! That's great. Soon it won't be as big of a deal with gpt4-32k
This is genius content. Thanks for your amazing work.
This is a great series. Thank you for your work!
Glad you enjoy it!
Wooow! Thank you so much , I was really thinking about this the other day when I saw your previous Video , This is so helpful , I am not a coder I used to use excel to do the refining method(I didn't actually have a name for it ) with GPT-3 API , but your way is more efficient and I can easily implement in my work flow , I so much appreciate it
Nice glad to hear it. All the magic is with LangChain and the team putting it together.
@@DataIndependent I was wondering , if you guys have paid courses ?
@@mikemansour1166 Nope, but happy to do an intro call if you need anything. If more is needed we can do a consulting arrangement
I’m glad I found this video!!!
me too!
Top notch explanations, thanks. Would be helpful to have a Typescript tutorial as well tho if you ever have some time :)
Thanks! It'll just be python for now but I'll keep this in mind. Checkout the LangChain discord for more ts help
Really Informative 😊
thank you for the going to the effort
Awesome - glad it helped and worked out
Brilliant explanation! thank you :)
Cheers, always learning new stuff watching your videos! def gonna try the last two methods, although what is concerning me is the cost of using openAI.
sure, it can do the summary of a 300 page document doing the refine method.. but at what cost?
would be interested to see what the cost is for the different solutions what are the diferences in cost, which way is more effective to run.
Thanks ❤you have helped me a lot🎉
Thank you for your work. It's amazing. But may I ask you about the slide? Can you share it for everyone ?
Super useful. Thanks :)
best tutorial for lanchcain ever!!!!
Has anything changed in the last year, ie any other methods to get past a models token limit?
Hi! Great video! I have a question. Is there any way of putting string insted of documents into the model?
No. Use the langchain.schema.Document object. Example python code for turning list of strings into python code
from langchain.schema import Document
list_of_strings = YOUR LIST OF STRINGS
list_of_documents = [Document(page_content=string) for string in list_of_strings]
#After initializing chain and llm
chain({"input_documents": list_of_documents, "question": YOUR_PROMPT})
It's really helpful, thanks
Glad it was helpful!
This is awesome lesson. Thx
Great videos Greg!
It is very helpful. Small observation. For a refine approach , I think the steps are dependent on each other. Not independent.
Can you please provide video on how to compare two financial pdfs with large docs using gpt3.5 turbo?
What type of comparing do you want to do?
@@DataIndependent Lets take an example that page 5 of one pdf is most similar to the page 9 of the another pdf.
Great job, Thx! Just subscribed :D
Nice thank you
This video is great, thanks
Glad you liked it!
Where do you stay up to date with the AI advancements, discord etc?
I have question : how do we do it for csv files ? thanks a lot !
i used japanese text to extract data by chunking but problem is even if i use smaller texts i get token limit error for example you requested 4103 tokens (103 in the messages, 4000 in the completion).
You explained how to fix this issue for text files. How do I handle big csv files?
What if the summaries from the Map Reduce method was over the max tokens for the final summarization call?
I *think* Langchain will map reduce it again. If not then you'll need to do that manually
Aren't there programs that automatically cut the files/docs into batches then it does it by itself?
I'm trying to search my entire Twitter History and have to split up the data in order to feed it to an LLM.
The question in the end read who was the friend that he got permission from, but the text you searched and showed stated that both him and his friend got permission - based on that excerpt, the answer seemed incorrect - though you probably just searched for the wrong snippet, I guess.
Excellent explanation using langchain methods to split a large document! Like your Langchain videos. 👍
A small question, in your rerank example for Q&A. Where are the loaded document(s) stored? Because it would be not efficient if u need to do the reload the docs every time u asks a question or if you create a chatbot where multiple users are asking questions..
The documents are stored on your local machine when you run langchain like that. Langchain will only send up the pieces of information it needs to your LLM
@@DataIndependent That is what I thought, but just to be sure..😀 A follow up question: Which "InstructGPT" model is used If the question is submitted to OpenAi? Davinci I assume? Can Langchain also use the new turbo 3.5 Chatgpt API chat model which is much cheaper?
Your videos are great helpful. Much appreciated. I've a lament question. We can acheive this by reading the doc using python script and we can get the output right. I know AI gives the result without writing many code. But what is the major difference between these two? Thanks in advance.
great video!
Another con from re rank would be the fact that the LLM will have to parse the entire document for every question right? I guess this scales from a paralellization standpoint but it could also cost a lot.
thanks for the clear explanation. So what would be a good method for questions and answers across multiple docs? Seems map re-rank is most performant but restricted to one doc
Depends how many documents you have. If you have a ton, then you'll likely want to do embeddings and store them in a vectorstore so you can get the similar ones back. Check out my "question a book" video for more on how to do that.
How do u reduce if ur also passing ur memory in the agent bcs I am getting that error bcs of the conversation buffer memory that is mentioned in my prompt template
excellent explanation as always, but your video previews always cover important parts of your slides.
I can see video was posted a year ago. There is one better approach, which I'm not sure was available year ago.
You can use RAG + Data Embedings. Each document chunk you can convert to vectors and store in vector store/database. Then also prompt can be converted into vector and matched against vector store/database, which will give you closest match. Then just use GPT model on best match found earlier.
Thanks for the videos man. One question - how can we increase the length of the final summary? I tried a custom prompt with something like "Write a summary of a minimum of 1000 words". But it seems to cut off the returned summary.
There is a parameter called "max_tokens" you'll want to adjust which will lengthen the output. You'll set it when you initialize your LLM
Like your studio philosophy. More 'workarounds'. ;-)
It's a symbiotic relationship!
@@DataIndependent _Thats_ for sure. Just wait till someone gets joining up YT comments with AI right...."Hey, wait a minute...you can't have that AI idea...That's *my* intellectual property."😆
Could you rank how much information is lost for each of the methods?
It differs per use case - hard to say!
I tried and it works perfectly, thanks! I only have one problem: responses are in general quite short and general, 5 lines at most. How can I obtain more lenghty answers?
How to implement this on a CSV data which includes the tasks to aggregate ?
For example, I've a supply chain data and the task is to retrieve the total amount spent by a customer.
Thanks.
nit picking but Rich Draves was the friend with him not who gave him the persmission? probably poor question more than the chain.
Good call and good nit - agreed. The question could be better :)
Can you update the link?
thanks it very very very helpful
Glad it helped!
what chunk size would you recommend? you say in the video that 400 is just for demonstration. what is overlap for?
Chunk size depends on your use case.
I've done 400-2000 and have had good success. Overlap, though I've used it, I haven't tested it enough to have an opinion
Can this be done with a large CSV or only text?
Here my problem: loaded a large amount of CSV data in Pinecone. Now my prompt is generating a response that is tool long. Thanks!
Does pinecone remove the need for chunking? Does it have infinite memory?
no, the doc is still split into chunks and then individual chunks are embedded into vectors which are compared against the prompt
Yep, exactly what zzamme1 said
@@DataIndependent which method is more accurate between this video or embedding in vectors? I tried this one for a book with 120K words and took 10 mins to run. Embedding in vectors would make it faster (hopefully down to 3 mins)? I just started using all this so just learning from the pros! :D thanks in advance but also thanks for your content. Top!
What about langchain memory?
PERFECT
How can I apply these solutions when using agents?
Check out the agent documentation on LangChain.com for a good start
langchain.readthedocs.io/en/latest/modules/agents.html
hi, I want to create MOM using documents which is slightly different than summarization, will these methods work fine?
What's a MOM?
Is there a chain for translation in which i can apply this ?
Not an out of the box chain. But you could do a custom map reduce chain with custom prompts for your purpose.
Check out my latest video on AI generated emails. You’d do the same thing but with different prompts for your use case
Question for anyone here:
What is your approach if you're scanning say 100 new web sites and you want openai to summarize the news articles and categorize them.
I can see setting up a loop and get openai to create a summary for 1 site at a time.
I can also see myself using langchain with prompts and memory to store all the results in one place and then generating the output?
Any suggestions on how a 'research script' would scale is appreciated.
Thank you.
If you want to generate summaries, I would keep it at one summary per article per openai call
So you'll eat a lot of tokens but the process will be straight forward
@@DataIndependent so are you saying I would use openai to provide a 'category' for the news article (one per call as you said) and then just use python to group/summarize those categories?
how to impment this for SQLDatabaseChain ?
You got anything?
yo man, amazing videos. What do you think about building an LLM based off your TH-cam channel so I can get your helpful answers to my questions?
Damn! You are God 🙏
would you please make a video about your environment or provide your python enviroenment as well? By loading the documents, I am getting this error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 10160: character maps to Edit. I had to make them pdf and use: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
There are tutorials around that explain how to use "pipenv" on other channels python channels. I'm still getting started - different channels use "pipenv" or Docker or Anaconda. I suppose it's good getting comfortable with various environments - I haven't been programming for a while - also learning panda.
Also this the only channel that a playlist of 24 videos breaking down langchain extensively - a lot of other videos are good introductory videos but this "cookbook" approach is helping me get going in programming again.
it would be really really helpful if could make a video on classification.
say subject classification , topic classification or chain of classifications
Sounds fun. What's a tactical example you'd like to see?
Taking a large document (100s pages of technical specification) and developing a classification language for content based on layout or based on conjunction of 2+ things in the document. Validating AI has a clear understanding of this new classification vocabulary. Then using that vocabulary to to query and allowing AI to use that vocabulary in its response.
Great
Thank you Caiyu
Yes, I was curious about these LLMs and quickly realized they are tyring to squeeze a lot of money before free opensource APIs show up. Not paying for no tokens on something that may be free in 6 months. These corporations are truly greedy. May try a month subcrisption of chatpdg and spent only $5.
Apparently we are going to have our own trained LLM's - even on mobile devices. I suppose today's LLM's will become commoditized but way more sophisticated "supermodel LLM's" will keep everyone glued to their screens.
This is amazing work.
Has anyone developed any technique that can hold memory with LLMs? i.e. an LLM that can save the context (the complete knowledge in the prompt) in some format to a local disk (memory). The memory is attached to the LLM so it can look up in the memory if required. The memory would work like a knowledge base.
Let me know if anyone is working on this or already worked on it. I need to dig into that.
This is nice, but I don't think any of these work for code. For example, I have a long Stored Proc and I want to generate Documentation for it, breaking it up will lose context and get all confused. Code can be self referential, ie, a variable in the first chunk might get referenced in the last chunk, but by this point context is gone.
Aligned w/ you, you'll need to chunk up another way or go graph to keep the connections alive. Check out what www.mendable.ai/ is doing, they may have a chunking/retrieval technique that works for you
Cool video!
Some coding skills are required I guess?
ya - coding skills will be required.
these are not good approaches for practical work, Create embedding of the large model and then write any prompt