Hi Sam, Enjoying all of your videos. I have 2 humble requests from you. 1. please do a series on LlamaIndex swell 2. Also need a tutorial for end-to-end production based RAG pipeline. you already doing too good for us. respect
Sam, great content! For multiple files of different formats, like pdf, csv, txt, could we load it to one directoryloader and store it into the same persistence directory?
Thanks for sharing. It can extract information from multiple files and show answer. If prompted correctly, can it actually make analysis or comparison among 2 or more files if they are from same domain? (ie, comparing 2-3 multiple papers and their result)
That is great! I searched tons of your videos, I cannot find the local version wit ollama and the embeddings model which ollama provides. Please forgive me my naive attempt to ask for, but I‘m enthusiastic, but a bloody beginner that tries to get into it. Thx so much for helping me out with maybe a video title or a hyperlink. ❤
Great video. I hope the next video (or the one after that) will not only use a local alternative for the embeddings but also a local LLM so no information has to get send to anywhere.
@@microgamawave No I haven't but if you think it works I will certainly check it out. The Pythia based one or LLaMa30B one? My guess is the 30B models will be much more likely to do this kind of task, but very hard for most people to run a 30B model.
@@samwitteveenai A local LLM could be fairly easily swapped out for a different one later, right? If so, don't let perfect be the enemy of good. Getting something that works decent locally, even if the prompt need some changes has some huge advantages over sending data to an external server. Even if you found a perfect one, odds are that a better one would come out before too long anyway. If the LLM and prompts for it as set up as config values, then the code remains static even if the LLM gets replaced. It would also be very cool if the local ChromeDB database could be strategically updated based on changes to specific source data files. In other words, if I have 100 source documents and only one of them changes, the other 99 files shouldn't need to be reprocessed to update the data stores of the embeddings. Or is that not possible with the ChromeDB datastore? I've got a hobby project where this sort of Q&A against local datafiles could be a game changer. Your videos do a great job explaining things and providing sample code that I've actually been able to get running locally. Keep up the great work.
Hi Sam, great videos! I have one question I am not sure if its the proper place to ask: I am playing around with langchain and one use case I try to implement is: I have a lot of documents (pdf), which I'd like to interact with (ask questions etc). The challenge is that these documents are project reports and the make conflicting statements based on the project context they are about. So for project a fact 1 is true, but for project b its not. The results when asking questions in this regard are quite dissatisfactory because contents are merged / mixed up foreseeably. Are you aware of an approach how to deal with that?
I would like to see some open source embeddings rather than OpenAI's🙏 And in general I think we all would like to see more open source, like langchain with open assistant model or whatever ❤
Thanks for the demo. Among Chroma, FAISS, Pinecone and Aws kendra which one eill you choose. Too many optiond😊. Among them which open source LLM will you choose gor extratung tabular data?
If you are getting the error "error ValueError: You are using a deprecated configuration of Chroma" while running the code from here, please upgrade your langchain to 0.0.236. I was on 0.0.234 and it seems there was backward compatibility issue with ChromaDB which was fixed in later versions
take a look at langchain docs "Chains > How-To Guides > Retrieval Question/Answering" it has a section discussing how to do it. also big fan of your music bro, partied hard to it 10-12 years ago
@@RedaReda-dq5fg yeah baby!!! I had to edit the library prompts file. Cause instead of saying I don’t know if the question is not in the context, I tell it to answer as you normally would. This way I can get my custom data in JSON format. If you leave the default prompt template the llm doesn’t know what JSON is etc. But I’ll take a look at the docs again, just could find it!
would be great to see a python version of this. I tried converting it myself and got stuck at the final step. I could get it to create the database, pull the relevant docs, show lens, similarity count and sources, but I couldn't get it to print the answer to the query
Hey Sam, I loved how you explained each and every step while coding! It was very helpful and comprehensible. I have a dumb question though, if I want to return links to the original documents in the llm response so that the user can visit the link and view the whole doc, then how can that be done?
You can store them as meta data when you ingest them and then return the meta data back with the answer etc. I did that in one other video about RAG etc. I have forgotten off hand which one it was
Hi Sam I am wanting to perform a continues or incremental embedding on a database with tables that changes constantly. This will make the vector db to always be up to date. How can I achieve this?
hi sam, let's say the content i'm retrieving from a vector db to pass as context to the llm (i am using gpt3.5) is large and i wanna generate questions based on this context but in my case gpt3.5 has max limit of 4096 tokens for which i had to iterate and change the context every time while calling this function "RetrievalQA.from_chain_type" which has retriever as an argument which takes the entire content it is retrieving as context but i want some of it to be passed as context every time or is there any other way ? i'm still a beginner
Great content man.... Have a question . Information regarding query is available in only 2 documents but we set k=5. Will it return only 2 or it will return 5 with 2 as correct answers and 3 incorrect?
Excellent! Thank you! May I ask you something? Is it possible to add a Google Sheets table to the model? Since the Google Sheets file is intended to be dinamic and editable... Should I load this data in the same code line of the other txt or pdf files?
Great content as always Sam! I'm very interested in an alternative to the OpenAI embeddings, the cost is fairly minimal but it would be nice to have an alternative. Haven't seen anyone demo that.
Hey, I have a slight problem, I am using a qa chain and a tool, an agent and an agent_chain I get the desired Result, but I am unable to print the source documents as it says 'TypeError: string indices must be integer' I can't seem to extract that source_documents part out of the llm's observation
I have a question: here there is no need to construct a prompt and send to llm? seems this prompt is getting created automatically at the backend. Can we also construct a prompt and use it ?
Hi Sam, I want to obtain answers from specific documents based on the input question instead of searching through all the documents in the vector store? is it possible can please guide me if it is possible? I am using 70 PDf files for car manuals, and I want to get the answer from specific docs, I am using ConversationalRetrievalChain
How do I use chroma retrieval if I host llm endpoint on GPU instance. I always get connection error even if I invoke the llm and embeddings. I am using fast api to expose api and using fastchat llm with chroma db
Great video. I've got a question. Would it be useful in some cases to create several indexes depending on the type of information and have retrievers for each with according description. For instance I could have an index for the resumees of all employees in the company and a retriever with a description such as "Use this retriever whenever you need information regarding the skills of employees in the company. I could have another index with quality procedures etc. Would you set it all to a single index and just let the vectorstor find appropriate information or would you have several indexes as I suggest ?
Excellent video!!!... it helped me a lot. I have a question: When I ask questions that are not in the documents, it still answer them. How I can avoid this?
hello Sir great content as always, Sending a lot loves of appreciation from Tanzania, my Qn is so far we can query and retrive these multiple documents, now wouldn't be interesting if we can it follow up questions like do this or that, i know we can use looping for this but am note fully sure how to imprement it, but i think it will be a great addition, Ahsante sana!
Cool to know where you are! What do you want to it to do after it gets the answer? You can always chain the output of this to the input of another chain. Give me a use case of how you would want to do.
Sam, what about an agent that does image embedings & similarity matching? That way figures / images in pdf-s could be retrieved based on a query image?
Hey Sam I’ve watched your Videos with great interest. I’ve been able to setup my own versions of your code so far. What I’m trying to do is to get some kind of an App with GUI. I‘ve tried with panel so far, but wasn’t really happy it’s the results.
@@samwitteveenai Yeah it should, but I was running into an error with one of the dependencies. It throws the error: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects.
awesome stuff! one question from me, what happens if there are a lot of docs with similar info but with small differences, and the user wants to specify which documents to get answers from, i.e. user select doc1 and doc2 instead of everything. Is there a way to achieve that in langchain?
This will all depend on how you index the data and if you set meta data and use that in the search. Adding meta data is something I will cover at some point, but its also in the docs etc.
In addition to using the HF embeddings, perhaps also instead of chatGPT use some open source model like Flan T5, or GPT2. The purpose to showcase a full openAI-independent solution, that can be locally deployed.
Thanks, good to know u can persist chromadb. Can u also persist Faiss? Is it possible in chroma or Faiss to store extra meta data together with the content? Thanks👍
Noob question. Whenever the OpenAI key is used and pointing to the OpenAI LLM, does that mean data from the vector db gets submitted to the OpenAI LLM for processing (ie, becomes public data)? Trying to understand how it all hangs together taking into consideration private vs public data privacy. Thanks.
Yes, when you use OpenAI's API for calculating embeddings, your input data is sent over to them. It does not exactly become public, but "Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law)." Or so they claim.
Current implementation of RetrievalQA is throwing error - ..Can't instantiate abstract class BaseRetriever with abstract method. Online search revelead that we would need to create a CustomRetriever that inherits from BaseRetriever. Do you have an example that explains how that can be done. Any guidance is appreciated!
Hello. Thank you for your videos. I have a question please: I am using Cognitive search index where I have my embeddings. This embeddings have a property called category and I want to filter by this. I am trying something like retriever = self.vector_store.as_retriever(search_kwargs={'filter': {'category': 'testCategory'}}) but does not work, It is generating the response from all the embeddings yet. Could you please help me to filter the embeddings by this field please? Maybe the sintaxis is wrong. I just follow the langchain documentation. Thank you.
Is this the Azure product? I haven't used that at all. Try if it works with another Vectorstore like Chroma. to see if it is the syntax or the product.
I have a large number of PDF documents that I would like to interact with using Langchain. My specific requirement is that I don't want all the documents to be subjected to vector search for each question. Instead, while storing the PDF files in the VectorDB, I would like to assign different sets of PDF files to different User IDs, which I believe could be achieved using Metadata. When it comes to retrieval, my desire is for the vector search to only be performed on a specific set of PDF files associated with a particular User ID. Can Langchain filter documents based on Metadata and then perform vector search on these filtered documents?
Over the weekend I've been playing with langchain. I fed it some legal texts and found that it's not very good at references; so whenever there would be some text like so... 245 (54). X is not allowed to act in a manner described in 112 (43b). And I'd ask it "In what manner is X not allowed to act?" The answer would either yield hallucinations or an enigmatic answer along the lines of "It's unknown". Now, I assume the problem lays basically with similarity - in this case 112 (43b) which explains some manners of behavior is in no way similar to 245 (54), and thus would not be returned as a context. But I was wondering, perhaps you know, if someone very smart out there maybe already created some chain, or other way, to account for that? My research didn't give me anything useful, but I'm admittedly rather new to the world of python and AIs, so it's possible I missed something.
The are lots of tricks for this kind of thing where you can use multiple ways of getting the data. They vary a lot from case to case. It sounds like what you are describing is multi hop where you need facts from multiple sources. I have worked on a project like this recently, so I may make a video if I can work out a way of not giving out proprietary info. There are also different ways to query which I will show more of too.
what kind of search on those images would you like to do? Can you give some examples. I am thinking of showing some things like this but want to work out what is most useful to people.
everyone is working with OpenAI as embeddings and LLM, it would be great to see MPT being used instead since most commercially available LLMs don't have 4000 tokens or more as inputs. but im not sure if you can fit MPT in a google colab free GPU or an average consumer GPU
When generating embeddings you usually split up a large document into chunks anyway and embed each chunk separately, so that the embedding is more focused and closer to the embedding of a future (presumably short) query used to retrieve it. As I understand it, embeddings of longer token sequences are averaging the individual token embeddings, so meaning is lost if you overdo the input length.
@@clray123 Sentence embeddings aren't usually just an average of the token embeddings, they are usually trained specifically for sentence similarity or with some models they are trained for question and answer similarity. I agree that if you sequence is too long that you risk getting the semantic meaning clouded also if they are too short or no stride/overlap you risk getting no useful semantic meaning in the vector.
I got this error: InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4331 tokens (4075 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.
This is a good idea to do a showdown of each of these, it will probably take a while to do. I do plan to show Pinecone and Qdrant at some point. I have no experience with weviate, but I do have a corporate customer who is using it, so will need to play with it at some point. Companies like Pinecone & Weaviate have raised hundreds of Mil so they really should be making a lot of content themselves.
@@samwitteveenai Theres also a Massive Text Embedding Benchmark (MTEB) Leaderboard on huggingface where openai's text-embedding-ada-002 is 6th place, others performing better and I guess you dont even need a paid api
Instead of ingesting pdf/txt could this setup be used for code as well? Would be awesome to see a video on how this could be used against a repo or similar.
Give this man a medal! Helping out beginners like me by going over the code line by line is very much appreciated!!! Thanks a ton!
I love your videos Sam. You are so good at explaining things in an easy to understand way. Thank you and keep up the great work 👍💪
So as always as a beginner in AI I learn a lot on this channel. Many thanks.
Your content and explanation are very clear. Thanks for sharing all this LLM topics.
Hey man love your content, thanks for pfpviding this information that is usually gatekeeped a lot for the newbies to AI. Cheers🎉
Looking forward to seeing how to use own embeddings rather than OpenAI's
thank you so much Sam! Really appreciate the videos you make.
Excellent video on retrieval QA with langchain
your videos are great!, very inspiring, and very clear
Finally, a chroma tutorial! Thank you!
Awesome 👍, keep up your good work on this langchain stuff. Subscribed 😎
Hi Sam, Enjoying all of your videos. I have 2 humble requests from you.
1. please do a series on LlamaIndex swell
2. Also need a tutorial for end-to-end production based RAG pipeline.
you already doing too good for us. respect
Thank you so much, this is really helpful!!🙂
Sam, great content! For multiple files of different formats, like pdf, csv, txt, could we load it to one directoryloader and store it into the same persistence directory?
Great Video, thank you sooooo much, I loved your content. Subscribed.
Wonderful video! Any advice or code you know of that can turn the source URLs into clicks links on the response?
Awesome, thanks!
Thanks for sharing. It can extract information from multiple files and show answer. If prompted correctly, can it actually make analysis or comparison among 2 or more files if they are from same domain? (ie, comparing 2-3 multiple papers and their result)
That is great! I searched tons of your videos, I cannot find the local version wit ollama and the embeddings model which ollama provides. Please forgive me my naive attempt to ask for, but I‘m enthusiastic, but a bloody beginner that tries to get into it. Thx so much for helping me out with maybe a video title or a hyperlink. ❤
Great video in AI. I am thinking to use gptq models (like wizard-vicuna, vicuna) with langchain, but no luck ;-(. Looking forward to your sharing.
Great video. I hope the next video (or the one after that) will not only use a local alternative for the embeddings but also a local LLM so no information has to get send to anywhere.
The challenge is I have yet to find a an open LLM that can do all these kind of tasks that don't need a lot of changes t the prompts.
@@samwitteveenai have u tried the open assistant model??
@@microgamawave No I haven't but if you think it works I will certainly check it out. The Pythia based one or LLaMa30B one? My guess is the 30B models will be much more likely to do this kind of task, but very hard for most people to run a 30B model.
@@samwitteveenai A local LLM could be fairly easily swapped out for a different one later, right? If so, don't let perfect be the enemy of good. Getting something that works decent locally, even if the prompt need some changes has some huge advantages over sending data to an external server. Even if you found a perfect one, odds are that a better one would come out before too long anyway. If the LLM and prompts for it as set up as config values, then the code remains static even if the LLM gets replaced.
It would also be very cool if the local ChromeDB database could be strategically updated based on changes to specific source data files. In other words, if I have 100 source documents and only one of them changes, the other 99 files shouldn't need to be reprocessed to update the data stores of the embeddings. Or is that not possible with the ChromeDB datastore?
I've got a hobby project where this sort of Q&A against local datafiles could be a game changer. Your videos do a great job explaining things and providing sample code that I've actually been able to get running locally. Keep up the great work.
@@samwitteveenai I did try it with the vicuna-13b model and with the assis model base on llama30b and in both cases it work very well.
Hi Sam, great videos! I have one question I am not sure if its the proper place to ask: I am playing around with langchain and one use case I try to implement is: I have a lot of documents (pdf), which I'd like to interact with (ask questions etc). The challenge is that these documents are project reports and the make conflicting statements based on the project context they are about. So for project a fact 1 is true, but for project b its not. The results when asking questions in this regard are quite dissatisfactory because contents are merged / mixed up foreseeably. Are you aware of an approach how to deal with that?
I would like to see some open source embeddings rather than OpenAI's🙏
And in general I think we all would like to see more open source, like langchain with open assistant model or whatever ❤
there are a few which use llama2 and a few guys which have shown a couple other local models. one is called LocalGPT
Thanks for the demo. Among Chroma, FAISS, Pinecone and Aws kendra which one eill you choose. Too many optiond😊. Among them which open source LLM will you choose gor extratung tabular data?
Do we need to store different file formats "loader" as different "db" persistence directories?
If you are getting the error "error ValueError: You are using a deprecated configuration of Chroma" while running the code from here, please upgrade your langchain to 0.0.236. I was on 0.0.234 and it seems there was backward compatibility issue with ChromaDB which was fixed in later versions
Amazing .. 😍😍😍😍😍
How do you edit the prompt template?
take a look at langchain docs "Chains > How-To Guides > Retrieval Question/Answering" it has a section discussing how to do it.
also big fan of your music bro, partied hard to it 10-12 years ago
Its just a string you can overwrite it.
@@RedaReda-dq5fg yeah baby!!! I had to edit the library prompts file. Cause instead of saying I don’t know if the question is not in the context, I tell it to answer as you normally would. This way I can get my custom data in JSON format. If you leave the default prompt template the llm doesn’t know what JSON is etc. But I’ll take a look at the docs again, just could find it!
@@samwitteveenai btw your tutorials are great!!!
can you please do short video following this one on prompt template in this case or add it in your code @@samwitteveenai Thank you.
would be great to see a python version of this. I tried converting it myself and got stuck at the final step. I could get it to create the database, pull the relevant docs, show lens, similarity count and sources, but I couldn't get it to print the answer to the query
Hey Sam, I loved how you explained each and every step while coding! It was very helpful and comprehensible. I have a dumb question though, if I want to return links to the original documents in the llm response so that the user can visit the link and view the whole doc, then how can that be done?
You can store them as meta data when you ingest them and then return the meta data back with the answer etc. I did that in one other video about RAG etc. I have forgotten off hand which one it was
Hey Sam awesome video as always! Is there some way to give a part of the quote from the source documents like a citation or something?
There are a few ways to do this (usually done with meta data) let me try to put this into a future example.
Hi Sam
I am wanting to perform a continues or incremental embedding on a database with tables that changes constantly. This will make the vector db to always be up to date. How can I achieve this?
hi sam, let's say the content i'm retrieving from a vector db to pass as context to the llm (i am using gpt3.5) is large and
i wanna generate questions based on this context but in my case gpt3.5 has max limit of 4096 tokens for which i had to iterate and
change the context every time while calling this function "RetrievalQA.from_chain_type" which has retriever as an argument which takes
the entire content it is retrieving as context but i want some of it to be passed as context every time or is there any other way ? i'm still a beginner
Hi Sam,
How can we use parent and child splitters before creating a chromaDB?
Great content man.... Have a question . Information regarding query is available in only 2 documents but we set k=5. Will it return only 2 or it will return 5 with 2 as correct answers and 3 incorrect?
Hi, you did not mention docs/chunk indexing before qa_chain. Is there a step in your notebook missing that you build an index/indexer in ChromaDb?
How can we do document comparison in langchain similar to llamaindex sub query?
Excellent! Thank you! May I ask you something? Is it possible to add a Google Sheets table to the model? Since the Google Sheets file is intended to be dinamic and editable... Should I load this data in the same code line of the other txt or pdf files?
Great content as always Sam! I'm very interested in an alternative to the OpenAI embeddings, the cost is fairly minimal but it would be nice to have an alternative. Haven't seen anyone demo that.
I am literally going to convert this one to that in the next LangChain video.
@@samwitteveenai GPT4All would be great. Free, local, and no GPU needed
Hey, I have a slight problem,
I am using a qa chain and a tool, an agent and an agent_chain
I get the desired Result, but I am unable to print the source documents as it says 'TypeError: string indices must be integer'
I can't seem to extract that source_documents part out of the llm's observation
I have a question: here there is no need to construct a prompt and send to llm? seems this prompt is getting created automatically at the backend. Can we also construct a prompt and use it ?
Hi Sam,
I want to obtain answers from specific documents based on the input question instead of searching through all the documents in the vector store? is it possible can please guide me if it is possible? I am using 70 PDf files for car manuals, and I want to get the answer from specific docs, I am using ConversationalRetrievalChain
How do I use chroma retrieval if I host llm endpoint on GPU instance. I always get connection error even if I invoke the llm and embeddings. I am using fast api to expose api and using fastchat llm with chroma db
Great video. I've got a question. Would it be useful in some cases to create several indexes depending on the type of information and have retrievers for each with according description.
For instance I could have an index for the resumees of all employees in the company and a retriever with a description such as "Use this retriever whenever you need information regarding the skills of employees in the company. I could have another index with quality procedures etc.
Would you set it all to a single index and just let the vectorstor find appropriate information or would you have several indexes as I suggest ?
Yes this is done a lot with different indexes. You can actually use Llama Index for this which I am planning a few videos on soon.
Excellent video!!!... it helped me a lot. I have a question: When I ask questions that are not in the documents, it still answer them. How I can avoid this?
😂😂
hello Sir great content as always, Sending a lot loves of appreciation from Tanzania, my Qn is so far we can query and retrive these multiple documents, now wouldn't be interesting if we can it follow up questions like do this or that, i know we can use looping for this but am note fully sure how to imprement it, but i think it will be a great addition, Ahsante sana!
Cool to know where you are! What do you want to it to do after it gets the answer? You can always chain the output of this to the input of another chain. Give me a use case of how you would want to do.
The use case would be similar to how we ask chatgpt to refine an answer... Would be helpful to clarify stuff
Exactly especially when your PDF are about data or relatable contents
It is possible to host chroma db vectors in an online vector db?
Thanks Sam for the tutorial. Do I need to have paid version of OpenAI API key for this?
For this one yes but there is one after this that is no OpenAI which doesn't need OpenAI keys
Sam, what about an agent that does image embedings & similarity matching? That way figures / images in pdf-s could be retrieved based on a query image?
yeah you could do this with BLIP2. I am not sure if they have it in LangChain but writing a tool for it shouldn't be that hard.
Hi it's great video ! Where can I find the distinct indexing for each document?
take a look in the db folder
Hey Sam I’ve watched your Videos with great interest. I’ve been able to setup my own versions of your code so far. What I’m trying to do is to get some kind of an App with GUI.
I‘ve tried with panel so far, but wasn’t really happy it’s the results.
what is 'panel'?
Have you tried gradio or streamlit?
I was literally looking at doing this yesterday! I am having trouble getting chromadb installed though.
it should pip install pretty straight forward.
@@samwitteveenai Yeah it should, but I was running into an error with one of the dependencies. It throws the error: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects.
awesome stuff! one question from me, what happens if there are a lot of docs with similar info but with small differences, and the user wants to specify which documents to get answers from, i.e. user select doc1 and doc2 instead of everything. Is there a way to achieve that in langchain?
This will all depend on how you index the data and if you set meta data and use that in the search. Adding meta data is something I will cover at some point, but its also in the docs etc.
@@samwitteveenai thanks that really helped
Great. Great. Great
Really waiting for the video using pinecone and using your own embedding to do same stuff
The video just dropped has own embeddings, Pincone will probably be next week.
In addition to using the HF embeddings, perhaps also instead of chatGPT use some open source model like Flan T5, or GPT2. The purpose to showcase a full openAI-independent solution, that can be locally deployed.
Your videos are great by the way, by far my most viewed AI models related content creator.
I just dropped the HF embeddings and will work on the source model example next.
Great video. Could you make a video on a hybrid chatbot?
Where one can upload multiple datasources and the gpt would understand which DB to retrieve the information from?
@@ronakdinesh working on something like this at the moment
Thanks, good to know u can persist chromadb. Can u also persist Faiss? Is it possible in chroma or Faiss to store extra meta data together with the content? Thanks👍
FAISS people tend to persist just as a pickle file. ChromaDB can do some Metadata, Pinecone and others tend to do more metadata options.
How to use falcon instead of openai here ?
where is the video where you did use faiss please
Is it possible to create embeddings with models like alpaca ,vicuna and koallla etc via langchain
and also use them to query those embedding ?
Its possible but not a good way to make embeddings for similarity. The model I show in the recent video is much much better.
how do i setup it on my local PC
Noob question. Whenever the OpenAI key is used and pointing to the OpenAI LLM, does that mean data from the vector db gets submitted to the OpenAI LLM for processing (ie, becomes public data)? Trying to understand how it all hangs together taking into consideration private vs public data privacy. Thanks.
Yes, when you use OpenAI's API for calculating embeddings, your input data is sent over to them. It does not exactly become public, but "Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law)." Or so they claim.
@@clray123 Cool. Thanks for the clarification.
Current implementation of RetrievalQA is throwing error - ..Can't instantiate abstract class BaseRetriever with abstract method. Online search revelead that we would need to create a CustomRetriever that inherits from BaseRetriever. Do you have an example that explains how that can be done. Any guidance is appreciated!
This vid is very old now. I am making a new long form video for LangChain for all this stuff. Will try to get it out soon.
Sam after this is setup, is there a simple method to add more files into the same Chroma index?
Yes Chroma has add_documents and add_texts which would allow you to do this.
Hey, just want to ask instead of using open AI as llm model. How we can use a Huggingface pipeline in this ?
Take a look I have some other No OpenAI videos for this.
Hello. Thank you for your videos. I have a question please: I am using Cognitive search index where I have my embeddings. This embeddings have a property called category and I want to filter by this. I am trying something like retriever = self.vector_store.as_retriever(search_kwargs={'filter': {'category': 'testCategory'}}) but does not work, It is generating the response from all the embeddings yet. Could you please help me to filter the embeddings by this field please? Maybe the sintaxis is wrong. I just follow the langchain documentation. Thank you.
Is this the Azure product? I haven't used that at all. Try if it works with another Vectorstore like Chroma. to see if it is the syntax or the product.
How do you point the SOURCE part back to the actual PDF file?
you can just take the returned sources and use that info in a UI link etc.
I have a large number of PDF documents that I would like to interact with using Langchain. My specific requirement is that I don't want all the documents to be subjected to vector search for each question. Instead, while storing the PDF files in the VectorDB, I would like to assign different sets of PDF files to different User IDs, which I believe could be achieved using Metadata.
When it comes to retrieval, my desire is for the vector search to only be performed on a specific set of PDF files associated with a particular User ID. Can Langchain filter documents based on Metadata and then perform vector search on these filtered documents?
You have to have it implemented in your application layer, it can be done.
Do we have to pay for the Embeddings again in case we re-use the model?
if you don't save them the will need to be calculated again. If they are OpenAI then yes will need to pay again.
@@samwitteveenai Thank you, Awesome Video
what version of chroma db you were using back then??
Not sure I think 1 or 2, that was about a year ago.
Over the weekend I've been playing with langchain. I fed it some legal texts and found that it's not very good at references; so whenever there would be some text like so...
245 (54). X is not allowed to act in a manner described in 112 (43b).
And I'd ask it "In what manner is X not allowed to act?" The answer would either yield hallucinations or an enigmatic answer along the lines of "It's unknown". Now, I assume the problem lays basically with similarity - in this case 112 (43b) which explains some manners of behavior is in no way similar to 245 (54), and thus would not be returned as a context. But I was wondering, perhaps you know, if someone very smart out there maybe already created some chain, or other way, to account for that? My research didn't give me anything useful, but I'm admittedly rather new to the world of python and AIs, so it's possible I missed something.
The are lots of tricks for this kind of thing where you can use multiple ways of getting the data. They vary a lot from case to case. It sounds like what you are describing is multi hop where you need facts from multiple sources. I have worked on a project like this recently, so I may make a video if I can work out a way of not giving out proprietary info. There are also different ways to query which I will show more of too.
Will this also work with a library of markdown text files?
yes
I am having a wierd problem when creating vectorstore in colab, it uses too much ram and session crashes. any suggestions?
There was an error in my code which I figured out when iterating over multiple docs.
thanks , what if we have pdf's with images . how can we search on images ?
what kind of search on those images would you like to do? Can you give some examples. I am thinking of showing some things like this but want to work out what is most useful to people.
What about csv files ?
everyone is working with OpenAI as embeddings and LLM, it would be great to see MPT being used instead since most commercially available LLMs don't have 4000 tokens or more as inputs.
but im not sure if you can fit MPT in a google colab free GPU or an average consumer GPU
The current MPT 7B doesn't play well with Colab Free. I will show some other models for embeddings in the next LangChain video
When generating embeddings you usually split up a large document into chunks anyway and embed each chunk separately, so that the embedding is more focused and closer to the embedding of a future (presumably short) query used to retrieve it. As I understand it, embeddings of longer token sequences are averaging the individual token embeddings, so meaning is lost if you overdo the input length.
@@clray123 Sentence embeddings aren't usually just an average of the token embeddings, they are usually trained specifically for sentence similarity or with some models they are trained for question and answer similarity. I agree that if you sequence is too long that you risk getting the semantic meaning clouded also if they are too short or no stride/overlap you risk getting no useful semantic meaning in the vector.
@@samwitteveenai Thanks for clarification!
I am getting a RateLimitError while creating the vectordb please provide a solution
how did you resolve it ?
I got this error:
InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4331 tokens (4075 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.
probably bring back too many contexts or their length is too long.
We have chroma, pinecone, redis, weaviate, faiss, qdrant, which is best?
This is a good idea to do a showdown of each of these, it will probably take a while to do. I do plan to show Pinecone and Qdrant at some point. I have no experience with weviate, but I do have a corporate customer who is using it, so will need to play with it at some point. Companies like Pinecone & Weaviate have raised hundreds of Mil so they really should be making a lot of content themselves.
@@samwitteveenai Theres also a Massive Text Embedding Benchmark (MTEB) Leaderboard on huggingface where openai's text-embedding-ada-002 is 6th place, others performing better and I guess you dont even need a paid api
@@nattyzaddy6555 Yes I showed that Leaderboard in the latest video.
hello, how can i get the link with answer from .txt files?
It can return the source info which would give you the file path etc.
how it is converting embeddings back to the text ?
Embeddings don't convert back to text we use them as a representation of the text for comparison etc.
@@samwitteveenai so when we query the embeddings and perticular segment of embeddings gets matched with then how that perticular text get return?
Instead of ingesting pdf/txt could this setup be used for code as well? Would be awesome to see a video on how this could be used against a repo or similar.
Yes there is a Python Loader you can use for that.
Bro- hitting a rate limit error, does this really need premium GPT?
how did you resolve it ?
wow
What did you use to scrub the site and create the .txt files? Awesome video as always and Tku for your work!🥳🦾
I wrote a simple script using Newspaper3k. its not perfect but super simple to do.
@@samwitteveenai awesome! Tku🦾