Advanced RAG 01 - Self Querying Retrieval
ฝัง
- เผยแพร่เมื่อ 8 ก.ค. 2024
- Colab: drp.li/T0SDK
My Links:
Twitter - / sam_witteveen
Linkedin - / samwitteveen
Github:
github.com/samwit/langchain-t... (updated)
github.com/samwit/llm-tutorials
00:00 Intro
00:47 Self Querying Diagram
01:55 Code Time
#langchain #openai #llm - วิทยาศาสตร์และเทคโนโลยี
![CAMPปลิ้น | EP.77[2/2] "รับทราบ" แก๊งเพื่อนรัก 10 ปี ที่ไปไหนก็มีตำนาน!](http://i.ytimg.com/vi/ifc0ofD51B0/mqdefault.jpg)
I am always curious to see what Langchain does under the hood. This is a super useful video, and I will dig a little deeper to see how the actual query and filters are determined. Thanks 🙏
Thanks a lot Sam for this new tutorial. I really like the way the documents are prepared and the precision of the RAG. Willl definitley give it a try very soon
This was a fantastic walk through and explanation thanks
Great insight. Content like this helps a lot to keep the costs low. Thanks 🎉
Perfect. Thank you.
This is just amazing! This could potentially change the way how we can utilize langchain. Great video 👏. Kudos!
What is langchain used for typically? This above thing could also be done with simple HF BERT type stuff too, right?
Thank you very much for sharing great videos.
Very nicely explained, thank you
this is a great video and a very interesting Colab: please continue with tutorials on this retrieval topic
I implemented RNG to submit together queries to make my potty mouth AI bots even more toxic. It's almost like jailbreaking them. I will remember this video for more professional things tho. Very cool!
Exactly what I was looking for, thanks 🎉
Great video Sam. Keen to uncover more of the hidden gems in langchain
Thank you so much for this.
love your work
I am building a pilot of Q&A chatbot for physicians based on RAG - thank you for video.
Great video, as always.
Nice feature of langchain, but it works only if the metadata are not in the scattered in the documents itself.... Thanks for the video👍
Such premium content.
Great video!
very interesting thx
this + an additional chain that takes user input and does some kind of automatic entity extraction and inserts them back into the vector database = best second brain software ever
Do you mean essentially chatting with a Vector DB and having the ability to add/incorporate new info back in to it? That would be extremely useful for building Knowledge Bases
I think there’s a good opportunity for developers to learn SpaCy to handle some NPL tasks, like named entity recognition or finding sentence boundaries, that can be used where you don’t need the full power (or costs) of an LLM. You could use GPT for rapid prototyping but then isolate the specialized tasks to spaCy or a local fine tuned Bert. The first programmers who apply this method will not only have something that costs less but also would have better traceability/interpretability.
@@J3R3MI6 yep
@@mshonle my idea is to combine both NER and LLM, turning user input into various embeddings that can be later queried via LLM. think of it as a digital diary/agenda, where you can write stuff but also retrieve with natural language.
Thx for the video :) can you update your vector database by a few lines ( if you want to add data to your knowledge base) automatically by running a python script or something like that?
Another way of putting it is that you should only be using LLM as a classifier, in order to resolve ambiguity of NL queries, then translate them into normal database queries (which you can also understand and debug, unlike all the magic LLM output). While it is clearly taking a step back from the ridiculous attempts at placing AI in the driver seat, this approach actually works for real applications and brings added value... just like OCR, voice, or image recognition brings added value to tasks that require such capabilities. And when you need reasoning, idk, maybe use a FOL theorem prover or something.
It depends on the kind of data you are looking for. If you are doing a deterministic search, having the LLM call a relational database would be a good choice. If you are looking for something a bit more fuzzy, a vector database would be the best choice. If the language model finds a tool that says, "Search for fish with green heads." Then it will use that tool. If you have a MySQL database with fish head color as a field, that will work great. If you have a bunch of random text and you want to find references to fish with green heads in that text, put the text into a vector database and search that.
@@s.patrickmarino7289The point is that vector proximity search is not that different from any other search done in a relational db. You don't need any special vector db or even LLM, just a db which supports vector-related indices and embed whatever fuzzy phrase you want to search for. You could even parse the whole NL query manually using something like Spacy (which can easily create a dependency and parts-of-speech graph for a sentence). The LLM bros are probably using these new dbs because they have no idea of what traditional dbs (combined with embedding) are capable of.
Waiting for next video with open source model and with csv data
Thanks for this great video. Just a newbie question from me: how to formulate RAG setup for this type of text data, representing activities of a person at various timestamps:
Day1 7-00-00 : breakfast
Day1 7-00-08 : workout
Day1 7-00-16 : reading
......
Day1 7-30-59 : cooking
......
Day20 7-00-00 : breakfast
Day20 7-00-08 : reading
Day20 7-00-16 : workout
......
Day20 7-30-59: gardening
Considering there are 20 days and each day involves 200 timestamps, the total number of entries will be 4000. My target is to count the number of occurrences of activites at some time stamps. How should I follow your method to construct docs and metadata_field_info and the query?
Hi, thank you very much for the video. Any idea on what needs to be adjusted to run it with ChatOllama? I tried ChatOllama(model='llama2') instead of the ChatOpenAI and it fails with some "invalid JSON" object error. Perhaps I will need to look inside what's going on inside langchain, but maybe someone already figured how to overcome that issue.
@sam if i am using bge embedding model which llm should you suggest to use in SelfQueryRetriever, or is there a opensource implementation.
Awesome content,please post a video on how to chat with google drive documents.
Hi Sam, how would you implement semantic search with data reference when retrieving results over a large number of different documents?
I am also very interested in this question
Hi Sam. In your code, you are using metadata type="string or list[string]", but it seems that you cannot use `list` as a type for metadata. Can you please confirm?
how to do for csv with numrical values,i have fianace data & want to get some p& loss from it.some charts out of it,its possible using this?
Thank you for this, and apologies for a newbie question: this is an example of highly structured data, right? Those documents with metadata seem pretty much like an elastic search, or even structured data in a DB.
Great video, Sam.
What if we had a wine that has two types of grapes?
I tried to add another grape: "" entry to the collab, and it replaces the first.
If we had several grapes separated by commas, e.g. grape: ", ", we now need the structured query to be a bit different:
Instead of using the Comparator.EQ, in the following example, it would use CONTAIN
query='Sémillon' filter=Comparison(comparator=, attribute='grape', value='Sémillon') limit=None
[docs]class Comparator(str, Enum):
"""Enumerator of the comparison operators."""
EQ = "eq"
NE = "ne"
GT = "gt"
GTE = "gte"
LT = "lt"
LTE = "lte"
CONTAIN = "contain"
LIKE = "like"
The question is, how to induce the structured query to use CONTAIN?
dufuq you jammerin on abt mate? llms do that all day baby.
Great explanation on the why and how of Self Querying Retrieval!
My RAG app retrieves from 2 types of documents. I want the LLM to first look in document A, and then only look in document B if it can't find the answer. I was trying to insert meta fields to distinguish the document types, and came across Langchain's self-querying. But I think I'm barking up the wrong tree.
Any suggestions for my use case?
Perhaps write some branching scenario for that case?
I think the topic of automatic metadata creation for Documents worth some time, for example Langchain's tagging.
Yeah this is certainly something you can do though it doesn't always work out perfect. I have had some funny tags when doing it before. I will add it to the list of possible videos
Hello Sam This was a really great video. But building a query retrieval system requires substantial resources so is there a way to optimize that query system to work on the least RAM or GPU and still work efficiently?
So a lot of it depends on the scale you are operating at. In regards to the GPU if it is just for the embeddings models the small bge models can run on cpu with ok speed. you can use an external vector store in the cloud which work pretty well. What is your use case?
@@samwitteveenai My use case is to generate recommendations on clothing datasets based on the product description
Firstly Thanks. I tried this and I used llama2 as llm and chromadb as vector store, the filter is getting detected correctly but the maximum no of records I am getting as a result are 4 even though there are more records which are satisfying the filter, please suggest
Hi Sam, great tutorial! :D
I have 3 questions:
1. Why do you use in the attribute description for "color", that it can be a list of strings? Is it possible to have a metadata field that is a list?
2. Do you know if it's possible to make the self query retriever use more fields than just page_content for semantic search?
3. Why do you use name as metadata field? Cause as far as i understand, the LLM doesn't know the exact names, so it wouldn't find the relevant document if the name is slightly missspelled from the user. Or are string metadata fields searched by semantic search and not by their exact value?
Would really appreciate some help! I hope the questions are clear, if not, feel free to ask! :D
Anyone have luck self querying on date-like strings (e.g., 2023-12-04) in Chroma?
I've tried metadata with string type, and also converting it to utc integer timestamps. Creating separate md fields for y/m/d seems clunky. The query constructor makes logical queries for me, but it seems there's not support for date types. Confused, would think this would be a major use-case.
Awesome, video!! That's so useful!! Does Self Querying Retrieval perform sorting as well? For example: "What are the oldest wines from France?"
Good question, I don't think so, but since they are all going into the LLM you can use that to sort them for the end user easily enough.
i suspect this is what tableau, microsoft, mongo etc all use for natural language querying
a plylist for this would be very useful
I think this is 1 out of 1. When he adds more, I suspect there will be a list.
Hey Sam! Great video, is this possible to build using flowiseai?
Yes I think you can use Flowwise to do most of the LangChain tasks etc
@@samwitteveenai Sounds great, maybe you have video or flow example? Because I didn't find something related to Self Querying Retrival in flowise. And I have very similar use case as in your video. I will be kindly appreciate for your help.
In the last example, llm automatically capitalized the first letter of country names. What if, in the dataset, they are not capitalized but are lowercase as you inputted?
Great Video❤
Video request or hint appreciated:
I would be interested in a video on a chatbot which builds up questions to narrow down search results on my unstructured data.
" builds up questions to narrow down search results" you mean like a memory?
@@samwitteveenai more like... human: please give me the carpet...ai: ok, shall i give you the red or the black one.... human: the red....ai: ok. You have what you need.
This based on a own dataset.
You would use the prompt for doing this kind of thing more than a RAG alone.
@@samwitteveenai this is clear. I am wondering what could be a path forward.
Excellent explanation. Thank you for sharing! I would like to know if Langchain is a good choice for production. Can you tell me, please?
It depends on your use case. There are many people using it for LLMs apps. I use it for some things and for others I don't.
I get it. Thanks for answering me!
Does it work with rerank?
How can one query pdf contains table and text as well
?
Can we use Self Query Retrievers with Open Source LLMs such as Llama-2? If yes then how?
I'm getting error on this statement
retriever.get_relevant_documents('What are some red wines')
TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not StringPromptValue
I assume that LangChain sends attribute info and ask LLM: make me filter with operators and used attributes in question I sent u. So again it is semantic query. Somehow example isn't crystal clear. It is blurry what should be treated as NOSQL query/metadata (lot of manual work included) as easily we can have flavour as meta data. Anyway I bet combined search will yield speed, even with relations databases as you can easily can translate semantic question to SQL query.
is this a query builder? what is new here i don't get it? It's been there since gpt 3.5 went out.
The real challenge is the text search where when you have "apple" in your field, and the user searches for "fruit", and does it in sub 100ms.
I suspect it will come down to having the AI decide, "I will use a vector to find this. That one I will look for it on my relational DB." It also seems to be about building different tools for different kinds of searches. Remember, this first video is a foundation Sam will be building on. The first one may not be doing anything all that exciting, but it is setting the stage for what comes next.
@@s.patrickmarino7289Algolia search is experminting with it, i have seen a demos where their AI search is doing the same with sub 100ms, as for query builder, chatgppt can do it since day 1.
I'm also not sure why you would use some dedicated weird vector db when you can just add vector functions to a production-ready db like ClickHouse.
For demo purposes I suppose. A lot of mpp databases do have vector capabilities, not just Clickhouse
@@alx8439 Yeah, Postgres also a good candidate.
it just keeps it all in the notebook easily. As I said in the video I wouldn't use OAI Embeddings either, but it makes it simple for people to see the Self Querying part.
love it! I fed the list of wine description and metadata to my local llm, gave it two examples of metadata_field_info and asked it to complete that info. it did so with flying colors. what a glorious feeling.
hey sam have you seen the latest open source model CulturaX
I see it as a dataset but didn't know there is a model with it
Unfortunately this doesn’t work with a vector store constructed with PGVector in a Postgres database😢
interesting why is that? They don't support meta data?
@@samwitteveenai langchain.retrievers.self_query.base import SelfQueryRetriever has a function def _get_builtin_translator that doesn’t implement the Translation for Postgresql
great video, us always ! plz i m having a big issue with the open source embedding BGE. i m gettin an error when trying to store in chroma : ValueError: Expected metadata value to be a str, int, float or bool, got None which is a
Try filtering complex metadata from the document using langchain_community.vectorstores.utils.filter_complex_metadata.
and i don't have none values in my data. any help?
not sure off hand . which of the BGE models are you using? make sure you have sentence_transformers installed also could be a version issue with Chroma, I have had that happen in the past
thank you for your answer. i m using chromadb 0.4.22, sentence-transformers 2.3.1 and BAAI/bge-base-en@@samwitteveenai
All of these demos are fine, but can we have a sample where function calling, these meta data works in tandem. Else these only work in isolation and not in reality. This will be video to watch. With real ingestion
What do you want the function calling to do? Access a tool for some inputs and RAG for others?
@@samwitteveenai yes. And this brilliant video of yours along with meta+semantic search.
Such a demo will be fantastic!
Fruity nodes? I thought it was fruity notes, but maybe AI has different opinions about that.
How is this advanced??
This is basic RAG, not advanced as the title suggests.
Whoa! I always thought that semantic searcher utilizes metadata automatically. Meaning, in your example, I thought that normal vectorstore.relevant_documents will automatically pick the relevant documents by looking to the content+metadata without help from LLM.
Great video!