Thanks for this, this talk was excellent. I've been looking to combine LLMs with KGs and have very similar intuitions when it comes to using the same embedding space for the KG as for the LLM. I really like your frame of having the right abstraction spaces to solve the problem at hand. Having written countless prompts, as well as looking at how humans have solved problems over the years, it seems to me that fostering the right context (abstraction space) is vital when trying to solve a new problem. Einstein's discoveries were, in part possible due to the context of his life experience that gave him intuitions to solve a certain type of problem. The cool thing with LLMs is that we can bootload intuition at will, allowing us to swap out abstraction spaces until we find a combination that gives us the right context to solve a problem. Great work!
Thanks for the nice words! Subscribers are not my main goal, I just want to create a community for discussion and pursuing of interesting ideas. My main interests are to find out how to create fast learning and adaptable agents, and I find all this discussion helpful! Do join the discord to discuss more:): discord.gg/bzp87AHJy5
"maybe the text itself is way more expressive than the knowledge graph" ouch 1:16:10 in the age of LLM, it seems that any representation that deviates from the source text is a serious gamble
Updated the companion notebook to this video as OpenAI API and StrictJSON has been updated: github.com/tanchongmin/strictjson/blob/main/Experiments/LLM%20with%20Knowledge%20Graphs.ipynb
Where are these livestreams done? Is this a college course? I've actually never this hyped for a presentation-driven video, you've done a really good job walking thru the papers :)
Hey, can check out my discord group (link in my profile) for the link and details. Typically they are on Tuesdays, 1230-2pm (GMT+8). My focus is on fast learning and adaptable systems, knowledge graphs can help in that aim of knowledge representation for faster learning.
Hi there, if you are talking about the context-dependent embeddings, I have an implementation of it on github.com/tanchongmin/strictjson/tree/main/Experiments I found the most performant way is to re-adjust the text in the form that it incorporates the context. That way, the embedding model can be used without change. Check out the video here: th-cam.com/video/YZTThlPg0rU/w-d-xo.html
I think with LLMs, you do not need clustering anymore to get similar vectors - you can just simply use the LLM embedding and do vector search over it for similar vectors. That said, perhaps we can consider clustering embeddings together to build graphs. This would be interesting if we could build up the graph by grouping similar vectors together. However, we might lose out on explanability if we do everything as vectors - it can be hard to map back to words and meaning easily understandable by us. I am more of the opinion of using explanable links for the graph by using known relations, and then using the LLM to derive the vector embedding for it. Happy to discuss more.
While I also appreciate the flexibility of knowledge graphs (KGs) as far as them being able to easily represent relationships, I too agree with you that KGs are not necessarily the best or most effective way to represent intelligence. I will stay in tune with your works. I hope to publish on this in the near future. Thanks for the presentation.
Glad it helps. I am actively pursuing my idea of multiple abstraction spaces, and KG can be one of them. The rest of how we store memory will depend on what kind of memory - semantic facts, episodic memory and so on. These can be stored in various ways like traditional databases, or even in video/image format.
@@johntanchongmin Thanks for sharing your research. I will particularly follow closely your work on context-dependent embeddings. That's an exciting angle to explore in depth.
@@rajathslr You can refer to the paper I covered in this video: arxiv.org/html/2306.08302v3 Can look at an example KG-RAG workflow here: www.nebula-graph.io/posts/KG_RAG_Bridging_Gap_Knowledge_and_Creativity_of_LLM_based_Agents
I went to read the paper, seems like Graphormers is a type of Graph Neural Networks, that adds in more information about node centrality into node vectors and edge information in the adjacency matrix. I think it still suffers from the problems associated with GNNs such as oversmoothing and oversquashing. I am of the opinion we should just use a Transformer like embedding rather than a graph one, but store it in a graph structure with various embeddings for different contexts. This avoids the problems with GNNs while maintaining an informative embedding space that is connected in a graph structure.
thanks for the content. It really states challenges for the present and future. He forget somehow about openAi, and gives google a role of good, remembering that #don't be evil# motto.... but is it something else behind that old motto?
1:13:25 "When was the MacNCheese pro announced?" --> Fail! But, I wonder - what if the source text was stored as alongside the triplet? So that way the graph capability could be used to efficiently identify the relevant records, but the original language would be retained.
Good thought. Storing things at multiple levels of abstraction can help with different problems. So the KG version can help with entity relations, and the original text can help with QA.
suppose we have large pdf about some person and we want rate him based on skills which is define in pdf so those kind of questions which is rely on not just based on subpart of text but whole text at that time how we can appporach this problem with KG and Vector Embedding stuff. and every time we can't call langchain summarization api (chain_type:stuff) becuase it's costly how we can solve this problem
You can try streaming. That is, take a part of the text and extract out the information you require. And keep updating the information with each part of the text until you are done. This way may be much better than extracting the information into KG first and then extracting skills, as there may be information loss when extracting into the KG if the skills are not clearly stated in the pdf.
In my opinion, it will be a meta-category to group a problem into like a macro space or a micro space, and then the relevant abstraction spaces get called to solve the problem from that categorisation.
at around 23:00 it is said that current knowledge graphs are 'too restrictive' because of context. but the way i see it, they are too broad. you still want that total knowledge available even if its not currently relevant, we just want to filter it down.. right?
I actually meant that it was too restrictive because current knowledge graphs are largely static, and the meaning of each node is pretty much independent of the other nodes. Granted, Graph Neural Networks can pass information from adjacent nodes, but they need to be trained extensively and may also not be the ideal form of representation due to oversquashing and oversmoothing. I am looking for a flexible, dynamic representation, that can change as the Knowledge Graph builds. This is what I call context-dependent knowledge graphs.
If you could give me a context for the decentralized knowledge graph, I can better answer your query. In general, it is better for the knowledge graph to be centralized, so that when you add entities and relations, they do not clash. If it is decentralized, there needs to be a conflict management system to see what gets added and what does not.
You're the first person I've hear mentioning this concept of context-dependent embeddings. I started tinkering with the same idea back in December of last year, never had a name for it. I was doing some self-reflection and thought about how some of my own behaviors and thoughts were contradictory sometimes- dependent on how my emotions were and such. If I could make a certain perspective of mine a 'node' it's embedding would very likely change given different contexts
Great info. I have a small question. KG parser what you are talking about expects KG as input however if we have a huge dataset constructing and sending such KG will cost more right?
For the knowledge graph parser I demonstrated using StrictJSON, yes, you will need to parse through every node to identify which are relevant. It is the most performant, but costs a lot. The alternative is to derive an embedding for the node, and an embedding for the query, use cosine similarity to find nodes that are similar to query and use it to augment context via Retrieval Augmented Generation. For context-dependent knowledge graphs, add on the parent node as context to the current node, and create a new vector that has context info as well. Based on what I have tried so far, using a combination of original vector plus the context-based vector is superior than just using the original node vector.
again..this is very very similar to Infranodus you take a look. Also (1) should this approach identify edge weight between topics? (2) does Langchain factor edge weights? (3) do you need a vector database for this approach?
@@RedCloudServices Had a look at Infranodus. It is quite interesting how it uses rule-based methods to extract out influential entities and relations and generate the graph. The edge weight can also be done via LLMs, especially if you have predefined criteria for each weight. You can do few-shot prompting and use LLMs to extract from an arbitrary text. Langchain as of the video does not have edge weights - it can be easily done with some prompting. Best if we have fixed categories to group the edge weights to. At the moment, the LLM to KG construction does not use a vector database - it simply extracts out (entity, relation, entity) triplets from the text. I can imagine vector similarity can be useful if we want to disambiguate entities. Entities with high cosine similarity can be treated as the same and we can build on to the existing entities on the KG. This of course requires the embedding space to be expressive enough to disambiguate between different entities and cluster similar entities together. More recently, I've been thinking about whether we can use more rule-based and more fixed methods of extracting out entities from the text. One problem of LLMs is that it may not get the entity you would like for your ontology. So few-shot prompting is quite important, and if you have rules to extract out the entities, it might perform better. In a longer term, I am also thinking if ontology is really needed. Can we just store the whole chunk and infer it based on the context at run-time? I have a new approach I term "Chain of Memories", you can check it out on my discord:)
Thanks for this, this talk was excellent. I've been looking to combine LLMs with KGs and have very similar intuitions when it comes to using the same embedding space for the KG as for the LLM. I really like your frame of having the right abstraction spaces to solve the problem at hand. Having written countless prompts, as well as looking at how humans have solved problems over the years, it seems to me that fostering the right context (abstraction space) is vital when trying to solve a new problem. Einstein's discoveries were, in part possible due to the context of his life experience that gave him intuitions to solve a certain type of problem. The cool thing with LLMs is that we can bootload intuition at will, allowing us to swap out abstraction spaces until we find a combination that gives us the right context to solve a problem. Great work!
Some valuable content here how u only have 2800 subscribers is a travesty
Thanks for the nice words! Subscribers are not my main goal, I just want to create a community for discussion and pursuing of interesting ideas.
My main interests are to find out how to create fast learning and adaptable agents, and I find all this discussion helpful!
Do join the discord to discuss more:): discord.gg/bzp87AHJy5
Slides: github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLM%20with%20Knowledge%20Graphs.pdf
"maybe the text itself is way more expressive than the knowledge graph"
ouch 1:16:10
in the age of LLM, it seems that any representation that deviates from the source text is a serious gamble
Updated the companion notebook to this video as OpenAI API and StrictJSON has been updated: github.com/tanchongmin/strictjson/blob/main/Experiments/LLM%20with%20Knowledge%20Graphs.ipynb
Where are these livestreams done? Is this a college course? I've actually never this hyped for a presentation-driven video, you've done a really good job walking thru the papers :)
Hey, can check out my discord group (link in my profile) for the link and details. Typically they are on Tuesdays, 1230-2pm (GMT+8).
My focus is on fast learning and adaptable systems, knowledge graphs can help in that aim of knowledge representation for faster learning.
@@johntanchongmin Amazing work ! Would love to connect and exchange knowledge ☺
@@agovil24Sure thing, can find me on my linkedin or discord. Check my profile page.
Curious how far you are 😅 i relistened to this and now second probably we are trying to enhance for the same solution
Hi there, if you are talking about the context-dependent embeddings, I have an implementation of it on github.com/tanchongmin/strictjson/tree/main/Experiments
I found the most performant way is to re-adjust the text in the form that it incorporates the context. That way, the embedding model can be used without change. Check out the video here: th-cam.com/video/YZTThlPg0rU/w-d-xo.html
Amazing work. Many thanks for your efforts sharing your knowledge
In the topic of graphs, how would you envision combining LLM with UMAP and HDBSCAN for data exploration? th-cam.com/video/r8dWZX8IGw8/w-d-xo.html
I think with LLMs, you do not need clustering anymore to get similar vectors - you can just simply use the LLM embedding and do vector search over it for similar vectors.
That said, perhaps we can consider clustering embeddings together to build graphs. This would be interesting if we could build up the graph by grouping similar vectors together. However, we might lose out on explanability if we do everything as vectors - it can be hard to map back to words and meaning easily understandable by us.
I am more of the opinion of using explanable links for the graph by using known relations, and then using the LLM to derive the vector embedding for it.
Happy to discuss more.
Knowledge packed video and excellent teaching skill.
While I also appreciate the flexibility of knowledge graphs (KGs) as far as them being able to easily represent relationships, I too agree with you that KGs are not necessarily the best or most effective way to represent intelligence. I will stay in tune with your works. I hope to publish on this in the near future. Thanks for the presentation.
Glad it helps. I am actively pursuing my idea of multiple abstraction spaces, and KG can be one of them. The rest of how we store memory will depend on what kind of memory - semantic facts, episodic memory and so on. These can be stored in various ways like traditional databases, or even in video/image format.
@@johntanchongmin Thanks for sharing your research. I will particularly follow closely your work on context-dependent embeddings. That's an exciting angle to explore in depth.
You stretched my mind, thank you for taking the time to share.
Update: StrictJSON is now a python package
Simply "pip install strictjson"
Head over to github.com/tanchongmin/strictjson to find out more
Forgive me if my question is not correct, Are we using LLMs to build a Knowledge graph here?
Indeed, I showcased how to use strict_json (LLM) to build a Knowledge Graph.
We also can use KGs to provide context for LLMs, so it works both ways.
@@johntanchongmin 'We also can use KGs to provide context for LLMs' Can you please send me any video/blog link where I can read more about this
@@rajathslr You can refer to the paper I covered in this video: arxiv.org/html/2306.08302v3
Can look at an example KG-RAG workflow here: www.nebula-graph.io/posts/KG_RAG_Bridging_Gap_Knowledge_and_Creativity_of_LLM_based_Agents
@@johntanchongmin Thanks for your time, you really rock!!
what about Graphormers ? no need for different embedding spaces: language token sequence is just a specialization of a graph
Wow let me go check it out, sounds promising
I went to read the paper, seems like Graphormers is a type of Graph Neural Networks, that adds in more information about node centrality into node vectors and edge information in the adjacency matrix.
I think it still suffers from the problems associated with GNNs such as oversmoothing and oversquashing. I am of the opinion we should just use a Transformer like embedding rather than a graph one, but store it in a graph structure with various embeddings for different contexts. This avoids the problems with GNNs while maintaining an informative embedding space that is connected in a graph structure.
@@johntanchongmin just like a mind map, right?
@@wilfredomartel7781 Indeed, a mind map (or a graph), with multiple vector embeddings for each node (each embedding with different tiers of contexts)
thanks for the content. It really states challenges for the present and future. He forget somehow about openAi, and gives google a role of good, remembering that #don't be evil# motto.... but is it something else behind that old motto?
Valuable content! Thank you for sharing:)
1:13:25 "When was the MacNCheese pro announced?" --> Fail!
But, I wonder - what if the source text was stored as alongside the triplet? So that way the graph capability could be used to efficiently identify the relevant records, but the original language would be retained.
Good thought. Storing things at multiple levels of abstraction can help with different problems. So the KG version can help with entity relations, and the original text can help with QA.
Thanks
suppose we have large pdf about some person and we want rate him based on skills which is define in pdf so those kind of questions which is rely on not just based on subpart of text but whole text at that time how we can appporach this problem with KG and Vector Embedding stuff. and every time we can't call langchain summarization api (chain_type:stuff) becuase it's costly how we can solve this problem
You can try streaming. That is, take a part of the text and extract out the information you require. And keep updating the information with each part of the text until you are done.
This way may be much better than extracting the information into KG first and then extracting skills, as there may be information loss when extracting into the KG if the skills are not clearly stated in the pdf.
1:31:30 "Mix and match" - I'd be interested in understanding how the AI might decide which space(s) to interrogate based on the input/prompt.
In my opinion, it will be a meta-category to group a problem into like a macro space or a micro space, and then the relevant abstraction spaces get called to solve the problem from that categorisation.
at around 23:00 it is said that current knowledge graphs are 'too restrictive' because of context. but the way i see it, they are too broad. you still want that total knowledge available even if its not currently relevant, we just want to filter it down.. right?
I actually meant that it was too restrictive because current knowledge graphs are largely static, and the meaning of each node is pretty much independent of the other nodes. Granted, Graph Neural Networks can pass information from adjacent nodes, but they need to be trained extensively and may also not be the ideal form of representation due to oversquashing and oversmoothing.
I am looking for a flexible, dynamic representation, that can change as the Knowledge Graph builds. This is what I call context-dependent knowledge graphs.
Does most of this apply to decentralized knowledge graphs?
If you could give me a context for the decentralized knowledge graph, I can better answer your query. In general, it is better for the knowledge graph to be centralized, so that when you add entities and relations, they do not clash. If it is decentralized, there needs to be a conflict management system to see what gets added and what does not.
You're the first person I've hear mentioning this concept of context-dependent embeddings. I started tinkering with the same idea back in December of last year, never had a name for it. I was doing some self-reflection and thought about how some of my own behaviors and thoughts were contradictory sometimes- dependent on how my emotions were and such. If I could make a certain perspective of mine a 'node' it's embedding would very likely change given different contexts
Nice, do let me know if you have any feedback / add-ons to this idea
Also, video on Context-Dependent Embeddings here: th-cam.com/video/YZTThlPg0rU/w-d-xo.html
Great info. I have a small question.
KG parser what you are talking about expects KG as input however if we have a huge dataset constructing and sending such KG will cost more right?
For the knowledge graph parser I demonstrated using StrictJSON, yes, you will need to parse through every node to identify which are relevant.
It is the most performant, but costs a lot.
The alternative is to derive an embedding for the node, and an embedding for the query, use cosine similarity to find nodes that are similar to query and use it to augment context via Retrieval Augmented Generation.
For context-dependent knowledge graphs, add on the parent node as context to the current node, and create a new vector that has context info as well. Based on what I have tried so far, using a combination of original vector plus the context-based vector is superior than just using the original node vector.
Knowledge Graphs were developed during a time of low data regimes and no large models.
Indeed, that is why in my opinion, we need to change our mentality of fixed nodes and relations, and go towards context dependent embeddings.
❤
again..this is very very similar to Infranodus you take a look. Also (1) should this approach identify edge weight between topics? (2) does Langchain factor edge weights? (3) do you need a vector database for this approach?
Could you let me know which approach you are referring to, so I can reply accordingly :)
@@johntanchongmin approach #3
@@RedCloudServices Had a look at Infranodus. It is quite interesting how it uses rule-based methods to extract out influential entities and relations and generate the graph.
The edge weight can also be done via LLMs, especially if you have predefined criteria for each weight. You can do few-shot prompting and use LLMs to extract from an arbitrary text.
Langchain as of the video does not have edge weights - it can be easily done with some prompting. Best if we have fixed categories to group the edge weights to.
At the moment, the LLM to KG construction does not use a vector database - it simply extracts out (entity, relation, entity) triplets from the text. I can imagine vector similarity can be useful if we want to disambiguate entities. Entities with high cosine similarity can be treated as the same and we can build on to the existing entities on the KG. This of course requires the embedding space to be expressive enough to disambiguate between different entities and cluster similar entities together.
More recently, I've been thinking about whether we can use more rule-based and more fixed methods of extracting out entities from the text. One problem of LLMs is that it may not get the entity you would like for your ontology. So few-shot prompting is quite important, and if you have rules to extract out the entities, it might perform better.
In a longer term, I am also thinking if ontology is really needed. Can we just store the whole chunk and infer it based on the context at run-time? I have a new approach I term "Chain of Memories", you can check it out on my discord:)