วีดีโอ

Haha. Brilliant. Love the behind-the-scene piece at the end. Very instructive. Thanks 🙏🏻

Here's a ChatGPT summary: - Retrieval bandage generation Bragg is effective in reducing hallucinations in large language models (LLMs). - Despite providing correct context and ground truth, LLMs often do not incorporate external knowledge correctly. - Viewer comment suggests LLMs may prioritize their internal knowledge over external information. - Research compares GPT-4, GPT-3.5, and MST-7B in balancing external information with internal knowledge. - GPT-4 is the most reliable model when using external information, followed by GPT-3.5 and MST-7B. - All models tend to stick to their internal knowledge if they believe external knowledge is less correct. - Rax can enhance accuracy, but its effectiveness depends on the model's confidence and prompting technique. - Different combinations of language models and embedding models can lead to varied results. - The study highlights the influence of different prompting techniques on how LLMs follow external knowledge. - SPI framework for auto-tuning prompts can improve LLMs' adherence to external knowledge. - SPI uses bootstrapping to create and refine examples, improving prompts based on specific metrics. - Entity linking can prevent incorrect answers by mapping and identifying words in text to entities in a knowledge graph. - Default Knowledge Graph is used for validation due to its extensive network of verified information sources. - Entity linker helps filter made-up information when LLMs hallucinate. - DSP RAC pipeline updated with entity type validity check improves output accuracy. - Custom DSP RAC pipeline integrates knowledge graph data to refine questions and retrieve relevant information. - Two metrics for DSP optimizer: entity type check and alignment with knowledge graph context. - Knowledge graph context ensures final answers align with ground truth. - Enhanced output incorporates specific passages and relationships from the knowledge graph. - Example shows knowledge graph confirming Elon Musk as the sole founder of SpaceX. - Optimized program sometimes fails to make LLMs stick to external knowledge. - Manual prompt tweaking may be necessary to ensure LLMs follow external knowledge strictly. - DSPive framework has a steep learning curve but can yield better results for experienced programmers. - Main message: Integrating knowledge graphs and entity linking with LLMs can improve accuracy, but manual prompt customization may still be necessary to ensure adherence to external knowledge.

This is the way...

I haven't seen any good examples of the Self-improving part of DSPy yet. Is it ready for mainstream use❓

thanks great video!

girl, u rock thank you so much for this!! Where can I follow you?

Love this kind of RAG comparison, would be better if the background looks less like a spa room.😅

Until I saw this, I was starting to think that there was something wrong with me not being able to achieve magical improvements in results by using DSPy over meticulously hand-crafted prompts targeted at the observed quirkiness of specific LLMs. Thank you for restoring my self confidence. And now I'm also going to incorporate graph databases into my RAG pipelines after watching a couple of your videos.

great video! thanks

Hey @lckgllm, could you also upload the missing `dspy-requirements-2.txt` in the repo? 🤗

Very interesting, thanks! But Musk wasn't a co-founder of Tesla either.

Thanks 🙏🏽

Great info and content.

Impressive a 996 worker can find time to put this together, keep it up! Ah yeah, RAG doesn't work, in fundamental ways... it can't.

Love it! Finally Neo4j has good usage!

Great video~ Very interested in KG' adaption in LLM. Kindly reminder: hide your api key in the video😉

Music is too annoying. Please dial it down

It is so funny how tech people do not understand why ChatGPT was a hit: exactly because you can use it without programming it.

How good are llms at generating cypher queries?

how can peraon be both smart and pretty

Is this parody? Considering the title… and the result

I’m sorry are the headers wrong the gpt and llama headers swapped on the nomic tests in your paper so either the header is wrong or what you said we’re all opposite

Hey I’d love to understand how you can improve agents at using tools. Thanks 🙏!

"Don't look at me" is an instruction no language model could ever follow.

Lovely 🌹

Very interesting, it seems we're not too far from getting a robust retrieval. Would explicitly asking in the system prompt to prioritize the given context over any prior knowledge change this behavior?

You got a topic I exactly was looking into.. Now subscribed....I will follow on X.

Thanks for the video! I tried this but with convert_to_graph_documents I kept getting empty graphs. While debugging I found an error message at an intermediate step, it said "Your plan doesn't have this feature", apparently it won't work with the free version of Diffbot? $299 per month would be a bit much for a small student project... Do you have any other advice for getting a solid Neo4J graph from text data? Crucially, I'd like metadata to be included with the relations, which didn't happen with the LLM Graph Transformer

10:38 Hypotheisis: what if providing additional data from KG does not override knowledge ("false or hallucinated facts") already inherently present in LLM. I wonder what would happen if you change labels of knowledge graph to same abstract names like Person1, Person2, Company1, Company2 etc. and try to run the exact same program. Would it dramatically change result?

Your question number 87 in your Jupyter notebook has the casing of Joe Hisaishi ‘s name differently than the documents, it is all lower case in the question. So it will be tokenized differently. Could this have influenced the quality of the answer perhaps?

Do you think the same depth and accuracy could be achieved with a metadata-filtered vector search followed by a reranker? I worry about having to maintain two data representations that are prone to drift

Which language model did you use? Llama 3 and Sonnet might offer improvements to recall over RAG context.

You are the best! Congrats!

ANNOUNCEMENT: Diffbot's APIs and the Diffbot Knowledge Graph have become FREE! You can freely access them by signing up at: app.diffbot.com No more free trials because it's free 🤩

I now see why i dont get good results with rag s.

Interesting, but professional accounts only. I guess I will stick with Neo4j.

Great video! I need to try this.

Definitely learned a lot, and now keen to see some kind of tutorial on how to implement the knowledge graph backed RAG. I’m a Ruby dev so something language agnostic would be particularly helpful.

This is amazing!! Thank you! I already felt much pain with managing the token size in early days of LangChain.. this was truly eye-opening!

As I'm reviewing this video again, I notice that at 3:54 you show the Prompt for assessing relevance. I've tried this kind of prompting before and I always find it hit or miss. It depends a lot on the model and how well it can assess the information and assign the correct value. But if we look at the prompt, ultimately we're asking it to check to see if key words from the original question show up in the results. If not, we give it a "yes" or "no", but then ask it to bundle the binary score into JSON with "score" as the key. I have two thoughts on this. One, we're asking it to follow a lot of instructions here. For models like GPT4, it probably won't be a problem, but for other models, well, some yes, some no. There's a reasonable chance that models not trained on JSON may foul that up. Even some that have been trained on JSON can foul it up, sometimes (especially if the temperature you set is too high). The other issue is that where it says "binary" and then "yes" or "no", some models may get confused by what is being asked for here. Binary can also mean that it will return the result 1 (yes), or 0 (no). Because that is what binary suggests. So you may wind up with some hit-or-miss results because of that. Another way to handle that is to simply as for a boolean answer, True or False. And don't ask it to assign it to JSON. Just have it return the result as a boolean value. Then you can pick it up, most likely, as a boolean value in the answer. But more importantly. Why ask the model to do this at all? This is just the kind of spot that causes hiccups. I'm thinking that with this kind of operation, wouldn't it be safer to simply interrogate the each page content for the key words using python search through the text? You could do something like this: # Define the keywords to search for keywords = ["Computer", "Document", "Batman"] # Define the text to search through page_content = "This page contains information about computers and documents. Batman is a popular superhero." # Search for the keywords in the page_content found_keywords = [] for keyword in keywords: if keyword.lower() in page_content.lower(): found_keywords.append(keyword) # Print the found keywords if found_keywords: print("The following keywords were found in the page content:") for keyword in found_keywords: KeyWordCount +=1 While I admit, this approach is not as cool as using the LLM... it would definitely be more reliable. You could then use the keywordcount as a way of scoring the document, and include it if the keywords meet a certain threshold. In other words, I think it's important to factor in that LLMs make mistakes (like all the time), and so when we design programs to use LLMs we want to carefully select when we're going to use the LLM, and choose not to use it whenever possible. So if we can get a result without using the LLM, not only are we more assured of accurate results, but it will also cost far less to run the program. The fewer calls to the LLMs the better. Anyway, just some thoughts. Not sure if that actually makes sense, but ... there you have it. I hope it does. As a general rule, though, I think the point I'm making here is pretty solid. Use the LLMs as sparingly as possible, and only for those things that you can't achieve without the LLM. Yah?

Long context <=> RAG Pen Pineapple Apple Pen

The most valuable contents on youtube are the least liked.. Often..

Another great video that brings up so many vital questions. I wish we had more time to dig into the details of what you just presented, and try to find solutions to the bottlenecks. Overall, I think the bottom line problem is that we are looking for ways to use LLMs to do what they are not actually good at because of the nature of their construct. LLMs are stochastic by nature. Therefore, because they deal in probabilities, you are not likely to get a 100% correct answer. So if you use RAG, the LLM that is doing the inferencing may wind up with invalid, or less probable responses. You can set the temperature to 0, and get the most probable based on the data, but as you point out, sometimes the data is the problem, and so you want the LLM to reason... which means you want a higher temperature, even at the risk of getting lower probability responses. It's a problem of having the fox guarding the hen house. You can't quite trust the LLM to validate its own assumptions. Of course, you could bring in more LLMs into the task and have them validate each other, but that introduces the probability that one or more of them get it wrong, throwing the effort into a tailspin. And if you go to the IntArWebZ to get information, well, as you pointed out, there's a LOT of junk out there. Totally unreliable. So where are your reliable data sources? And how can we trust LLMs to evaluate them? We've entered into a strange new world with LLMs. They seem to be great at processing language, but they're really kind of awful at finding facts, and doing reasoning (because even if they were good at reasoning, the deficit of a facts-system would still throw them off). What I imagine is a system (program) by which we have numerous dials and meters that we watch and manipulate as we explore the Sea of Information. When the objects we are looking at appear bright and solid, we are being told the system thinks they have a high probability of being factually correct, or correctly reasoned. So we swim towards them, but along the way, some moderately translucent information catches our eyes, and we wonder... is that lower probability thing worth looking at? So we swim towards it, and then we find out, why yes, inside there was a nugget of gold, hidden by the LLMs incorrect evaluation of its probability. And so we store that in our vector Knowledge-Tank, and keep swimming as we search for interesting ideas, and the facts to support them. I see a sort of 3D universe of vector graphs with visual properties that correspond to different aspects of what we need to know in order to understand what we're looking at. Super Sci-Fi stuff, I know. But that's where my mind wanders to when watching your videos. Very thought-provoking! Anyway, enough of that. Thanks again for sharing your explorations. Looking forward to your future videos.

What a great demo, and idea for a comparison of techniques! As for why you got those weird answers about companies Musk co-founded... You chose a very interesting question, because even humans with full knowledge of the subject would have a hard time answering :) For example, while Musk bought Tesla as a pre-existing company (and thus did not co-found it), part of his purchase agreement was that he could legally call himself a co-founder. So is he or isn't he a co-founder? Murky legal/marketing vs. normal understanding, polluted article/web knowledge set, etc.

Very well illustrated! Thanks

Thank you for your explanation of integrating KG to RAG

That explanation was awesome

great video thx

Thank you for this video! Really great. I'm also a bit new to DSPY, but am having a great time learning it. This is really the right way to explore, imo. You set up comparative tests and then look carefully at the results to think about what might be going on in order to find the best methodology. Yep. That's the way to do it. Some thoughts come to mind. 1) take the same code and question, and try it with different models for the embedding. What I've noticed is that the selection of models can have a significant influence on the outcomes. 2) perhaps try creating a validator function for when you take the data and convert it into English as a way to have the LLM double-check the results to make sure they are accurate. I've been doing that with a code generator I'm working on, and it seems pretty helpful. If the LLM determines the generated code does't match the requirements, then it recursively tries again until it gets it (I send the rationale of the failure back in on each pass to help it find its way -- up to five times max) 3) I'll be curious to see how much optimization influences the outcome! Anyway, fun to follow you on your journey. Thanks for posting!

Thanks for the awesome summary about raptor