@@WW_AI_Adventures actually my prof actually asked me to make like a search engine where you would make graphs of research papers from pubmed and using a graph convolutional neural network ,when the user types a keyword it would get the most similar paper to it from the graph So really generating results based on that is what my project is on ,still figuring out the details but yeah this helps If you know something else that could help out do suggest
Ah graph rag sounds perfect for this. Graph Convolutional networks sound like they might be overkill though, but I'm no expert, I'd probably try with just graph-rag first (with a cheaper LLM than openAI though at first)
Hi @WW_AI_Adventures, thank you for sharing and presenting such rich content so nicely. I was able to run the example code successfully, but I noticed that some of the `.parquet` files had missing columns like `description_embedding`, `rank`, and others. Do you have any idea what might be causing this?
@@WW_AI_Adventures thanks for the prompt response! It seems the error occurs when microsoft_to_neo4j.py tries to read the columns 'name' and 'description_embedding' from create_final_entities.parquet, and 'rank' from create_final_relationships.parquet, as these columns are missing from these files. However, the rest of the columns are present. Don't know what is causing it configuration, models or something else.
@MuhammadSadiq-k3q it could be that Microsoft has changed their format for the parquet files. Try inspecting the files manually to see if the names have changed
Did you use a different llm for the native RAG and the local global RAG? Wouldn't that make the benchmark biased, results will be better for the local global RAG with GPT-4 running for it.
Hiya! Thanks for commenting - Yes I used the same LLM for both. GPT-4o. To be honest this isn't a strict benchmark, just my exploration of the two together. Naïve RAG will simply never be able to include all of the text data from a large corpus in its context window. GraphRAG gets around this by precomputing summaries ahead of time - so it will always have an advantage, at the cost of this ahead of time summarisation which may not be possible for an incredibly large corpus!
@@dusktildawn-ue8jq I'm not a big fan of books, but sentdex on TH-cam and his site realpython.com/ are really good and I have used them before. He has good tutorials in most domains as well
@@jackbauer322 thanks for the suggestion. LightRag does look good. However, if you want a true global summary of all of your text, then I don't know if it will be able to do as good a job!
I love it! The video is full of value, as well as your other content. Subscribed, definitely want to see more of this :)
@@franknillard Thanks a bunch!
Thank you for sharing such rich content. It's quite easy to connect with. . The entire world needs to come see what is being offered here about Gen AI
Yo you are a saviour needed this for my bachelor's thesis project
@@mathew5880 xD haha so glad I could help. What's your thesis on?
@@WW_AI_Adventures actually my prof actually asked me to make like a search engine where you would make graphs of research papers from pubmed and using a graph convolutional neural network ,when the user types a keyword it would get the most similar paper to it from the graph
So really generating results based on that is what my project is on ,still figuring out the details but yeah this helps
If you know something else that could help out do suggest
Working on something similar for my thesis as well.
Ah graph rag sounds perfect for this. Graph Convolutional networks sound like they might be overkill though, but I'm no expert, I'd probably try with just graph-rag first (with a cheaper LLM than openAI though at first)
@@WW_AI_Adventures yeah I guess your probably right but will try out both and see how it goes
Really high quality content man!
@@peterroshdy1269 thanks a bunch!
What kind of stuff are you interested in seeing in this space?
Just more content like that with demos to cool ai applications and use cases
@@peterroshdy1269 happy to oblige!
Hi @WW_AI_Adventures, thank you for sharing and presenting such rich content so nicely. I was able to run the example code successfully, but I noticed that some of the `.parquet` files had missing columns like `description_embedding`, `rank`, and others. Do you have any idea what might be causing this?
Hiya! Thanks for the feedback. Do you have an error you can share?
@@WW_AI_Adventures thanks for the prompt response! It seems the error occurs when microsoft_to_neo4j.py tries to read the columns 'name' and 'description_embedding' from create_final_entities.parquet, and 'rank' from create_final_relationships.parquet, as these columns are missing from these files. However, the rest of the columns are present. Don't know what is causing it configuration, models or something else.
@MuhammadSadiq-k3q it could be that Microsoft has changed their format for the parquet files. Try inspecting the files manually to see if the names have changed
Well done mate.
Thanks Dong 🙏
Great demo
@@Maskra_ thanks 🙏🙏 what did you like about it?
Did you use a different llm for the native RAG and the local global RAG?
Wouldn't that make the benchmark biased, results will be better for the local global RAG with GPT-4 running for it.
Hiya!
Thanks for commenting - Yes I used the same LLM for both. GPT-4o.
To be honest this isn't a strict benchmark, just my exploration of the two together. Naïve RAG will simply never be able to include all of the text data from a large corpus in its context window.
GraphRAG gets around this by precomputing summaries ahead of time - so it will always have an advantage, at the cost of this ahead of time summarisation which may not be possible for an incredibly large corpus!
Are you wearing Indonesian Batik? Great short lesson, thank you for sharing!
@@treflatface Thanks! I don't think so but I just looked this up and I love the colours of Batik.
Hey, can you recommend a book for learning Python? I'm new to it
@@dusktildawn-ue8jq I'm not a big fan of books, but sentdex on TH-cam and his site realpython.com/ are really good and I have used them before. He has good tutorials in most domains as well
@@WW_AI_Adventures thanks
graphrag is NOT be used because it consumes too many tokens , use lightrag instead, please make a video with lightrag
@@jackbauer322 thanks for the suggestion. LightRag does look good. However, if you want a true global summary of all of your text, then I don't know if it will be able to do as good a job!