Using Langchain and Open Source Vector DB Chroma for Semantic Search with OpenAI's LLM | Code
ฝัง
- เผยแพร่เมื่อ 23 ก.ย. 2024
- Discover the power of LangChain, Chroma DB, and OpenAI's Large Language Models (LLM) in this step-by-step guide. Dive into semantic search capabilities using an open-source vector database, Chroma DB.
AI Demos: www.aidemos.com/
AIDemos.com is your go-to directory for video demos of the latest AI tools. AI Demos's goal is to educate and inform about the possibilities of AI.
Code and Explanation: blog.futuresma...
Building a Document-based Question Answering System with LangChain, Pinecone, and LLMs like GPT-4.
• Building a Document-ba...
Revolutionizing Search: How to Combine Semantic Search with GPT-3 Q&A
• Revolutionizing Search...
GPT-4 API Tutorial: • GPT-4 API: Real-World ...
ChatGPT API Tutorial: • Using OpenAI's ChatGPT...
Building a GPT-4 Chatbot using ChatGPT API and Streamlit Chat
• Building a GPT-4 Chatb...
🚀 Top Rated Plus Data Science Freelancer with 8+ years of experience, specializing in NLP and Back-End Development. Founder of FutureSmart AI, helping clients build custom AI NLP applications using cutting-edge models and techniques. Former Lead Data Scientist at Oracle, primarily working on NLP and MLOps.
💡 As a Freelancer on Upwork, I have earned over $100K with a 100% Job Success rate, creating custom NLP solutions using GPT-3, ChatGPT, GPT-4, and Hugging Face Transformers. Expert in building applications involving semantic search, sentence transformers, vector databases, and more.
This video was very helpful. Good explanation. Appreciate your time and effort in putting it togather.
Dear Pradip,
I have been following your video tutorials for the past three weeks, and I have found them to be very informative and comprehensive. However, I am having some difficulty understanding how to create a custom agent in Langchain with class and class decorators and LLMchain.
I would be grateful if you could create a video tutorial that specifically addresses this topic. I believe that this would be a valuable resource for many people who are interested in learning how to use Langchain to create custom agents.
Sure. I will explore it and cover
Thanks for the great video . TBH , I was struggling with this from 2 days .. and your video helped me .
Glad it helped!
Thanks for the video. This was very useful. Information was pretty clear for the beginners too.
Glad to hear it!
Great work Pradip
Thanks for Sharing!
My pleasure!
Can you make a video where we persist the Chroma db on cloud and show us how to add new files, delete new files, embed these files and ask questions with our new db? Would really really really appreciate it! Thank you in advance
I will try
Brilliant work !!!!
Thank you! Cheers!
Hello sir, my name is Nehat. I am from India. I am full stack developer and very much interested in AI. I have seen your videos and trust me I learn a lot from those videos. You have an amazing quality of teaching and making things easy to understand. I would be so many if you teach me those skills and show me the new world of AI. Please do let me know if there is any paid courses I can take. Thank you!
Hi Nehat, Thanks. I dont have any paid course. I mostly focused on my freelancing work and share my learning here on youtube. IF you have any doubts you can ask here.
cannot import name 'Document' from 'langchain'
getting this error
also i m facing some problem is it possible to connect you.
Very nicely explained!
Can we do this job without OpenAI API key? can we do it locally on our own computer?
For open ai you will require key. if you want to only test semantic search and see relevant docs you dont need key
@@FutureSmartAI Thanks Pradip- Just sent you an email, please see.
@@FutureSmartAI this api key free or paid ?
@@riyadhmollik+1
Hey Pradip, good video.
Do you have any plans for similar videos on LLAMA2 and vector dbs?
Yes I will be
I would like to know how to HOST this CUSTOM model in cloud... please? anyone?
Could you share your 「Chroma DB with Langchain.ipynb 」file?
blog.futuresmart.ai/using-langchain-and-open-source-vector-db-chroma-for-semantic-search-with-openais-llm
is there any difference in embeddings if we use OpenAi embedding rather than langchain ?
langchain dont have their own embedings. we can use open ai or other open source embding like sentence transformers
Hi @Pradip, what if I need a summarised answer for my query, for example, what is the average health score of all my pumps? Health score is just a number in all my documents. So will my model get all that data or it will go out of context limit while fetching the data ?
Thank you Pradip for this great tutorial! 👏It worked for my dataset.🎉
Do you have an alternative code to create chroma_db from:
either: splitting a Pandas DataFrame's column
or: splitting a PySpark SQL DataFrame's column of table content ?
instead of: docs = split_docs(documents) and db = Chroma.from_documents(docs, embeddings)
My dataset is too large and creating thousands of .txt files is not sustainable.
Why dont you use Langchain SQL Agents
Thanks for your video. I have tried to implement the same on my side, however the text pieces returned by similarity_search for the query look unrelevant. I'm using all-MiniLM-L6-v2 model for the embedding and the following settings of the text_splitter - chunk_size=1024,chunk_overlap=20. What can be the reason of the poor seimilarity search results? P.S. input files are not in English language. It is in Uzbek. Can it be also the reason?
how long will it take to embed and store a large dataset of around 50 MB in Chroma
Thanks for your helpful video!
I am loading python code.
Could you please suggest the best vector semantic similarity search here to get most relevant top k results?
you mean best embeding model?
How can we do vector search for dynamic data, can nodes update automatically
as we get new data we need to add more docs to existing index or collection
Hi,I have a json file with content and metadata. How to split content into chunks in that case.
I'm looking for the same
Have you got any leads?
@@koushiksherugar8680 yes,you can check json document loader integration
in langchain documentation
Hey i am facing an issue which is DirectoryLoader runs for either very long time or crashes. :( how can I solve this?
hi pradip
In the past video
You used streamlit, langchain, and pinecone.
I want to use ChomaDB instead of pinecone
I don’t know what I should do.
Can you give me some tips?
you should be able to do it yourself. try combining code from both of these videos. Even you can ask chatGPT to do that
is it possible to dockerize this code with fastapi?
thanks
You're welcome!
If different models were used for generating embedding would the performance of semantic search be effected ?
Yes. www.sbert.net/docs/pretrained_models.html
I have one doubt Does Azure cognitive service provides vector db service too or not? Please help me
They have something Azure Cognitive Search but you can always use other vector databases.
learn.microsoft.com/en-us/semantic-kernel/memories/vector-db
Can you guide me how can I connect LLM and pine Cone and open Ai embedding to connect with mongo database and create chat bot ?
can i upload more than 1 pdf file when i integrate with streamlit
Great video! But if I get one dollar for every "you know" you said, I wouldn't need to worry about learning ChromaDB lol jk.
Deal!
Does this video say how you can use a local model with the vector db?
Hi Vector DB is local not LLM
sir i want to split json file how can I do it?