This is an awesome video. I was able to incorporate a bunch of pdfs into weaviate and make queries, but I wanted to modify the chatbot part of the project a bit. It seems like I can ask a question and get a response, but what if I want to further the conversation and ask another question based on the previous response. Basically, I want the chatbot to remember history so I can make a full conversation as opposed to a question/response. Is there an easy way to implement this because I can imagine a scenario where not only does it have to remember the previous context, but it also has to rerun a similarity search in weaviate.
Hey Joe! Happy to hear the video was useful! It is definitely possible to add chat history to your query. Langchain has some great documentation about this: python.langchain.com/docs/use_cases/question_answering/how_to/chat_vector_db. If you would be interested I might create a tutorial about this :)
Hey Michael! You can use many different LLMs using Langchain, such as the ones available via HuggingFace. On this page you can find an overview of all LLMs integrated in Langchain: python.langchain.com/docs/integrations/llms/ I hope this helps 😁
Very helpful, thanks. Just curious, but what if I wanted instead of the reply "I don't know" to continue with the usual results from ChatGPT ... if the answer is not in my data, then something more general via the usual gpt model would be shown ?
Hello Lore thanks for video its very helpful, i have a question to you What is the best method to upload a large JSON dataset with over 12,000 entries, each containing 'title', 'description', and 'author' keys, where the 'description' text in each entry averages around 1000 characters, to Weaviate?
Great video However, I have found some inconsistencies between the code you are shoeing and the file linked on your description I have managed to work it out, thought it is something to note. Looking forwards to see more of your videos !
Very clear explanation provided. Thank you Lore. But I have one question. Can we use custom embeddings instead of openai embeddings? Instead of using vectorizers like text2vec-open ai hugging face transformers etc.? If so how to add text into vector store based on these embeddings?
Using the Langchain framework you can also access the Cohore embeddings models. I would advise you to take a look at their documentation: python.langchain.com/docs/modules/data_connection/text_embedding/ 😊
Hello Lore thanks for video again. I have vectorstore like that "vectorstore = Weaviate(client, "Techs","description", attributes = ["url","author","title","path"])" and i have search in my documents its find 2 similarty data and ai combine them and answer my question So have can i these datas path below the answer need to give all finded documents path in there In this example its find 2 document and must be give 2 path url from there for example : Answer:.... Soruce 1 : {path 1} Source 2 : {path 2}
Thank you for the video, it was very informative. I have a question: I want to upload a very long text to Weaviate. I've installed Weaviate via Docker and wrote the PHP code to post via the API. However, when I try to post a long text, the Weaviate Docker container crashes. Is there a limit for the text size that can be sent to Weaviate, or is there a specific approach I should follow?
Hey Erdem! Glad to hear you liked the video 😁 Regarding your question, I would advise you to split your text into smaller chunks. In the video, I used chunk_size=1000 but you can lower this if you want. I hope this resolves the issue!
How do you need to change the code when connecting to a local Weaviate docker image rather than than the cloud sandbox (which is valid only 14 days) ? Secondly, what needs to be changed when I want my own vector database to sit on top of the GPT-3 LLM, as an enhancement rather than a replacement?
Hey! Thanks for your comment! I currently don't have a GitHub repository for the code but you can find most of the code in my Medium article: medium.com/p/78ecdbe383c8#c4d3-df9225f3246. I hope this helps! :)
@lorevanoudenhove4946 Running the code at cell in 11:17 part, I get this error: {'error': [{'message': 'update vector: unmarshal response body: json: invalid number literal, trying to unmarshal "\\"rate_limit_exceeded\\"" into Number'}]}
Wonderful Lore, can you make a video on how to use the weaviate instance locally using docker?
Super interesting video, very informative, want to learn more creating chatbots!!
Great overview thank you so much for putting this together! very helpful!
Great video!
Just a suggestion:
It would help if you could share the ipynb files (e.g. thru collab) from your tutorials. :) Most TH-camrs do it. :)
Thank you so much for the feedback! I'll upload the notebooks soon 🙂
You can find my Google Collab and the data used in the tutorial here: drive.google.com/drive/folders/1n-cz65obR2gI4uyYoHdEa3ts99J965mr?usp=sharing 😊
This is an awesome video. I was able to incorporate a bunch of pdfs into weaviate and make queries, but I wanted to modify the chatbot part of the project a bit. It seems like I can ask a question and get a response, but what if I want to further the conversation and ask another question based on the previous response. Basically, I want the chatbot to remember history so I can make a full conversation as opposed to a question/response. Is there an easy way to implement this because I can imagine a scenario where not only does it have to remember the previous context, but it also has to rerun a similarity search in weaviate.
Hey Joe! Happy to hear the video was useful! It is definitely possible to add chat history to your query. Langchain has some great documentation about this: python.langchain.com/docs/use_cases/question_answering/how_to/chat_vector_db. If you would be interested I might create a tutorial about this :)
Excllent and detailed information good job!
What open source LLMs can be used with this design?
Hey Michael! You can use many different LLMs using Langchain, such as the ones available via HuggingFace. On this page you can find an overview of all LLMs integrated in Langchain: python.langchain.com/docs/integrations/llms/ I hope this helps 😁
Very helpful, thanks. Just curious, but what if I wanted instead of the reply "I don't know" to continue with the usual results from ChatGPT ... if the answer is not in my data, then something more general via the usual gpt model would be shown ?
you'd need to change the langchain qa class method prompt template. Or use a different chain.
Awesome video, Lore!
Thanks! 😁
Hello Lore thanks for video its very helpful, i have a question to you
What is the best method to upload a large JSON dataset with over 12,000 entries, each containing 'title', 'description', and 'author' keys, where the 'description' text in each entry averages around 1000 characters, to Weaviate?
Great video
However, I have found some inconsistencies between the code you are shoeing and the file linked on your description
I have managed to work it out, thought it is something to note.
Looking forwards to see more of your videos !
Thank you for your feedback! Highly appreciated!
Very clear explanation provided. Thank you Lore. But I have one question. Can we use custom embeddings instead of openai embeddings? Instead of using vectorizers like text2vec-open ai hugging face transformers etc.? If so how to add text into vector store based on these embeddings?
Using the Langchain framework you can also access the Cohore embeddings models. I would advise you to take a look at their documentation: python.langchain.com/docs/modules/data_connection/text_embedding/ 😊
Hello Lore thanks for video again.
I have vectorstore like that "vectorstore = Weaviate(client, "Techs","description", attributes = ["url","author","title","path"])"
and i have search in my documents its find 2 similarty data and ai combine them and answer my question
So have can i these datas path below the answer need to give all finded documents path in there
In this example its find 2 document and must be give 2 path url from there
for example :
Answer:....
Soruce 1 : {path 1}
Source 2 : {path 2}
Thank you for the video, it was very informative. I have a question: I want to upload a very long text to Weaviate. I've installed Weaviate via Docker and wrote the PHP code to post via the API. However, when I try to post a long text, the Weaviate Docker container crashes. Is there a limit for the text size that can be sent to Weaviate, or is there a specific approach I should follow?
Hey Erdem! Glad to hear you liked the video 😁 Regarding your question, I would advise you to split your text into smaller chunks. In the video, I used chunk_size=1000 but you can lower this if you want. I hope this resolves the issue!
How do you need to change the code when connecting to a local Weaviate docker image rather than than the cloud sandbox (which is valid only 14 days) ?
Secondly, what needs to be changed when I want my own vector database to sit on top of the GPT-3 LLM, as an enhancement rather than a replacement?
Please how can i get api of weaviate ???
You should be able to retrieve the api of your weaviate vector database via the Weaviate console, if you click on details.
github?
Hey! Thanks for your comment! I currently don't have a GitHub repository for the code but you can find most of the code in my Medium article: medium.com/p/78ecdbe383c8#c4d3-df9225f3246. I hope this helps! :)
@@lorevanoudenhove hey thanks for your reply
@lorevanoudenhove4946
Running the code at cell in 11:17 part, I get this error:
{'error': [{'message': 'update vector: unmarshal response body: json: invalid number literal, trying to unmarshal "\\"rate_limit_exceeded\\"" into Number'}]}