Code snippets and bit of explanation shweta-lodha.medium.com/step-by-step-tutorial-to-integrate-openai-with-azure-cognitive-search-vector-search-4d75b400675c
Just loving your contents! There are a lot of grey areas onto the LLMs and how they work if we integrate Azure services, especially the vector DB part. Using FAISS, Chroma, etc are good for POCs, that too performed internally.... but something which gets accepted in enterprise level is the one which you are talking about in this video!! Thank you
Thank you Shweta, really helpful for me to understand how the Cognitive search works. I have another question, wouldn't it be easier to use Azure openAI instead of OpenAI as the chat language model towards the end? I'll duplicate the code on my end. Thank you so much for sharing this with us.
Hi Shweta, your content is very clear and useful! How might we get the full .ipynb source code so we can try out the solution ourselves? For instance, generate_embeddings function is not shown.Thank you for all that you do!
This implementation varies based on your choice - whether you are going with OpenAI embedding, hugging face or any other. Purpose of this function is to generate the embedding for the given text
It doesn’t actually matter from DB side, whether you are taking single or multiple docs. It only differs at a point where you are reading your docs. If you have multiple docs, you can load using some directory loader
I am following as per the tutorial but getting this error while creating the Index: vector_search_configuration is not a known attribute of class and will be ignored
This line is not fully displayed in the video could you kindly provide the rest of the line, please? SearchField(name="embedding", type=SearchFieldDataType.Collection (SearchFieldDataType.Single), searchable=True, vector_search) Thanks
I don't have the complete code, but I hope this snippet would work: fields = [ SimpleField(name="documentId", type=SearchFieldDataType.String, filterable=True, sortable=True, key=True), SearchableField(name="content", type=SearchFieldDataType.String), SearchField(name="embedding", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions = 1536, vector_search_configuration ="my-vector-config") ] index = SearchIndex( name=index_name, fields=fields, vector_search=vector_search )
Code snippets and bit of explanation shweta-lodha.medium.com/step-by-step-tutorial-to-integrate-openai-with-azure-cognitive-search-vector-search-4d75b400675c
Very straight to the point. Just the video i needed to connect many isolated concepts! Thank you very much. !
Glad you find it useful. Stay tuned for more such bites 👍
Just loving your contents! There are a lot of grey areas onto the LLMs and how they work if we integrate Azure services, especially the vector DB part. Using FAISS, Chroma, etc are good for POCs, that too performed internally.... but something which gets accepted in enterprise level is the one which you are talking about in this video!!
Thank you
Thank you 😊
I could not find "generate_embeddings" function definition.
Thank you Shweta, really helpful for me to understand how the Cognitive search works. I have another question, wouldn't it be easier to use Azure openAI instead of OpenAI as the chat language model towards the end? I'll duplicate the code on my end. Thank you so much for sharing this with us.
thanks this is really good where is generate_embeddings function located and which embedding model are you using? share the notebook?
Thank u!!!!, your videos have helped me a lot.
I'm so glad!
Hi Thank you for the video ,i am getting httpresponseerror invalid parameter can u help me with this?
Thanks Shwetha
Hi Shweta, your content is very clear and useful! How might we get the full .ipynb source code so we can try out the solution ourselves? For instance, generate_embeddings function is not shown.Thank you for all that you do!
It is there on my Medium blog. Medium link in present on my profile page
Many thanks@@shweta-lodha
@@shweta-lodha medium member only
This is so good, thank you so much
Glad to know that 🥰
Hi! Can you please share the notebook that you used in the video? The medium site is for members only.
That is the only option I have today. Sorry for that
What is generate_embeddings ?, not there in your blog
This implementation varies based on your choice - whether you are going with OpenAI embedding, hugging face or any other. Purpose of this function is to generate the embedding for the given text
Hi, how would you suggest to implement this solution with multiple documents?
It doesn’t actually matter from DB side, whether you are taking single or multiple docs. It only differs at a point where you are reading your docs. If you have multiple docs, you can load using some directory loader
Hi @shweta lodha, can we rename the index name in azure cognitive search.
You can’t
I am following as per the tutorial but getting this error while creating the Index:
vector_search_configuration is not a known attribute of class and will be ignored
Are you sure, it’s an error and not a warning?
In production, how do you deploy this python code ?
You can create a web app and do it
can we use pdf files in the sameway?
No, reader would be different
thanks for the video.
You're welcome
Thanks 🙏🏼 to you
Always welcome
This line is not fully displayed in the video could you kindly provide the rest of the line, please?
SearchField(name="embedding", type=SearchFieldDataType.Collection (SearchFieldDataType.Single), searchable=True, vector_search)
Thanks
as well as generate_embedding function
I don't have the complete code, but I hope this snippet would work:
fields = [
SimpleField(name="documentId", type=SearchFieldDataType.String, filterable=True, sortable=True, key=True),
SearchableField(name="content", type=SearchFieldDataType.String),
SearchField(name="embedding", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions = 1536, vector_search_configuration ="my-vector-config")
]
index = SearchIndex(
name=index_name,
fields=fields,
vector_search=vector_search
)
response = openai.Embedding.create(
input="text", engine="text-embedding-ada-002")
embeddings = response['data'][0]['embedding']
Thanks!@@shweta-lodha