This is a high-level system design of how to build a simple semantic Search engine for Q&A data References: 1. ElasticSearch+ BERT: www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch 2. FAISS: engineering.fb.com/data-infrastructure/faiss-a-library-for-efficient-similarity-search/ 3. DiskANN: www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/
Very nice. Been following your series for a while during quarantine from Toronto while I work on some complex and stochastic analysis, these videos are VERY good and some of the best applied videos I've seen yet on youtube - some theorists like myself certainly really appreciate this content, thank you
We are working on doing a couple of public live sessions on the code-walkthrough of this system(ElasticSearch+BERT) using publicly available datasets. Stay tuned! We have just made an announcement for these public live sessions here: th-cam.com/video/6q_ez_Cfcik/w-d-xo.html
Thanks AppliedAi for these informative sessions. Would be really helpful if you can come up with some good Cost structure & Curated course for your alumni Students who ended up doing all things till self driving cars. I mean special course that include all newly added material after self driving cars with Tf sessions.
Thank you Jeetendra. Thank you for the suggestion. We are brainstorming a few ideas on the same lines. We will let you know as soon as we have something in place.
Why not use KNN which runs with log(n) complexity? To further optimize, you can use a funnel approach with a simple linear model which focuses on recall and then apply BERT on the smaller output set of the funnel.
The Kd-tree based approach for k-NN which has a log(n) time complexity does not work well when you have high dimensional data. When applying the first linear model, what feature would you employ? How do you get all similar questions using a linear model which is often used for classification and regression.
sir i am thinking to build a project name - "Automatic sketch of a suspect or criminal using genetic algorithm " based on input given by the user such as eye color,face structure,etc. But don't know from where to start. Sir Please give me some suggestion like....which topics should i study more for this ?.....or which approach should i take to build this ?.....how can i build the optimization function ?.........i will be really thankful to you if you show some light on this. Thank you in advance sir.
Hi, sir I tried using elastic search + Bert but sometimes elastic search fails and when the elastic search fails bert is not able to extract the correct answer so is there any other idea?
Could you be more specific on what you means my ES "fails"? Does that mean ES faila to find relevant questions even when BERT vectors are similar? Are the BERT vectors themselves not similar? You will need debug and understand the source of the problem. Note that BERT is not always perfect. Like all ML models it will make errors. We just have to fine-tune these models to make them work better on our problem at hand.
Whoosh is a Python package for search. We have not used it extensively. Hence, can't comment in depth about it. But, elastic search is much more mature and is used extensively in large scale production environments at scale at many larger internet companies.
This is a high-level system design of how to build a simple semantic Search engine for Q&A data
References:
1. ElasticSearch+ BERT: www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch
2. FAISS: engineering.fb.com/data-infrastructure/faiss-a-library-for-efficient-similarity-search/
3. DiskANN: www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/
Very nice. Been following your series for a while during quarantine from Toronto while I work on some complex and stochastic analysis, these videos are VERY good and some of the best applied videos I've seen yet on youtube - some theorists like myself certainly really appreciate this content, thank you
We are working on doing a couple of public live sessions on the code-walkthrough of this system(ElasticSearch+BERT) using publicly available datasets. Stay tuned! We have just made an announcement for these public live sessions here: th-cam.com/video/6q_ez_Cfcik/w-d-xo.html
eagerly waiting sir..:)
Please make it publicly available... sir !!
As mentioned above, these will be public sessions.
Awesome Sir! It will help us a lot. Thank You!
@@AppliedAICourse Thank you
Amazing video! Looking forward to see more of these experience-driven videos!!!!!🔥🔥
Thanks AppliedAi for these informative sessions. Would be really helpful if you can come up with some good Cost structure & Curated course for your alumni Students who ended up doing all things till self driving cars. I mean special course that include all newly added material after self driving cars with Tf sessions.
Thank you Jeetendra. Thank you for the suggestion. We are brainstorming a few ideas on the same lines. We will let you know as soon as we have something in place.
Why not use KNN which runs with log(n) complexity? To further optimize, you can use a funnel approach with a simple linear model which focuses on recall and then apply BERT on the smaller output set of the funnel.
The Kd-tree based approach for k-NN which has a log(n) time complexity does not work well when you have high dimensional data. When applying the first linear model, what feature would you employ? How do you get all similar questions using a linear model which is often used for classification and regression.
sir i am thinking to build a project name - "Automatic sketch of a suspect or criminal using genetic algorithm " based on input given by the user such as eye color,face structure,etc. But don't know from where to start. Sir Please give me some suggestion like....which topics should i study more for this ?.....or which approach should i take to build this ?.....how can i build the optimization function ?.........i will be really thankful to you if you show some light on this.
Thank you in advance sir.
Hi, sir I tried using elastic search + Bert but sometimes elastic search fails and when the elastic search fails bert is not able to extract the correct answer so is there any other idea?
Could you be more specific on what you means my ES "fails"? Does that mean ES faila to find relevant questions even when BERT vectors are similar? Are the BERT vectors themselves not similar? You will need debug and understand the source of the problem. Note that BERT is not always perfect. Like all ML models it will make errors. We just have to fine-tune these models to make them work better on our problem at hand.
Sir is whoosh better than elastic search
Whoosh is a Python package for search. We have not used it extensively. Hence, can't comment in depth about it. But, elastic search is much more mature and is used extensively in large scale production environments at scale at many larger internet companies.