Want to master Clean Architecture? Go here: bit.ly/3PupkOJ Want to unlock Modular Monoliths? Go here: bit.ly/3SXlzSt Join a community of 1000+ .NET developers: www.patreon.com/milanjovanovic
Thanks a lot Milan for this useful information. I haven't had any hands on experience with Postgres GIN indexes so far and this made me motivated to dive deep into the research.
use this pair with an ollama-HuggingFace local LLM model, and you will get a RAG/LLM (I am implementing it right now with aspire, it would be awesome to see your implementation if you do it). If postgres vector generation is not good enought (I doubt it), then I will still use it as a vector database and will use another embedding model generator (BI-Encoder). I started doing videos just to showcase my habilities (for job matters), and you Milan were a part that motivated me to do it.
@@MilanJovanovicTech (update) your video was an excellent groundbase, I decided to use and image of PostgresML (is an open project of postgresSQL with extensions and predefined software) which has machine learning capabilities (it uses pgvector too), it avoid to move the data to other services like ollama (less network overhead) and download the embeddings and LLM's models direct to database server (wich I will use only for semmantic search I load the info via Kafka as service bus), I am struggling a little bit but keep on improving to integrated it with net aspire, I will let you know new updates and if I release a video, if you want it give it a look can a be a good video in your channel (my channel is not for content creation purposes).
After adding a tsvector in Entity Framework, is there a way to automatically regenerate its values when updating an existing entity or creating a new one? Or do we need to manually implement a trigger for updating the tsvector? and also EF.Functions.PhraseToTsQuery does not work for partial word matching. Could you provide some guidance on this?
Hello, Great Video ! What about partial matching ? This full text search functionality works with full word matching strings not partial strings. Any idea how we could achieve that ?
Thanks for the video; I learned a lot of things. The only thing I don't understand in your first example is why you would run a search on non-indexed columns to begin with.
Thanks, Milan, for the video. Amazing content as usual!! What if I want Language invariant FTS? The column could have multiple languages and I want it invariant, what will be the changes to this solution?
Thanks for the informations Milan. It was really useful. How we can make a choose between Elasticsearch and PostgreSql for full-text search? Also Can we implement this feature using Marten?
| How we can make a choose between Elasticsearch and PostgreSql for full-text search? You choose based on your requirements and if you can implement what you need to build.
Can you please make a video of PostgreSQL partition with EF core? I tried, when I scaffolding using dotnet scaffold command it is retriving only the partitioned table, but it is supposed to be retrive partitioning table. I need your advice here.
That was a great video! I never heard abot lunr, I`m gonna check it! By the way, Milan. there is something like that using ef core with SQL Server. Full Text Search in SQL server is a pain!
can i enter two languages in to_tsvector i have the same column that can have both english and arabic text what is the best way to deal with case like this?
Thanks for great content! Does anyone know, MS SQL Server have the same features? As I saw, for native Full Text Search you can only setup one field per table for Full Text Search feature, but it will also search slowly if you use Contains method to search from the middle of the some word for example. But in the same time it will search much faster if you search from the start of each word.
@Milan Great videos Thanks for your work. Can you please also make a video or blog on AntiFrogeryTokken using Angular + .NET 8 Core to prevent CSRF attacks, use anti-forgery tokens
Hello! Yet its good to know that Postgres has text index - its nowhere near compared to Elastic engine in terms of API and functionality. Also, if you're using normalized data - gl with sharding. All in all, using relational db for searches seems off and weird. if your requirements for search aint high and something basic - then just go for Mongo, at least you wont be having problems when sharding.
Want to master Clean Architecture? Go here: bit.ly/3PupkOJ
Want to unlock Modular Monoliths? Go here: bit.ly/3SXlzSt
Join a community of 1000+ .NET developers: www.patreon.com/milanjovanovic
Thanks a lot Milan for this useful information. I haven't had any hands on experience with Postgres GIN indexes so far and this made me motivated to dive deep into the research.
My pleasure!
Awesome video. Postgres is incredibly powerful, and it seems to satisfy a lot of use cases for typical software solutions.
Postgres is incredibly versatile :)
Finally, this is how it's done!
Should I explore more heavy-duty options?
@@MilanJovanovicTech this is fine for me. If others request alternatives then you can consider it.
Thank you so much Milan. This was very useful.
Glad it was helpful!
Awesome video Milan. But in 4:57 how can you have the queries like that .I am using dotnet 9 and i don't know how to configure like that
OpenTelemetry takes care of that
@@MilanJovanovicTech can you give me some docs for that. Cause i found nothing to config like what you got in video. Thank you Milan
postgres also has the ILIKE that doesnt require to lower the columns
I think that's exposed via an EF.Functions.ILike
Awesome video, enjoyed it
Glad to hear it!
use this pair with an ollama-HuggingFace local LLM model, and you will get a RAG/LLM (I am implementing it right now with aspire, it would be awesome to see your implementation if you do it).
If postgres vector generation is not good enought (I doubt it), then I will still use it as a vector database and will use another embedding model generator (BI-Encoder).
I started doing videos just to showcase my habilities (for job matters), and you Milan were a part that motivated me to do it.
I'd love to see your implementation when you make a video. Let me know when (if) you release it.
@@MilanJovanovicTech (update) your video was an excellent groundbase, I decided to use and image of PostgresML (is an open project of postgresSQL with extensions and predefined software) which has machine learning capabilities (it uses pgvector too), it avoid to move the data to other services like ollama (less network overhead) and download the embeddings and LLM's models direct to database server (wich I will use only for semmantic search I load the info via Kafka as service bus), I am struggling a little bit but keep on improving to integrated it with net aspire, I will let you know new updates and if I release a video, if you want it give it a look can a be a good video in your channel (my channel is not for content creation purposes).
After adding a tsvector in Entity Framework, is there a way to automatically regenerate its values when updating an existing entity or creating a new one? Or do we need to manually implement a trigger for updating the tsvector? and also EF.Functions.PhraseToTsQuery does not work for partial word matching. Could you provide some guidance on this?
A tsvector column is automatically updated when the row is modified
Pretty cool, will most likely swap to postgres for my home project and try it out :D
I'm sure you'll see some nice results
Thanks, it's big nice work!
Glad you like it!
Hello Milan,
Awesome video. I wanted to ask if there's a way you can apply fuzziness to your search with the method you have used
You'd probably have to implement custom support for this in SQL, maybe via a function that you can call as EF.Function
Hello, Great Video ! What about partial matching ? This full text search functionality works with full word matching strings not partial strings. Any idea how we could achieve that ?
I think that web search covers that aspect
@MilanJovanovicTech I tried it but it didn't. Best I did until now is to used a combined search (vector + ILike function)
Thanks for the video; I learned a lot of things. The only thing I don't understand in your first example is why you would run a search on non-indexed columns to begin with.
Highlighting an inefficient approach
Hello.
Any news for the Cursor AI technology? You said you gonna upload in depth video
Next week!
Thanks, Milan, for the video. Amazing content as usual!!
What if I want Language invariant FTS? The column could have multiple languages and I want it invariant, what will be the changes to this solution?
Need to do some additional work in the database, let me see if I can find some resources
Very good.
What about if you want pagination. How do you deal with it using FTS in Postgres?
Skip/Take?
We sort by Rank, and apply basic pagination
Hello! Can you tell me which package needs to be installed for the studio to see the HasMethod() extension method? I still can't find it
HasMethod?
Hi, this video is a great help. How about for SQL Server database? ts vector is not supported in SQL Server. Thank you.
No, SQL would use its own implementation
Thx Milan!
Most welcome :)
Thanks for the informations Milan. It was really useful. How we can make a choose between Elasticsearch and PostgreSql for full-text search? Also Can we implement this feature using Marten?
| How we can make a choose between Elasticsearch and PostgreSql for full-text search?
You choose based on your requirements and if you can implement what you need to build.
@@MilanJovanovicTech Thanks Milan
Can you please make a video of PostgreSQL partition with EF core?
I tried, when I scaffolding using dotnet scaffold command it is retriving only the partitioned table, but it is supposed to be retrive partitioning table. I need your advice here.
Will check it out
If you have two tables of users and posts with a relationship one to many
How can you create the vector to use full text-search?
Has to be in one table for the index
Milan, does this work with Microsoft SQL Server?
This code - not exactly. But SQL Server also has full-text search, you'd just need to adapt the implementation.
nice, I like. what will happen when the user is not searching for exact terms but just parts of them. for example: mono instead of monolith?
Then it won't work so great
@@MilanJovanovicTech So perhaps a hybrid search would also work well in your use case.
What would a multi language system look like? How should be a config value?
It's a bit more complicated, need to do some additional setup in DB
That was a great video! I never heard abot lunr, I`m gonna check it! By the way, Milan. there is something like that using ef core with SQL Server. Full Text Search in SQL server is a pain!
Full writeup on Lunr here: www.milanjovanovic.tech/blog/how-i-implemented-full-text-search-on-my-website
can i enter two languages in to_tsvector i have the same column that can have both english and arabic text what is the best way to deal with case like this?
With a bit of added configuration, yes
@@MilanJovanovicTech my next question would be how? i didn't find any searching only any resources would be greatly appreciated, thanks.
Thanks for great content! Does anyone know, MS SQL Server have the same features? As I saw, for native Full Text Search you can only setup one field per table for Full Text Search feature, but it will also search slowly if you use Contains method to search from the middle of the some word for example. But in the same time it will search much faster if you search from the start of each word.
SQL Server supports full-text search. Not sure how you'd use it from EF Core, but there should be a function like EF.FreeText
very good video ;)
Thank you very much!
What's the client that you use to trace your calls?
That's the Aspire Dashboard
Is your VS theme available somewhere?
It's just ReSharper
@@MilanJovanovicTech Thanks!
Hi, thanks a lot, but how to do you search Vietnamese ?
Gotta add language support for viet terms
Thanks, but pg_trgm wouldn't be better to search because can look for parts of the word? Could you delve more deeply into this topic?
I can explore this in a future video, sure
Waiting for your .Net Api course when you are launching it
November
Can MSsql be used?
Yes, SQL Server supports this feature. Not sure how you'd use it from EF Core, but there should be a function like EF.FreeText
Then he would have to change the title of the video
@@mtsmithtube 🤣
Thank you awsome video.
Does sombody here know is there also something similar for MS SQL Server?
Of course: learn.microsoft.com/en-us/sql/relational-databases/search/full-text-search
@@MilanJovanovicTech Thanks 👍
@Milan Great videos Thanks for your work. Can you please also make a video or blog on AntiFrogeryTokken using Angular + .NET 8 Core to prevent CSRF attacks, use anti-forgery tokens
Perhaps
Hello!
Yet its good to know that Postgres has text index - its nowhere near compared to Elastic engine in terms of API and functionality.
Also, if you're using normalized data - gl with sharding.
All in all, using relational db for searches seems off and weird.
if your requirements for search aint high and something basic - then just go for Mongo, at least you wont be having problems when sharding.
I've only used Cursor
Then you have couple of people who use your system in france and everything falls apart. Not mentioning the Turkish "i" issue.
Solvable problems
First
Faster than me even 🔥
@@MilanJovanovicTech 😁😁
Good one,
in a world where data security is important this DOESN'T WORK on encrypted column.
Thanks for sharing.
Which DB/Service supports full-text search on encrypted columns?
Thank you so much Milan. This was very useful.
Glad it was helpful!!