Getting Started with ChromaDB - Lowest Learning Curve Vector Database For Semantic Search
ฝัง
- เผยแพร่เมื่อ 6 ต.ค. 2023
- Start testing out semantic searches on a vector database within minutes. Everything works locally and is free. Don't need to sign up for a cloud vector database account or learn Langchain first.
Buy Me a Coffee: www.buymeacoffee.com/johnnycode
Get the code: github.com/johnnycode8/chroma...
ChromaDB Playlist: • ChromaDB Vector Databa... - วิทยาศาสตร์และเทคโนโลยี
Please help me out with a subscribe if this video helped you 😀 AND I would love to know what you're doing in your ChromaDB project.
Check out my new video: How to vectorize 33K embeddings to ChromaDB in 3 minutes: th-cam.com/video/7FvdwwvqrD4/w-d-xo.html
Hi Johnny, my names johnny nice to meet u lol.
Now I can put chroma db in my resume. Thanks you creating such a crisp and straightforward tutorial.
Short and Sweet! Thank you very much.
Concise and precise are what you did here.
Such an amazing video man, Thanks for this valuable Knowledge
Perfect, all i was interested in was persisting the database. Thanks for adding that at the end
Excellent tutorial. Thank You!
You have a new subscriber. This video was very timely for my proof of concept.
Man, that was amazing. Thank you so much.
great demo and concise
Awesome demo
This is a perfect tutorial, thanks!
Excellent tutorial.
Thanks for the excellent explanation
Good Intro / Review. Thank You from a NEW Subscriber !!!
Very helpful! thank you
Amazing video thank you so much 🙂❤
This helped a lot! 👍
Thank you my guy 🙂
Very interesting !
Rad. Thanks.
The POC use case I want to try is to query vehicle parts based on a user description which can be somewhat vague and keyword search is not ideal most of the time. I'd also want to do search-as-you-type with a vector db. My understanding is that there are embedding models trained on vehicle parts and seeing how easy it so specify a new embedding model although you have to remember the collection, I hope I can prove my POC. In addition, I want to use the persisted collection for other ideas as well. Thanks for making this video it really helps.
Thanks for sharing, Ken. Your app would have came in handy for me when I was searching for what turned out to be a "cap" or "flute" oil filter wrench :D
Thanks!
Thank you for the support!!!!😀😀😀
thank you
Thank you so much for an easy-to-follow, practical example!
I’m curious about something… at one point in the video you note illustrate that the default embeddings are returning something unrelated (sesame ball) as the #1 choice. Your solution is to swap it out for another embedding provider.
But how would you go about digging in here and debugging further?
Unfortunately, when it comes to the search results, we are at the mercy of how 'smart' the embedding model is. The term 'sesame ball' is a translation and unique to this restaurant's menu, so I wouldn't expect models to know the meaning of the term. Somehow, its vector representation is close to the word shrimp for the first model, but we don't have a way to 'debug' it. Here are some things that we have control of:
1. Changing the amount of text (short phrase, sentences, paragraphs, vs pages) per embedding. The more that is packed into one embedding, the harder it is for the model to be accurate.
2. Switch distance function to another one like Cosine Similarity: docs.trychroma.com/usage-guide#changing-the-distance-function
3. Switch to a more powerful, possibly paid, model, see listed of model and the section on Custom Embedding Functions: docs.trychroma.com/embeddings
4. Fine tune a model to understand the terms used by your organization.
@@johnnycode Awesome, thanks for the incredibly thorough and helpful answer!
@johnnycode, Is there any idea how to manage this error that occurs when I try to load previously saved chromadb file, e.g. "vectordb": InvalidDimensionException: Embedding dimension 384 does not match collection dimensionality 768?
You must use the same Embedding Function that you used to create that database. Embedding Functions convert text to a matrix of numbers and different Embedding Functions use different dimensions, so you can't use them interchangeably.
what do you do when all-MiniLM-L6-v2 is not very good at judging whats similar? it gets it wrong a lot!
Here are a few suggestions, hope this helps:
1. Are you embedding a short phrase, a few sentences, paragraphs, or pages? The more that is packed into 1 embedding, the harder it is for the model to be accurate.
2. Try switching the distance function to another one like Cosine Similarity: docs.trychroma.com/usage-guide#changing-the-distance-function
3. Switch to a more powerful, possibly paid, model, see listed of model and the section on Custom Embedding Functions: docs.trychroma.com/embeddings
Hi there I have a csv file with 150 rows. I have created collection added the document to the collection. when i query the collection the document field is giving me None. ids field gives the correct id but document field is none. What should I do. Is there any size for the field returned by document
If you use my code and the CSV provided in my GitHub repo (github.com/johnnycode8/chromadb_quickstart), does the document field show up? If the document field does show up, then you should check your loading logic. If the document field does not show up, check the part of the video that talks about using "include":
collection.query(...
include=[ "documents" ]
)
I don't see a publish document field size limit. Are you loading extremely long docs?
I have a CSV file with 150 rows. I have created a collection and added my document to it. when i query it the document field always contains none but the id field give me the correct id but document field is always none. Is there any size constraint for the document field? How should I solve it?
If you use my code and the CSV provided in my GitHub repo (github.com/johnnycode8/chromadb_quickstart), does the document field show up? If the document field does show up, then you should check your loading logic. If the document field does not show up, check the part of the video that talks about using "include":
collection.query(...
include=[ "documents" ]
)
I don't see a publish document field size limit. Are you loading extremely long docs?
@@johnnycodeGot it! the problem was because of passing multiple columns from a row in csv file.
Have you come across this warning: Add of existing embedding ID: 1
Add of existing embedding ID: 2
...till all ids
I am just querying the database only but I am getting this warning also.
I think you should create a fresh database and collection and try again. If you had tried to insert records into the same IDs, it causes weird issues.
@@johnnycode Okay. I deleted the collection and did it again.... It worked Thanks!!!
How do the models know that shrimp and prawn is the same thing? Like how did the first model not get all 5 dishes, but the second model found all 5 dishes?
The models are trained to understand language like ChatGPT’s models. The second model in the video is a larger “smarter” model, so it performs better than the 1st. The disadvantages of a larger model is that it takes up more storage space, uses more computing power and memory, and is slower than a smaller model.
Where user defind db space we can create
Sorry, I don’t understand your question.
@@johnnycode where it will take the memory if it is default database, user can manually create the db and assign that path?
You can run ChromaDB in-memory if you are prototyping and don't need to retain the data. However, if you want to retain the data, use persistence mode: client = chromadb.PersistentClient(path="/path/to/save/to")
You can see that I use persistence mode in my other videos:
th-cam.com/play/PL58zEckBH8fA-R1ifTjTIjrdc3QKSk6hI.html
I hope this answers your question.
do you have a list of your code or colab link?
Here you go: github.com/johnnycode8/chromadb_quickstart
Thanks@@johnnycode
Thanks. I do have one or two questions if you don't mind, first client = chromadb.PersistentClient(path='content/drive') does not create the db on colab , the folder exist. It just defaults to in memory and store it there, not sure if that is because of colab? Also when I retrieve using ' document = collection.get(ids=[document_id],
include=['documents'])' still brings the entire record instead of just documents, {'ids': ['kk'], 'embeddings': None, 'metadatas': None, 'documents': ['**Section 1: Numbers 1-5 in ......' am i doing this wrong? Thanks a bunch
For question 1: path='content/drive' points to your Google Drive folder. Change it to something like path='content/myvectordb' or path='content/drive/My Drive/Colab Notebooks/myvectordb'.
For question 2: The 'get' function will always return the entire record structure, but you can see that the data is not returned for your example: embeddings:None,metadatas:None.
I tried all sorts of combinations for persist directory like './', 'content', './content' nothing works.
UserWarning: Unsupported Windows version (11). ONNX Runtime supports Windows 10 and above, only. беда с вами...
U should really sell ur code everyone's benefiting for free.