Data processing & Tokenizing l Basic to Advance | Generative AI Series

[@dps158]

Hi, can you help me find which tab on the Hugging Face page lists embedding models? I couldn't find a specific categorization for embedding models anywhere.

[@CodeCommander-d6t]

search embedding models huggingface on google

[@dps158]

Hi, could you please explain how to select a tokenizer model from the Hugging Face library? I'm new to this field and would like some guidance. On the Hugging Face webpage under the "Models" tab, which category of models should we choose for specific tasks? For example, in the case of tokenization, which category should we select to choose an appropriate model?

[@CodeCommander-d6t]

The model you want to use same model you can use as tokenizer here.

[@dps158]

@@CodeCommander-d6t Thank you for your reply, sir. I’ve explored various TH-cam playlists, but I haven’t found satisfactory answers to the following:
The field of generative AI is evolving rapidly, and the LangChain framework, in particular, is changing at a fast pace. I need assistance with two specific aspects:
Whenever you decide to use a library, could you please provide the following details:
The purpose of the library.
Where I can find its functions and documentation for reference.