The RAG chatbot you demonstrate is an excellent lesson with HuggingFaceEmbeddings. Regarding how to do it outside GPT being generic enough to have your own vectorDB on demand for any model I had wondered how that was done. Thanks for covering this really great stuff!
Very interesting tutorial. Seems like a great, lower-budget, and lower-dev-time alternative to building directly off of Open AI API's. And I am also a fan of European metal!
Hey Julien, great job with the video. For QnA on corpus I'd recommend to generate hypothetical questions for each paragraph & ingesting them as well since those would have better similarity to the user input which is usually a question & can also help constrain the model to answer only closed domain questions.
Thank you Julien - this is super useful and comes at the right time during my writing season (you know what I'm talking about :-) ) As someone else mentioned in the comment, I also received an error when calling Textract. I solved it by adding `pip install amazon-textract-textractor -qU` - hope it might help others
thanks it was really informative, can do demonstrate fine tuning LLM's with lora and Qlora? In your experience, RAG has better performer over fine tuning ?
Llama2 fine-tuning with Qlora: th-cam.com/video/Zev6F0T1L3Y/w-d-xo.html. IMHO RAG and fine-tuning solve different problems and are complementary. RAG lets you access fresh company data and gives you some domain adaptation. Fine-tuning gives you better domain adaptation and lets you customize guardrails and tone of voice.
FAISS is a simple lightweight open source solution. Pinecone is a fully managed, closed source DB running in the cloud. Depends what you're looking for, and how much work you want to do on managing the solution :)
Hi Julien, Thank you for this video. It's helping me learn a lot. I was trying to run the code. When I attempt the zero shot example, my output is quite different from whats shown in the video. I tried to split it, but I get something like this - [answers: * 1) The trend is to invest more in solar energy in China. * 2) The trend is to invest less in solar energy in China. * 3) The trend is to invest the same amount of money in solar energy in China. * 4) The trend is to invest more in solar energy in the United States. * 5) The trend is to invest less in solar energy in the United States. ] Can you please explain why this is happening and how it can be fixed?
Thanks for the tutorial.In my case,i can't use Mistral somehow due to some restrictions on AWS test account.I have used FLAN-T5 but it is giving this error.ValueError: Error raised by inference endpoint: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (422) from primary with message "Failed to deserialize the JSON body into the target type: missing field `inputs` at line 1 column 503".
The input format for T5 is quite different, so sending a Mistral-formatted message won't work. Not sure what restriction you're facing, but maybe TinyLlama would work? I think you would only have to adapt the prompting format in the content handler.
Thanks@@juliensimonfr , Basically it is question rephrase request by passing entire chart history, tried this approach which has cost and token limit problem Looking for other alternative for the same
I want to embed large data. In this case, if I want to embed document without a GPU notebook ml.t3.medium, is it possible to deploy the embedding model as well in some ml.g5.large GPU instance to make the processing faster?
Hi Julien. The code is not working when I try to run it. I think the error I am getting is related to Sagemaker credentials. I made an account just now but don't know where to get information where I can plug into your code to make this work.
Start here: docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html. Create a notebook instance and make sure its IAM role includes the SageMakerFullAccess and TexttractFullAccess managed policies. Once you've done that, the notebook will run as is.
Thanks for your reply! It seems that this leads me to make a Jupyter notebook. How do I integrate this to do what you're showing on Colab in the tutorial?@@juliensimonfr
Seems like a great video, but I can't move from the starting line. You seem to be skipping over very important details about how to deal with the HuggingFace tokens, AWS security keys, regional compatibility settings with Sagemaker, etc. For example, when running the copied SageMaker code, I get "ValueError: Must setup local AWS configuration with a region supported by SageMaker", but no region seems I try seems to work. Did you cut all the authentication code from your demo? Obviously you don't want to disclose security keys, but at least show/explain that part of the setup code and simply redact the sensitive information.
@@juliensimonfr Yes, clearly I'll need to do this! Nonetheless, as an educator myself, I think my point is still useful. It's helpful to learners to mention parts that you skip over. You don't have to teach it in this video, but it would be helpful to mention that there are steps one must perform that are not shown in this video.
Thx for the video :) can you update your vector database by a few lines ( if you want to add data to your knowledge base) automatically by running a python script or something like that?
The RAG chatbot you demonstrate is an excellent lesson with HuggingFaceEmbeddings. Regarding how to do it outside GPT being generic enough to have your own vectorDB on demand for any model I had wondered how that was done. Thanks for covering this really great stuff!
Glad it was helpful!
always making great and timely videos.
Glad you like them!
Very interesting tutorial. Seems like a great, lower-budget, and lower-dev-time alternative to building directly off of Open AI API's. And I am also a fan of European metal!
Absolutely. Rock on 🤘
Thank you for your lectures.
You are very welcome
Hey Julien, Thanks for an insightful talk last night at the AWS center!
You're welcome. Thanks for coming!
thanks julien, one can learn so much from these!
That's the idea 😀
Hey Julien, great job with the video. For QnA on corpus I'd recommend to generate hypothetical questions for each paragraph & ingesting them as well since those would have better similarity to the user input which is usually a question & can also help constrain the model to answer only closed domain questions.
Yes, that's a nice trick. I tried to keep things simple here ;)
Thank you, gonna check it out tomorrow!
Have fun!
Thanks for this clear explanation.
Glad it was helpful!
Hi Julien, thanks for your video, pretty clear explained ;-)
Glad it was helpful!
Superr video, Thanks for trying using open source solutions...
Glad you liked it
Thanks for the video!
You're welcome!
Thank you Julien - this is super useful and comes at the right time during my writing season (you know what I'm talking about :-) ) As someone else mentioned in the comment, I also received an error when calling Textract. I solved it by adding `pip install amazon-textract-textractor -qU` - hope it might help others
Ok, good to know. Thanks Seb and good luck with the writing ;)
also 'pip install pip install faiss-cpu' :-)
Sagemaker with langchain streaming option is generating output
Thanks a lot! It was very, very helpful.
You're welcome.
It throws : KeyError: 'Blocks'
after running the cell with boto3.client('textrac') thrown by the loader.load(), from parser in langchain
Thanks Julien! very nice video. very curious if there are some compare between bge-small with ada-002 when used in RAG.
Hi, please check our embeddings leaderboard at huggingface.co/spaces/mteb/leaderboard. ada-002 is #15, bge-small is #8 :)
thanks it was really informative, can do demonstrate fine tuning LLM's with lora and Qlora? In your experience, RAG has better performer over fine tuning ?
Llama2 fine-tuning with Qlora: th-cam.com/video/Zev6F0T1L3Y/w-d-xo.html. IMHO RAG and fine-tuning solve different problems and are complementary. RAG lets you access fresh company data and gives you some domain adaptation. Fine-tuning gives you better domain adaptation and lets you customize guardrails and tone of voice.
Thanks Julien, for the good tutorial! Some use pinecone, do you see differences/advantages of using faiss over pinecone? Thank you
FAISS is a simple lightweight open source solution. Pinecone is a fully managed, closed source DB running in the cloud. Depends what you're looking for, and how much work you want to do on managing the solution :)
Hi Julien,
Thank you for this video. It's helping me learn a lot. I was trying to run the code. When I attempt the zero shot example, my output is quite different from whats shown in the video. I tried to split it, but I get something like this - [answers:
* 1) The trend is to invest more in solar energy in China.
* 2) The trend is to invest less in solar energy in China.
* 3) The trend is to invest the same amount of money in solar energy in China.
* 4) The trend is to invest more in solar energy in the United States.
* 5) The trend is to invest less in solar energy in the United States. ] Can you please explain why this is happening and how it can be fixed?
Thanks for the tutorial.In my case,i can't use Mistral somehow due to some restrictions on AWS test account.I have used FLAN-T5 but it is giving this error.ValueError: Error raised by inference endpoint: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (422) from primary with message "Failed to deserialize the JSON body into the target type: missing field `inputs` at line 1 column 503".
The input format for T5 is quite different, so sending a Mistral-formatted message won't work. Not sure what restriction you're facing, but maybe TinyLlama would work? I think you would only have to adapt the prompting format in the content handler.
Great video, But what if user question is related to chat history and it may contain short cuts like he/she/that/it etc then how to handle such cases
Langchain has different ways to handle this, e.g. python.langchain.com/docs/modules/memory/types/buffer
Thanks@@juliensimonfr ,
Basically it is question rephrase request by passing entire chart history, tried this approach which has cost and token limit problem
Looking for other alternative for the same
I want to embed large data. In this case, if I want to embed document without a GPU notebook ml.t3.medium, is it possible to deploy the embedding model as well in some ml.g5.large GPU instance to make the processing faster?
Sure, it's what you would do for production.
can you tell how to get the key for sagemaker to work here?
Not sure what you mean. Are you looking for a SageMaker tutorial ? See docs.aws.amazon.com/sagemaker/latest/dg/gs.html
Hi Julien. The code is not working when I try to run it. I think the error I am getting is related to Sagemaker credentials. I made an account just now but don't know where to get information where I can plug into your code to make this work.
Start here: docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html. Create a notebook instance and make sure its IAM role includes the SageMakerFullAccess and TexttractFullAccess managed policies. Once you've done that, the notebook will run as is.
Thanks for your reply! It seems that this leads me to make a Jupyter notebook. How do I integrate this to do what you're showing on Colab in the tutorial?@@juliensimonfr
Seems like a great video, but I can't move from the starting line. You seem to be skipping over very important details about how to deal with the HuggingFace tokens, AWS security keys, regional compatibility settings with Sagemaker, etc. For example, when running the copied SageMaker code, I get "ValueError: Must setup local AWS configuration with a region supported by SageMaker", but no region seems I try seems to work. Did you cut all the authentication code from your demo? Obviously you don't want to disclose security keys, but at least show/explain that part of the setup code and simply redact the sensitive information.
How about going through Hugging Face 101 and SageMaker 101 first?
@@juliensimonfr Yes, clearly I'll need to do this! Nonetheless, as an educator myself, I think my point is still useful. It's helpful to learners to mention parts that you skip over. You don't have to teach it in this video, but it would be helpful to mention that there are steps one must perform that are not shown in this video.
Thanks!
You bet!
Thx for the video :) can you update your vector database by a few lines ( if you want to add data to your knowledge base) automatically by running a python script or something like that?
Sure, you can keep adding embeddings anytime you want.
oh thats nice :) thx for the answer@@juliensimonfr
how can I call onto my react frontend?
A SageMaker endpoint is an HTTPS API, so you can plug it in anything. You should be able to find lots of examples out there.
Y r u deploying first in sage maker
Because I don't want to manage any infrastructure :)
godlike