Gemini Pro + LangChain - Chains, Mini RAG, PAL + Multimodal

Gemini Pro is out and you can use it with LangChain
Colab: drp.li/Keymg
Blog Post: ai.google.dev/
My Links:
Twitter - / sam_witteveen
Linkedin - / samwitteveen
Github:
github.com/sam... (updated)
github.com/sam...

[@breakitdown4346]

Hi Sam! I just wanted to say I've been watching your videos for over a year now, and they are always so informative! You really take your time explaining every step and make the learning material very easy to digest no matter what the experience level is. Thank you for all your hard work. Keep it up!

[@samwitteveenai]

Thanks for the kind words. Glad the videos are helpful.

[@Slimshady68356]

​@@samwitteveenaiyeah man, you r doing gret for the community

[@314_anizbinnowshad5]

Thank you Sam, Really been searching to integrate langchain and Gemini

[@tanveerbashir8393]

Thank you, Sam !!! Really it's very informative video. Keep doing this great work.

[@broofya3805]

Good job, Sam! Always worth it

[@mrchongnoi]

Great video as usual. Really enjoyed it.

[@ShlomiSchwartz]

Thanks again, for another great tutorial :)

[@umarfarooque3687]

thank you for this video. I would love to see if you could make video on using llm with tools like gmail, linkedin etc to automate day to day work

[@satvalite1854]

Hey sam great vid
would love a video for a basic for csv and pdf querying using gemini

[@glouie]

Amazingly clear. Thank you so much. I believe that even a non-programmer like me will be able to start playing around with these functions.
I have a large Zotero database containing PDF files related to Alzheimer's Disease. I would like to input the database to create a RAG What would you recommend? The simplest use case would be to provide scientists with the ability to retrieve documents and statements related to a human conceptualized grant submission. More interesting, would be a series of prompts to create an AI colleague for hypothesis generation based on an LLM review on the human selected literature.

[@jesusleguiza77]

Hi Sam, there is some way for use function calling too? Regards

[@yuanshisklse]

Thank you Sam, Really great example on integrating langchain and Gemini. Could you direct me to the notebook that you demoed just now?

[@henkhbit5748]

Thanks for the video👍 Multimodal RAG with open source would be nice ...

[@Dovgonosyk]

Thank you for the video, really informative! What would you recommend if I need a custom llm working with a set of documents(pdf, csv, etc)? Looks like a task for a langchain+gemini+cloudstore, but still can’t figure out the best way of doing it…

[@AIdevel]

Hi how to restrict the model answers in RAG apps only to the knowledge base and reply with sorry if the answer is not available effectively the retrieval returns empty list and then model start to acts as ChatGPT to generate from its parametric knowledge

[@furbasherpa2598]

Hello Sam, where you have given the vectorstore with text data, what to do if I have an excel sheet data that I would like the gemini pro to retrieve

[@generic-youtube-user]

hello Sam. Thanks fo this! Just wanted to ask do you have any general overview for performance of GPT-4 against Gemini's competitor model for purely text generation tasks? Actually GPT-4 is very slow for generating text but it is smart enough formany applications. Its latency causes me to have second thoughts using it.

[@advaithr1984]

Hey Sam, I haven't tested out Gemini pro as of now and am planning to do in near future. But as compared to GPT 4 or any Palm based models what significant improvement can you notice while playing around with Gemini in terms of RAG or any NLP based program?

[@seththunder2077]

Hey sam. Can you please explain more about the DocArrayInMemorySearch? I did not really understand it

[@innocentanyaele5986]

When i run the chain.invoke({"question": "what is gemini?"}) in the "More Complicated Chain - Mini RAG" section, I get the error
f"Message of '{message.type}' type not supported by Gemini."
AttributeError: 'tuple' object has no attribute 'type'

[@youshouldknowme2773]

i've met the same error. Have you solved yet?

[@samwitteveenai]

let me look into it. I know they have done some updates since I shot the video. Gemini also sometimes gets weird errors very occasionally when it thinks something is NSFW.

[@innocentanyaele5986]

Alright, thank you @@samwitteveenai

[@innocentanyaele5986]

@@youshouldknowme2773 sadly, no

[@samwitteveenai]

@@innocentanyaele5986 what error are you seeing? "AttributeError: 'tuple' object has no attribute 'type'" Do you get the error every single time ?

[@d3nt391]

Is there any benefit to using it through Langchain vs just the genai module? Langchain seems to overcomplicate everything

[@samwitteveenai]

You can hook it up to RAG and VectorStores easily and also make use of the premade chains like the PAL Chain I show in the video. That said you certainly don't have to use LangChain to do these things if you are comfortable writing some code yourself.

[@d3nt391]

@@samwitteveenai thank you for the thoughtful reply!

[@jarekmor]

It is worth to mention about LangSmith platform where you can debug, test, evaluate, monitor chains and integration of it with Langchain framework.

[@alealejandroooooo]

Interested in multimodal RAG with an open source multimodal model. 🙂

[@MuhammadAsif-mm4py]

Anyone who can help me with the IDE he is using and how to get started

[@samwitteveenai]

Most of the code is using Colab which has a free tier and runs in the cloud from Google

[@Akhil-me6gk]

Hey Sam very good video but I think it still does not support the creation of conversation chains like chatbots with ConversationRetrievalChain. I was trying it but it gave me some errors while running it.

[@samwitteveenai]

Yeah you will need to remake some of them as Gemini currently doesn't support System Prompts and needs to always go Human -> AI -> Human etc you can't have 2 human prompts in a row or a system prompt. You can feed the memory in as a context though pretty easily. I might make some more vids for this .

[@Akhil-me6gk]

@@samwitteveenai Yeh sure that would be very helpful :)

[@caiyu538]

👍

[@velugucharan8096]

Bro i know little bit of Deep learning and machine but i want to take lang chain course from where i want to star Imagine i am beginner suggest me some TH-cam free source to learn lang chain and generative AI PLEASE BRI

[@samwitteveenai]

I have a whole bunch on this channel that you can checkout. I am thinking of making a new updated set of basics for the start of 2024 as well.

[@velugucharan8096]

@@samwitteveenai please do that as much as possible broo

[@aaroldaaroldson708]

Why don’t you just go through their documentation? It is beginner friendly

[@advaithr1984]

@samwitteveenai himself has a lot of content out there. Other useful youtuber are @krishnaik06 , @Analyticsvidhya , @NicholasRenotte. These have always been my go to channels. All four mentioned above have done significant content in the field of NLP, Langchain. Have the Documentation to your side all time.

[@putinninovacuna8976]

This is quite unhelpful. You’re not demonstrating how to implement this in a real web application and looking on Google’s documentation is so inadequate all you get from googles tutorials are snippets of code for notebooks, but they don’t teach how to integrate Vertex AI Vector Search into a real web application, not just notebooks.

[@samwitteveenai]

When I made this video there was no VertexAI LangChain. I am still frustrated with the Google support for LangChain so haven't made any more vids about it I have one shot but it breaks in one part so I never released it. Hopefully will do some more soon