How to build Multimodal Retrieval-Augmented Generation (RAG) with Gemini

[@GoogleDevelopers]

Check out all the AI videos at Google I/O 2024 → goo.gle/io24-ai-yt

[@gangababu2063]

IO2024_Multimodal_RAG_Demo.ipynb can't find this notebook

[@PeterLappo]

Pretty useless video without sample code.

[@jprak123asd]

I wanted to extend my heartfelt thanks for the excellent session on how Retrieval-Augmented Generation (RAG) can be used to train Large Language Models (LLMs) to build expert systems in the retail, software, automotive, and other sectors.
Your explanation was incredibly clear and insightful, making a complex topic easily understandable. I truly felt like Dr. Watson listening to Sherlock Holmes unravel the mysteries of the universe, marveling at the clarity and depth of the information presented.
Your efforts in breaking down the concepts and applications of RAG in such a straightforward manner have left me feeling both enlightened and excited about the potential this technology holds for our industry.
Thank you once again for your time and for sharing your expertise. I look forward to exploring and implementing these innovative solutions in our own projects

[@thyagarajesh184]

Impressive technology. Look forward to using it for my project.

[@charlesbabbage6786]

Could'nt find the exact notebook used here.

[@dumbol8126]

will there be an opensource version of this, or atleast a paper

[@sarvariabhinav]

WHERE IS THE SAMPLE CODE??????? This is very frustrating to showcase but not share code.

[@diegomoralessepulved]

@googledevelopers I second this comment.. could you please share that notebook?

[@noetic4681]

agree

[@zuowang5185]

How do you handle terabytes of enterprise data, just do embedding groups? Should you generate sub questions first? How do you handle large amount of users?

[@nestorbao2108]

Why do you use multimodal embedding model if you summarize images and ground them into text?

[@mariaescobar8003]

When I use RAG, Am I sharing my data with the model/company? or is it private with an extracost?

[@vichupayyan]

Rag is an architecture i believe. with out without it - whatever happening to the data same applies

[@hitmusicworldwide]

Not necessarily. You can keep the data local. You only use the LLM for it's ability to summarize and generate responses as well as queries

[@mohamedkarim-p7j]

Thank for sharing👍

[@hasszhao]

where is this notebook in the cookbook repo?

[@d.d.z.]

Same question

[@shubhamsharma5631]

33:18

[@Chitragar]

I have a notebook in Kaggle named Multimodal RAG Gemini - should help, YT removing links for some reason.

[@d.d.z.]

@@Chitragar thank you

[@cullenharris1837]

​@@shubhamsharma5631 I challenge you to find it. That is simply a link to the general github which is convoluted , not the exact notebook which is difficult to find.

[@homeandr1]

Hello Jeff, could be that there is a mistake 24:00 in a for loop instead of “for i, s in enumerate(texts + table_summaries + image_summaries)” should be “for i, s in enumerate(text_summaries + table_summaries + image_summaries)”

[@TL735]

Nice, but why don't you develop a simple drag-and-drop RAG? e.g. I add a drive folder link and Google generates a RAG chat based on its content.

[@RiccardoCarlessoGoogle]

Is there a link to the python notebook? I'd love to play with it!

[@nagpalvikas]

Is "unstructured" the best choice here for parsing PDF? Any better alternatives?

[@ai_asymmetric]

Llamaparse

[@You_Only_LiveOnce]

langchain would be a good choice

[@yadav-r]

Can I use the fine tuned Gemini RAG model via API from a mobile app?

[@evanrfraser]

Fantastic. Thank You!

[@ammarfasih3866]

where is the notebook?
Can someone please share the link?

[@IndianLeopard7]

Wat about Copyright and Ethical issues? How much do u guys charge for using ur model? And as per IBM and Oracle embeddings are nothing new so why use urs?

[@nagarathnabheggade8410]

This example briefs about text and PDF, do we have any for video how de we use RAG, Vector store for Video can anyone give some reference

[@descarded]

im not sure if there are existing libraries to do that, maybe check docs. although here's my intuitive approach. video is basically series of images with some history/context attached to previous and subsequent frames. so if you keep that history across frames intact by either providing previous frames as input, or keep a local vector of it all, you can make it work. not sure if its the best approach, but i m open for discussion

[@julianayue402]

Can you please provide the source code? It would be great help!! Thank you!

[@ajanieniola9172]

Wonderful

[@ai_asymmetric]

dense embeddings are never enough for RAG system

[@oldmansgoldenwords]

You can get blue driver and get all error codes and example

[@SonuChaudhary]

Where is the code link?

[@pra8495]

github link please

[@shubhamsharma5631]

33:18

[@kaushikdas5115]

@@shubhamsharma5631 can we run the code without subscription?

[@SB-md2km]

Ok but someone could literally look any of this up online or look for it in a manual, etc. w/out using AI...

[@adithiyag4616]

Please share the colab link

[@gnanasenthil654321]

Yes please do

[@shubhamsharma5631]

33:18

[@KitchenAIdev]

Hmm... interestin...

[@dr.p.srinivasaragavanperum2911]

Happy

[@user-xx3mr6vx9u]

Haha we just need your browsing history

[@dr.p.srinivasaragavanperum2911]

🎉

[@Inceptionxg]

After Muaadh Rilwan's post on LinkedIn

[@dr.p.srinivasaragavanperum2911]

❤

[@fast-path]

🥺

[@JH-bb8in]

This shows how garbage Langchain is as a library. Extremely verbose and intransparent.

[@imai-pg3cz]

Is there any framework better than Langchain?

[@gokusaiyan1128]

can you tell me more about it please :)

[@ohmatokita5990]

So if I'm using the 2nd way, what's the name of the multidality-modal would be?