Store OpenAI Embedding Vectors in a Pinecone Vector Database with Bubble.io (Similarity Search)

แชร์
ฝัง
  • เผยแพร่เมื่อ 24 ธ.ค. 2024

ความคิดเห็น •

  • @malekaix
    @malekaix ปีที่แล้ว +2

    This is so helpful. For real thanks so much! My quickest subscribe ever

    • @cerumai
      @cerumai  ปีที่แล้ว

      Thanks for the feedback! I appreciate it

  • @jalengonel
    @jalengonel ปีที่แล้ว +1

    You are a godsend for making this. The perfect pace to follow along, very intuitive, great explanations. Thank you!

  • @Mehdi-tq9ps
    @Mehdi-tq9ps ปีที่แล้ว +2

    Wanted to thank you for the, concise, insightful, and easy to understand videos man. Love it👌🏼

    • @cerumai
      @cerumai  ปีที่แล้ว

      Thank you, I appreciate it!

  • @Augmented_AI
    @Augmented_AI ปีที่แล้ว +1

    How is pinecone and langchain different?

    • @cerumai
      @cerumai  ปีที่แล้ว +1

      Langchain is the overarching development framework for creating applications that are powered by LLMs, it comes with w/ a whole suite of tools and can do so much! (chaining models, schema's, prompt templates, agents, and many more!
      meanwhile pinecone, simply put is just a vector database. Think of it like a database to store your uploaded/indexed content which can be later referenced with a framework like langchain.
      I have a video that goes over some very high-level concepts of Langchain I recommend you watch, but I hope this helps out!

    • @Augmented_AI
      @Augmented_AI ปีที่แล้ว +1

      @@cerumai Thanks for the detailed explanation :)

  • @patrickgonsalves
    @patrickgonsalves ปีที่แล้ว

    Did I miss when you set up the api for pinecone? I didn’t see the details for ingestion or the other. Did you skip it?

  • @salemmohammad2701
    @salemmohammad2701 ปีที่แล้ว

    in this way the The chunks will be retrieved and displayed as is without generating a text from them using Openai Generation, right?

  • @marcelokugler3339
    @marcelokugler3339 ปีที่แล้ว +1

    hey! Thanks a lot for this!!! But 1 question..... I added a large text inside the Upsert field once and saved to Pinecone...it´s a large text - not too large, but large!... but when I make a question, it returns ALL the text I inserted.... Shouldn't I have received the answer only to what I asked for? Or should I have divided into paragraphs and kept inserting? Again - thanks so much for this!! What I want it add some text (let´s say 20 pages of docs....and get answer from it...

    • @cerumai
      @cerumai  ปีที่แล้ว

      Hey thank you! So with the way it's currently set up right now it will just return the metadata that was closest to your query, which in your case is just the large text string (all of it).
      We're not chunking any of it up and plugging it into a new prompt to use as context, this is actually referred to as few shot prompting. We're simply just just doing and returning a similarity search.
      Yes although this may be inefficient to manually chunk it up yourself, this can certainly be done if you choose to inject the data in chunks.
      I'm going to make a follow up that will perhaps do a few shot prompt as I mentioned above. This will be very powerful!

    • @marcelokugler3339
      @marcelokugler3339 ปีที่แล้ว +1

      @@cerumai Thanks for your quick reply! I will keep my eye on this!!! But do you know how to do it ? I mean - upload files and then ask questions about these files and get answers from it ? I know it is possible in Bubble, but I am still trying to figure this out!!

    • @cerumai
      @cerumai  ปีที่แล้ว +1

      @@marcelokugler3339 you would need to plug-in that metadata that’s returned back from Pinecone in as {context} into a new GPT-4 prompt.
      I actually have a conceptual style video on this with a framework called langchain

  • @salemmohammad2701
    @salemmohammad2701 ปีที่แล้ว

    If the text is long, is there a need to chunk it before embedding it?

    • @cerumai
      @cerumai  ปีที่แล้ว

      No need to chunk data if it fits within OpenAI’s context window. However it can be cost prohibitive to fit large context sizes upon each prompt at scale

    • @salemmohammad2701
      @salemmohammad2701 ปีที่แล้ว

      @@cerumai I mean if the text that is sent to pinecone is long, do we need to chunk it and send every chunk in a separate api call to to openai for embedding then to pinecone

  • @jpedrorw
    @jpedrorw ปีที่แล้ว

    Great tutorial.. I was fine-tuning gpt3 davinci with my company data. Whats the difference in this case? I mean, rather than using the davinci model from openai, the aiming goal I see is to ask natural questions and get back the information you seek. My question is, is it more productive doing stuff like this in pinecone (but chunking in someway) or fine-tuning openai models?
    I would love to see a tutorial from you with pinecone with chunk data so we can optimize our response.
    Keep it up and thank you!

    • @cerumai
      @cerumai  ปีที่แล้ว +1

      I'd say the 2 terms can be used interchangeably however the methods of doing so vary greatly.
      Using pinecone (a vectorized DB) to then perform a similarity search for context paired with a user-generated prompt is a lot more efficient & scalable way to deploy the models IMO, and you're leveraging the already pre-trained models (such as text-davinci & GPT4) they're really good to perform tasks.

    • @Magamax54
      @Magamax54 ปีที่แล้ว

      Hi, I am doing more or less the same for my company where I want to implement a chatbot related to the company's information, I wanted to know if you had the answer to your question please ? What is the best way to proceed between using openai's chatgpt chat model or this alternative with pinecone ? Thank you for your response

    • @cerumai
      @cerumai  ปีที่แล้ว +1

      @@Magamax54 With most "custom trained" chatbot systems today, I'd say a majority of them use in-context learning via chunking which is a cost efficient way and just simpler way to get the desired results.
      Since you're using the already trained LLMs underneath the operation in accordance with custom data as context.
      Hope this helps

  • @Bangs_Theory
    @Bangs_Theory ปีที่แล้ว +1

    Hey grest video! Is there any chance you might do a similar video, but using chunking for large datasets? Would live to get a tutorial on that. Thanks in advance.

    • @cerumai
      @cerumai  ปีที่แล้ว +2

      Yes I was actually planning on doing this!

    • @MisCopilotos
      @MisCopilotos ปีที่แล้ว +1

      @@cerumai looking forward!

    • @cerumai
      @cerumai  ปีที่แล้ว +1

      @@MisCopilotos check out my most recent video where I do this with another tool

    • @MisCopilotos
      @MisCopilotos ปีที่แล้ว

      @@cerumai thanks! This one, right? th-cam.com/video/O7NhQGu1m6c/w-d-xo.html