Semantic Chunking for RAG

In this event, we’ll learn how the semantic chunking algorithm works! Text is split into sentences that are converted vectors through an embedding model. Similarity is measured between each pair of consecutive sentences. If sentences are too similar, as defined by a threshold, additional chunks are created. We can ensure that if any two consecutive sentences are too different from one another, additional chunks can be created. In theory, this will allow us to achieve better results during retrieval within our RAG system.
Event page: lu.ma/chunkingrag
Have a question for a speaker? Drop them here:
app.sli.do/event/eQnuJrAp9sN3...
Speakers:
Dr. Greg, Co-Founder & CEO
/ gregloughane
The Wiz, Co-Founder & CTO
/ csalexiuk
Join our community to start building, shipping, and sharing with us today!
/ discord
Apply for our new AI Engineering Bootcamp on Maven today!
bit.ly/aie1
How'd we do? Share your feedback and suggestions for future events.
forms.gle/1UxkU7LbfV14f77p7

[@AI-Makerspace]

Google Colab notebook: colab.research.google.com/drive/1gGLd-rdPsM1iy4JmL1V1mfZm90CmDcXR?usp=sharing
Event Slides: www.canva.com/design/DAGAtxFPH2M/3oo8gElRKU21fQH-ZzYNNA/view?DAGAtxFPH2M&

[@bananamaker4877]

Love this video and new strategy of semantic chunking. Thanks to Greg and Chris for explaining this concept the way how it should be. Again thanks for making it open source.

[@AI-Makerspace]

Thanks bananamaker!! We enjoyed getting down into the weeds of some often-overlooked pieces today, and we're also fans of the new strategy! Look for more content like this from us soon!

[@damiangilgonzalez8011]

Awesome job guys! I wached this video with my coffe this morning and it was a perfect way to start my day (learning, drinking a coffe and lisening a really good spekears/teachers)

[@AI-Makerspace]

This is awesome Damian - thank you! We're pumped we got to spend the morning with you :)

[@JankayYashwant]

Please make many more awesome explainers like this!

[@AI-Makerspace]

You can count on it @JankayYashwant!

[@NhatNguyen-bq6jj]

Can you introduce some related articles? Thanks!

[@channel_panel193]

heyyy u guys look familiar from the fourthbrain bootcamp i took! nice

[@zugbob]

When doing RAG in general is it best to insert it into the system prompt or to have an assistant message for it?

[@AI-Makerspace]

It's really up to you - and depends on if you're using examples or not.

[@MrDespik]

You forgot to show how we can combine semantic chunking with parent document retriever)
I mean what chunks we need to use as parents and as childs.

[@AI-Makerspace]

I'm sorry! We didn't intend to explore this in the session!