Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math
ฝัง
- เผยแพร่เมื่อ 24 ก.ค. 2024
- In this video I will explain Direct Preference Optimization (DPO), an alignment technique for language models introduced in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model".
I start by introducing language models and how they are used for text generation. After briefly introducing the topic of AI alignment, I start by reviewing Reinforcement Learning (RL), a topic that is necessary to understand the reward model and its loss function.
I derive step by step the loss function of the reward model under the Bradley-Terry model of preferences, a derivation that is missing in the DPO paper.
Using the Bradley-Terry model, I build the loss of the DPO algorithm, not only explaining its math derivation, but also giving intuition on how it works.
In the last part, I describe how to use the loss practically, that is, how to calculate the log probabilities using a Transformer model, by showing how it is implemented in the Hugging Face library.
DPO paper: Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S. and Finn, C., 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. arxiv.org/abs/2305.18290
If you're interested in how to derive the optimal solution to the RL constrained optimization problem, I highly recommend the following paper (Appendinx A, equation 36):
Peng XB, Kumar A, Zhang G, Levine S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. 2019 Oct 1. arxiv.org/abs/1910.00177
Slides PDF: github.com/hkproj/dpo-notes
Chapters
00:00:00 - Introduction
00:02:10 - Intro to Language Models
00:04:08 - AI Alignment
00:05:11 - Intro to RL
00:08:19 - RL for Language Models
00:10:44 - Reward model
00:13:07 - The Bradley-Terry model
00:21:34 - Optimization Objective
00:29:52 - DPO: deriving its loss
00:41:05 - Computing the log probabilities
00:47:27 - Conclusion - วิทยาศาสตร์และเทคโนโลยี
Very clear explanations!! Please, continue making such good videos!
The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way
我们在领英联系吧，我有个微信小群，你可以参加
@@umarjamilai我也想加
@@umarjamilai我看到b站也有你的账号
Thank you! It's very clear explaination. It helps for reading the original paper. Looking forward to new topics.
I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared
Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!
wow your explanation is so clear and complete... you are godsend, keep doing it. Sei un fenomeno
You explained very clearly. Thanks!
Thanks for making these videos. Concise and clear
Legend is back, the GOAT, if my guess is right next will be ORPO or Q*
Actually, the next video is going to be a totally new topic not related specifically to language models. Stay tuned!
@@umarjamilai waiting
These lectures are amazing. Thank you!
Thanks so much Umar, always learn a lot from your video!
New video🎉 can't wait to watch. Although having used DPO in production for a while now!
Amazing! Great job once again!
My Kind Request Please Increase volume little bit , just little bit. Otherwise your videos Outstanding . Best I can say.
Best explanation so far !!!!...
Thanks for your lecture. I wonder could you explain the vision language models
Thank you very much for this video, please make ORPO as well.
Amazing explanation. Would it be possible to make a video on the theory and implementation of automatic differentiation (autograd).
Amazing Video! Please do one on SPIN (Self Play Fine-tuning) as well
Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets
Love from India sir, you are a legend 😊😊
love ur videos umar !!
Enjoyed the style in which the video is presented. Which video editor/tools do you use to make your videos? Thanks.
I use PowerPoint for the slides, Adobe Premiere for video editing
@@umarjamilai What do you use to draw on your slides? I am assuming you connected an ipad to your screen.
I love ur videos so much. please make a video about mamba or mamba vision
There's already a video about Mamba, check it out
Thank you. can you make video about liquid neural network?
Great 👍
great explaination, thanks. how about the recent work: KTO: Model Alignment as Prospect Theoretic Optimization? can you compare it with DPO?😁
thank you so much for your effort! could you make a video about tokenizers like BPE and sentencepiece from scratch? I would be very appreciate of it!
Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?
Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.
Hello Umar,
Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you
great video. love from india.
Thank you very much for the explanation.
I had one questions. Are the dataset of preferences always made of two and only two answers?
According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.
@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.
Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?
The potential problem that I think could happen is that having random text may decrease the loss and the policy may not even change much
Valeu!
what is the ppt software you are using sir ?
Hi Umar,
If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model?
- The chat model applied LoRA
- Or the chat model itself without LoRA?
Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.
@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.
Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.
I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better
Sorry for the trouble, I'll keep it in mind for the next videos!
“digital biscuits”, lol