วีดีโอ

How long have you lived in Europe and what countries exactly?

Thanks, will do!

Thank you!

I am not sure I understand the question.

I cannot share the slide but you can see the diagrams in my newsletter: newsletter.theaiedge.io/p/understanding-the-self-attention

I am liking reading that!

I am glad to read that!

Well I do!

It is coming! It will take time

My accent + my speaking skills are my weaknesses. Working on it and I think I am improving!

@@TheMLTechLead Thanks for your reply but absolutely no apology necessary!! I think it is an excellent video and helpful information. Much appreciation for posting. I am a professor in a business school and always looking for insights into how to teach the technical side of technology in the context of business. Your explanation has been very helpful.

Tree-based method can naturally be used to measure Shapley values without approximation: shap.readthedocs.io/en/latest/tabular_examples.html

So we have an ensemble of trees F that predicts y such that F(x) = \hat{y}. The error is y - F(x) = e. We want to add a tree that predicts the error T(x) = \hat{e} = e + error = y - F(x) + error. Therefore F(x) + T(x) = y + error

Why

“Give me the exam solutions pls”

It is in the script shown in the video and I am not adding the target of the current row, which is equivalent to subtracting it if I were to add it.

​@@TheMLTechLeadUnderstood ! Thanks for replying .

Inside the model, we compute the self-attentions. They are pretty much just a measure of interaction between the different tokens in the input sequence. Inside the attention layer, we have the queries, the keys and the values. The keys and queries are used to compute the self-attentions and the resulting hidden states is the weighted average of the values where we use the attentions as weights. At that point, the order of the tokens in completely lost because we are just summing stuff together without knowing in what order they were before the sum. That is why we keep the position information through the positional encoding. We systematically add the same vector for the same position so the model starts to understand how that shift relates to that position. The value of the same token varies depending on its position. To be fair, we do it a bit differently in 2024. Video coming!

@@TheMLTechLead Looking forward to it! After commenting, I read about RoPE (can't say fully understood it) and learnable positional embeddings. P.S. I really liked your idea of using routing in attention, a bit ambitious goal, but I want to use it to train a small language model or I will see if it is possible to simply add it in a pre-trained model without losing the learned weights.

@@jaskiratbenipal8255 I may not make a video about RoPE but I wrote something about it here: www.linkedin.com/posts/damienbenveniste_most-modern-llms-are-built-using-the-rope-activity-7188571849084096515-mmUk. For the routed self-attentions, I am looking forward to see somebody implementing those and training a model with it.

@@TheMLTechLead I tried it, I trained a language model from scratch for next character prediction (to have small vocabulary). The results were good using normal attention, the model was able to form words and phrases and some jibberish that looked like words. With the routed attention (I tried 0.1 and 0.3 sparcity values), it started to diverge and model was not converging at all after first epoch. The training time did decrease from 34 to 24 mins.

That is my weakness!

@@TheMLTechLead It's not that bad. As a English speaker that knows/speaks French, I can tell you are French. It's not had to hear and understand what you are saying. I'd say keep going and make any improvements along the way.

not true at all, i am watching in 1.5x and can still understand everything

Thanks @TheMLTechLead! You did a great job in summarizing the key ideas.

Ok! Yeah I figured it was annoying!

It can be used for any computer vision ML task.

@TheMLTechLead great! I was thinking of image generation for a given prompt or user input.., what would the process be?

Oh no, you would need a very different model for that. Although, the vision transformer can be an element of it.

@@TheMLTechLead got you!

If I need somebody, I know where to look!

Why do you expect them to be different?

Maybe you are asking what is the difference between boosting in general and gradient boosting in particular? To be fair, my video in not going deep enough to highlight the differences. In a coming video, I am going to go into the details of how XGBoost works and I believe that should clear up the confusion.

Happy to hear that!

Absolutely!