วีดีโอ

best

Can I ask what tablet/app you use to produce these videos? Thank you!

Thanks. These are quality videos. The more you go into the details and math behind the main algorithm, the better.

I tried running this but nothing happens when I run jupyter lab. Gives me the "not a command" message.

really appreciate for explaining such papers, please keep them coming <3

Diffusion forcing? Please

8:00 "It's just MLP" yeah, I felt bamboolzed. MLP originally was upscale, relu, downscale; later got beautiful gradient-bearing curves of ReLU replacement and additional multiplication(eg GeGLU instead of GELU). Here during initial explanation I felt that up proj and down proj got renamed to K, V. It doesn't feel like cross attention at all as their K V don't depend on any part of the input. 23:01 It may be because they moved lots parms around: finetuning MLP layers works so much worse than finetuning QKVO. By having MLP inside they have more parms to play a role on multi head attention which is still used. It also might be training. Their results at image classification feel feel more revealing than NLP tasks: dataset is way more fixed than "pick your favorite part of pile, filter bad parts" and models have the same weight class, and there is no different tokenizers(they use same with pythia so comparing ppl against it doesn't raise eyebrows) In ideal world they would also train at least pythia 160M from scratch on the same training data, but considering it's "up to 300B tokens", it's not exactly surprising they didn't. Also in ideal world they would put in ablation study "hey, let's not add non-linearity to Q K V proj and instead of new weights for 3 P-attentions we simply make dimension of heads bigger keeping around the same parm count" Also Pythia has an outdated MLP architecture: it use gelu, not geglu. P attention doesn't use GeGLU as well, but their architecture replaces MLP which already is replaced in modern LLMs. Speaking of their results in images: Their winning model in table 2 image classification has more parameters that ViT. If it's excluded, their small 86M model loses to another 86M model, though their 307M model still wins. (Also I remember trying somewhat similar but more naive idea on imdb dataset: replace Q,K,Vs proj with self attentions. Idea was walk toward super hierarchical nested attention: if you replace one Q projection with attention, you can go deeper and replace projection inside nested self-attn of Q with another attention and once walking over this inception is done the initial layer knows so well what token it has to attend to other token. Worked awful. Results were worse, number of parms went through the roof(as single proj got replaced with several, though will not be surprised if better hyper parms can make it at least bad rather than awful)

I think in pattention (cross-attention), the tokens can kinda communicate with each other, so it does not act like an MLP layer.

Thanks

Great vids! But I didn't figure out how the training process is accomplished?

Thank you for this great video. I was wondering if you could please also share the notes you wrote.

Thanks for making this video! These are very helpful for cold-start students like me - coming back to reading papers after a long time :)

Lol

noice

Thank you!

Thank you bro. Great explanation!

Firstly: Congrats on your Cottention preprint! Thanks for covering this paper. It flew under my radar, but I've been looking for an analysis just like this. I've long suspected that most attention heads care more about semantic than positional information, and that RoPE has only been successful because bad positional information handling hurts more than bad semantic handling. At worst, RoPE forces purely-semantic key/query features to be split over 2 channels instead of 1, so that they are rotationally invariant. Unfortunately this wasn't tested. With Gemma-2B's enormous head dim of 256 (vs 128 in Llama and 96 in phi-3.5-mini, 64 in Falcon-7B and TinyLlama-1.1B), it's one of the least sensitive models to this channel redundancy effect. The p-RoPE test has an unfortunate flaw: they truncate frequencies instead of redistributing them. The 0.25-Rope test has a maximum frequency of 2500, so the lowest frequency component actually loops before the end of the 8k context! Positional confusion from this might explain the worse result. I'd also love to see an analysis of a network with learnable RoPE frequencies. However, another perspective is that hybrid SSM+Attention architectures get almost no benefit from RoPE. Jamba didn't need it at all, and GoldFinch only needed it to work around a post-training position extrapolation issue in RWKV. RoPE might be obviated soon if hybrid architectures catch on.

so well explained. You saved my day. Thank you so much and keep posting stuff like that

Thanks you a lot for sharing this! It helps me a lot

congrats on writing a paper! i notice that another recent paper from NVIDIA uses a unit vector for attention (nGPT) where the dot product is naturally equal to cosine as the lengths are one. are these two works related to each other in any way?

I wish you would have left your comfort zone and started talking more about empirical results or conclusion.

@gabrielmongaras: Hi, thanks very much for the very detailed and easy to understand explanation. Just one thing, for me it isn't super clear from the video why with the parallel pattern the error compounds. Thanks again!

Thanks so much!

Thanks for sharing! Struggled a bit with understanding flows, but you explained everything really nicely

Thanks a ton mr random internet guy, wonderful video

best video out there!

I stopped watching when you named the image height "L", BLASPHAMY!

does the acceptance meaning choosing n tokens from n heads where each head has k tokens to offer? how is the sequence decided then? is head1=position1, head2=position2 and as such?

Thanks for your helpful explanation <3

Am I correct in thinking that the rotational embedding goes from 0 to 360 degrees? In that case, wont the first word of the sequence be very close to the last word in the sequence? Did they account for this?

great piece of content, thanks for sharing! Cleared up the understanding for me

The video clearly describes the ideas in the paper, which is great. However, I’m a bit confused about one part. It seems there is no distinction between L∗(C@B^T )@X and L∗(B ^T @X)@C.

What is the application you're using to annotate or take notes?

im at the step where i am running the second cell in the infer-interface file. it keeps giving me an error that says something along the lines of: ImportError: Using `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes` However i then did that command in the terminal and in the notebook, and it still gives the error, any suggestions? Also thanks, great video for my use case in general, i hope to be able to do some good work after i can get this practice down

Technically impressive that its possible however I only see limited application for this. Pactically most models below 8 bit quantization are way less "aware" of input and context. If I alter a situation ina 8b model it can adjust its output accordingly any model below that is very rigid. That being said maybe you can metigate those effects when you train them on that quantization to begin with and not compress it when it was trained on higher values if it makes sense what I say...

34:00 wait, how? I don't think "(L。QK)V" is equal to "Q*cumsum(KV)"

How is the equality in DDPM established in 17:49?

Isn't the recurrence A_t@h_(t-1). So a_1 should be multiplied with h_0 and so on?

Good video, we need these kind of advanced paper explanation.

You did an excellent job in your explanations! I can tell you recently had to grasp the topic since it's pretty new! I'm curious as to how you think the architecture should be expanded to larger problems?

great one, really liked it thanks

My only topology knowledge comes from "Experiments in Topology" by Barr where one chapter was something like a court case about making holes and using them and repairing or something like that. Book was written in 1960s and I read it when I was teen in early 2000s. The book covered the most basics of topology and was quite fun even for teenager me as it had fun exercises.

Huge thanks dude, this explanation has saved me much agony.