วีดีโอ

Please cover mixture of million experts

I'm not sure how to feel about FFFs to be honest. They do have nice properties in inference, but it's easy to forget that depending on the paradigm, some MoE setups have favorable training dynamics (taking fewer total operations to train the model), and I find it a lot harder to visualize how to actually implement FFFs off the top of my head because there's not really any educational content on the low level functionality of the technique. On the other hand, as noted, they are faster on CPU than MoE implementations, and you could actually imagine a situation where even if you were training on CPU, for some reason, if you were doing a style of training which was hybrid inference/training (ie: Let's Verify Step by Step, Scaling laws with Board games, etc) where you need to perform a lot of inference per training step and you're actually generating synthetic data for the model as part of the training process, you may actually come out favorably with FFFs, for example. I really do think that in the end in order for them to get adoption what they need most is educational content / articles / independent implementations for people to really get a feel for the things they can be used for.

I look forward to your videos so much, coughs and all. I love the high level overviews and your thoughts connecting various papers and concepts together. Feel better and thank you! <3

Interesting. Strangely the fast feed forward paper has only 4 citations. Good GitHub project implementation though.

Could be useful where latency is more important than accuracy. For example in real time object detection on edge devices.

this is very good for remote LLMs to run locally, less data to use less latency to run the models remotely

This has to create more hallucinations / easier adversarial examples, not nice.

Sounds like there needs to be a model trained on models to generate new models. Those models then compete the best one, then go to retrain the generative model.

how are you so prolific what the hell man

25:30 CoLA = Corpus of Linguistic Acceptability - grammar checking with a binary output: acceptable or unacceptable

Maybe I am just projecting my own hopes and dreams onto the first paper but it kind of sounds like a viable strategy to train an architecture that can decide how much it needs to think about an answer. It feels like this concept is almost builtin already into the architecture but just not being used in this way. Like, the process of narrowing down which weights to actually use for inference could stop at varying depths. In my mind this sounds almost like a kind of tree of nodes where you essentially have different parts of the underlying model at the leaf nodes. Is that actually kind of what is happening here? It might not be possible to figure out a good way for a model like that to reliably learn the right selection of weights depending on how hard the problem is though. It seems strange that this possibility wasn't mentioned anywhere. I guess I will have to go through the paper and probably also the code to see if this is actually possible? Because I can imagine a whole lof of practical implementation limitations and maybe the architecture can not even be bent into this kind of shape to begin with.

Oy

bro love youre channel

Omg I built one of these once! Didn’t use a binary tree, so I’ll have to try that!

Since the FFF Tree divides the input space into regions, and each layer is the input to the next layer, how would FFF interact with Grokking? We've seen grokking work by also forming regions within the parameter space. So could FFF regions help guide grokking and speed it up? Or am I confusing the input space regions and parameter space regions?

Built something similar with LSTMs a while ago where I thought to myself since attention heads act only on a sub space of feature vectors I could save parameters by breaking up the feature vector. I saw no improvement. (That’s my own fault though)

coolbeans

YEW

oi that's pretty cool

6:12 I think the down arrow is signifying a newline character, so that would be predicting: " mbf[" which basically means "a value of the array mbf, with proper indentation" which isn't *that* weird of a token

No need to bash the brain - the LLM is just a slice of some feature of the human brain.

MoE will also always end up with one modality per expert too even if you try, unlike now, to make the embeddings themselves truly multimodal. That's why you need a coherent single network without experts to get the most out of multimodality (true multimodal embeddings). RAG is the way forward I think for agentic behavior, where it also learns to process certain info more or less. As a thought process I guess. A diffusion type of effect where it constructs the generation one part at a time and then gives a final generation when it's finished.

Oooooh , I get it , your a 👾

Lets use emojis instead for emotional clearity😂😂😂

#1 yapper

You need to show us your LLM training rig sometime.

I always had the impression that most of the model merging that is going on is kind of like brute-force on the benchmarks. You create a new model with similar performance but the way you merged them happened to adjust the weights in a way that it nudges the model to give more correct answers on benchmarks, not because of actual performance improvements but just by chance and because of prompt sensitivity. This of course only works sometimes which is why there are lots of terrible merges and some decent ones. Furthermore, if you actually use these merged models they almost always perform way worse in a more open-ended situation that isn't just a regular benchmark. But even though the model merging started out as a kind of benchmark-gaming, this paper seems to impy that there might actually be something more here. If you can use this to pick the best parts of different models in a relatively cheap way, then the idea of multiple diverging online-training models might actually be a viable way towards more rl-based training goals that interact with the real world. I really hope these techniques actually work out and are not just an evolutionary algorithm learning how to overfit on benchmarks because I wouldn't be surprised if that is still all that is happening even in this paper. We are current far too willing to take papers and their results at face value and look at them in a generally positive light. That is not how the scientific process is supposed to work. Science is about being less wrong so the focus should be more on figuring out which of our ideas might be wrong, and less on which might be right.

We dig out complexity of yesterday's axioms that gave it room for oder error we put it in our world tech & and material sciences if we don't re allocate affinities in the quest for eqaul measure it stagnates like all our grand theory in all feilds have . My ancestors was ready to math map the left over finger print code of life dna long before the Swedish scientist first identified it. General knowledge falls out in the laps of eccentric movements of thought then it has to fight the greater community who says stuff like it's just book or a rock no lost civilization is under it in turkey. This is how we are. Astronomical energy density measured in biology per scale is very impressive. It only took us reforming English digging up the past founding a nation industrialize the world to build computation. Had civil war ,ww1 & 2 plus infighting not occured leaving the eccentric fundamentalist Christian alone the transitor age would've occurred before darwin . We would've had usa shining capital on the hill long long ago. Whatever best helps simulate fine-tuned atoms lattus structure and body along with longitude latitude really hasn't taken off in hardware yet, I would imagine this is years out before form & shape products and services are realized & built . Maths is obviously a necessity In set up more so than after the fact architecture .

Thanks for doing all of this man. I'm using this along with some advanced distilling prompts to document all these.

0 views in 1 minute? Tunadorable fell off 😔😔

I like your idea at 9:00, that would be useful for training large numbers of experts very sparsely

The problem with MoE isn't necessarily RAM, but memory bandwidth. You can think of the problem in terms of arithmetic intensity (FLOPS/Byte). In the case of batched inference, you have a higher arithmetic intensity using dense models since the weights are shared across each token / batch. However, with MoE, you can't guarantee this behavior where in worst case each token needs a different set of weights (i.e. minimal arithmetic intensity). And with modern hardware, the bottleneck is memory bandwidth not compute (i.e. why flash attention is effective). Merging segments like this is only going to exacerbate the bandwidth issue, especially if each token in the batch uses a different weighted combination of experts. One solution to this problem was proposed by the switch transformer (and what GPT4 is likely doing), where the experts live on different GPUs and are routed over the NVLink rather than data dependent memory reads, but that also won't work for merging the FFN. This work is still interesting though, and has stronger applicability to summarization rather than generation (e.g. if you trained the experts without a fixed fallback dropout, maybe you could use a common FFN for generation, but use MoE for building the KV cache?).

Loving the more assertive tone.

Where's my "oi" in the beginning of the video? 😢

This is interesting. I find the segment length of 256 also surprising. It does seem very long. To me it didn't seem like having a fixed segment length is required by the approach, so to me it would make sense to have a maximum segment length and additionally split segments in text for example with each paragraph or sentence end or new line with a minimum segment length of something like 16 tokens or so. The way it is done in this paper, the segment boundaries do not care if there was a semantic shift in the middle of the segment, which I would assume is not good for expert specialization.

Can we think of MoE models as being like dense networks trained with an extremely strong sparse/dropout process, but where this network ablation has very strong correlations rather than being sampled from white noise?

Did they mention whether their technique reduces information squashing, like you covered in that other paper this week?

Hmm... splitting on spaces.... in a non-LLM programming context, that's called... (wait for it)... tokenization.

Fascinating. So using the combo of norm layer and gradient descent is sub-optimal in terms of computational efficiency? The more you know.

Reminds me of the transformer/attention permutation invariance that David Ha used for reinforcement learning