Gemini 1.5 Pro: This video is about scaling transformers through sparsity. In the video, Erwin and Barry discuss a new approach called switch transformers, which is a simplified mixture of experts variant along with some other improved training and fine-tuning techniques. The main points are summarized below: * **Motivation for Sparse Transformers**: Large models perform better, but training them is computationally expensive. Sparse transformers address this challenge by applying different weights to different inputs, resulting in less computation needed. * **Switch Transformers**: This is a new approach to sparse transformers that replaces some feed-forward layers with a switch layer. The switch layer routes the input to different experts (sub-networks), and only the output from the most probable expert is used. * **Training Sparse Transformers**: The authors propose three techniques for improving the training of sparse models: * Selective precision: Trains the models in lower precision formats, which are faster to compute. * Initialization tricks and training tricks: These allow the models to be trained more stably, especially as the models grow in size. * Starting from a known good sparse model (mixture of experts) and slowly expanding to more complex architectures. * **Properties of Sparse Transformers**: The authors find that sparse transformers can have similar pre-training perplexity to dense models, but perform better on knowledge-heavy tasks. However, they can underperform on reasoning-heavy metrics. * **Fine-tuning Sparse Transformers**: The authors show that sparse models can perform well on downstream tasks when the flops (floating-point operations) and sparsity are scaled appropriately. * **Multilingual Training**: Sparse transformers are particularly useful for multilingual training, where experts can potentially specialize across languages. * **Distillation**: The authors propose a technique for distilling a sparse model down to a smaller dense model. This can be useful for reducing the number of parameters needed to serve the model. Overall, switch transformers are a promising approach to scaling transformers through sparsity. They can achieve good performance while reducing computational costs.
![[LIVE] : ONE ลุมพินี 86 | คู่เอก "คมเพชร vs ชาติพยัคฆ์"](http://i.ytimg.com/vi/m3q_dRYDuU8/mqdefault.jpg)
Good job drinking from your cup without muting yourself.
MUTE YOUR MIKE when sipping coffee
Gemini 1.5 Pro: This video is about scaling transformers through sparsity. In the video, Erwin and Barry discuss a new approach called switch transformers, which is a simplified mixture of experts variant along with some other improved training and fine-tuning techniques.
The main points are summarized below:
* **Motivation for Sparse Transformers**: Large models perform better, but training them is computationally expensive. Sparse transformers address this challenge by applying different weights to different inputs, resulting in less computation needed.
* **Switch Transformers**: This is a new approach to sparse transformers that replaces some feed-forward layers with a switch layer. The switch layer routes the input to different experts (sub-networks), and only the output from the most probable expert is used.
* **Training Sparse Transformers**: The authors propose three techniques for improving the training of sparse models:
* Selective precision: Trains the models in lower precision formats, which are faster to compute.
* Initialization tricks and training tricks: These allow the models to be trained more stably, especially as the models grow in size.
* Starting from a known good sparse model (mixture of experts) and slowly expanding to more complex architectures.
* **Properties of Sparse Transformers**: The authors find that sparse transformers can have similar pre-training perplexity to dense models, but perform better on knowledge-heavy tasks. However, they can underperform on reasoning-heavy metrics.
* **Fine-tuning Sparse Transformers**: The authors show that sparse models can perform well on downstream tasks when the flops (floating-point operations) and sparsity are scaled appropriately.
* **Multilingual Training**: Sparse transformers are particularly useful for multilingual training, where experts can potentially specialize across languages.
* **Distillation**: The authors propose a technique for distilling a sparse model down to a smaller dense model. This can be useful for reducing the number of parameters needed to serve the model.
Overall, switch transformers are a promising approach to scaling transformers through sparsity. They can achieve good performance while reducing computational costs.
haha, I love the coffee slurping.
would love if slides are available, tried to find it on their website but no luck :(
Barret Z is all over this topic. Barret I hope you’re correct, I’m betting on you here.
20:10 plasticity
Wow
😂😂😂😂
pr໐๓໐Ş๓ 🤷