วีดีโอ

nice vedioeasy to understand the principle

Is there an impementation somewhere

Are these models also good for segmentation ?

nice video!

thank you for the clear explanation

Thanks for the video Can you also do a video on how to use these models from github in google colab please?

Great explanation! It is interesting to see how attention matrices aid in interpretability research and in getting better representations! I wonder how this could be applied to other modalities (such as audio).

Sorry but this paper is quite a briefing to 1 bit LLMs and the video itself is not explaining anything more than reading it out loud. There are multiple questions like what is the viable option to train such models, how it influences activation functions and what's the real benefit here as it suggests without multiplications today's GPUs would not be required which is not really true there. And requirement for new optimized hardware is not really a cool path to go forward.

Thanks, but remove the music in background.

I love this approach. I created a version using groq and open-webui ! It rocks !!

It reminds me of BYOL, but with an enhanced training scheme

Another excellent episode. Your channel is underappreciated.

I think the math wizard is removed from huggingface

what is the background music btw

Thank you

How feasible is it to adapt BitNet b1.58's ternary quantization (-1, 0, 1) for quantum computing using qutrits, given the current state of qutrit-based hardware, error correction, and the development of specialized quantum algorithms?

Thanks a lot!

A little bit like how I’m using Perplexity that lets me refresh a response with a different model. Except I’m using my human brain to choose an ideal model or draw info from different ones. Or how back in the day, the best search engine was this one that used multiple services (dogpile? Can’t remember). Maybe my comparison is bad, but definitely look forward to a tool that combines multiple LLMs.

This channel is amazing! I feel inspired by the simplicity of the ideas and their results. So many low-hanging fruits!

thank you so much for all your complex explanation :)

hey do you think we can have more than 0.7 frame par second if you render only 500X500 with a 4090 as hardware

Great job on the video. I read this paper a while ago, and this is a great explanation. I hear the accent, so if you’re ever in tel aviv I’d be happy to meet up.

So interesting! 👀

Is it over already? I was sleeping. 😴

Thank you for making this excellent video about our work! Minor note: at the end you mention 18-minute training time for our instruction-following ReFT, but that number is only for the small 1K subset of Ultrafeedback (last row in table). It takes a couple hours to train on the whole dataset, but we wanted to show that ReFT is also data-efficient through that number.

Thanks

Is there any code associated with this study?

I believe this applicable only for single request? If you have change of experts, you will most likely have many experts active for various requests. Is my understanding correct? thank you.

Or else let's do Distillation + less bit transformer 😅

Interesting, but what does "JOINT" mean in this context?

Thanks!

Good content, but i think it would be better if you went a little more in depth into all the smaller details, for me it felt like it was going a bit too fast and didnt understand a bit of what was happening, so i would prefer if there were more in depth explanation of the technical details.

Is there benchmarking data available for larger LLMs like GPT4-Turbo or Claude-3-Opus?

The thinking around BitNet b1.58 is intimately tied to the .gif in the paper “Stanford engineers propose a simpler design for quantum computers.” See the short .gif in action. Funding for that research began prior to 2021. Funding was provided largely by the US Department of Defense. Guess who virtually IS the US military by virtue of having a $ 3 T market cap to keep secret projects, secret? Thats right. Microsoft.

Model weights will make a lot more sense

This is something I predicted would happen in AI. It's cool to see a concrete usage of it. Ternary computers are the most efficienty computers and base 3 is the most efficient base. So this isn't surprising. Read up on Radix Economy to learn more.

but how is the accuracy ?

We had a lecture about single bit neural networks at one of my uni courses, some 5 years ago. It was interesting.

Thanks!

Well explained! Thanks for the well-written script, it helped me so much. Keep going!

It's a bit difficult to understand your accent, probably because I'm not a native speaker. Do you consider using an AI synthesized voice?

wonder why people don’t use this approach from the beginning. It’s like LLMs in assembly language. And as far as I know, every linear operator has a kernel. The kernel means that a linear operator H always maps the zero vector to itself. When we use a computer, we represent the zero vector as a column matrix of n zeros. Since the layers of LLMs are in the same vector space, we have H\vec{0} = \vec{0} for any H. I apologize for my bad LaTeX, but \vec{0} is supposed to be a vector. It’s important to remember that 0 is the trivial element in the kernel. For example, let Z be the set of all integers, and let H be the multiplication operator. Then, in ordinary algebra, we have positive, zero, and negative integers. The operator is \cdot, not x. The multiplication operator is often used in quantum mechanics of many particles, where the vector space grows exponentially, just like the number of bits for multiple objects.

This might have advantage even more when we get dedicated hardware, since tri-state logic is already a thing in CMOS. A dedicated tri-state matrix multiplication architecture for this type of networks should be easy to engineer with modern processes. NVIDIA should be all over that.

🎉😊❤

what is that accent?

2:13

I just found your channel. It is amazing. Congratulations. Your numbers will grow soon, I am sure. Great quality and great content.

So in summary everything is either Yes=1, Never mind =0, No = - 1 If only women were so simple to work out.

1:56 is the "same peformance" with "perito improvement" just an illustration of theoretical prediction or actual model weights data from real trit-model?

This is great! FYI, you can create a model of your voice in ElevenLabs, do a voice-to-voice transformation, and out would come perfectly pronounced English. I found this out by accident because I created a model of Arnold Schwarzenegger's voice, but everything I made it say LOST the accent but kept his tone of voice, LOL