Phi-2, Imagen-2, Optimus-Gen-2: Small New Models to Change the World?
ฝัง
- เผยแพร่เมื่อ 12 ธ.ค. 2023
- Phi-2 is a tiny model that could fit on a phone, but it outperforms huge language models like Llama 2. I explain more about how it was made and what it means. Then we see Imagen-2, the most stunning text-to-image yet, at least according to Google' images. We then glimpse Optimus 2, smaller in the sense that it's 10kg lighter! With more degrees of freedom, it's movements look a lot more humanoid. And then the full launch of AI Insiders, plus a recap of why we shouldn't use the MMLU to 2 decimal places!
/ aiexplained
phi2 now on HuggingFace: huggingface.co/microsoft/phi-2
Bubeck Video, min 19: • Textbooks Are All You ...
Phi 2: www.microsoft.com/en-us/resea...
Shital Shah: / 1734882570603753814
Shoggoth: / 1702488701782352097
Mamba 3B: www.together.ai/blog/mamba-3b...
Phi 1.5B: arxiv.org/abs/2309.05463
Phi 1: arxiv.org/abs/2306.11644
Microsoft Prompting: www.microsoft.com/en-us/resea...
SmartGPT Video: • SmartGPT: Major Benchm...
The Information: www.theinformation.com/articl...
Imagen 2: / 1734954295655534780
deepmind.google/technologies/...
/ 1734763060244386074
Greg Technology: / 1734544659953623509
Swyx: www.latent.space/
AI Engineer: youtube.com/@aiDotEngineer?si...
Shawn Wang: x.com/swyx?s=09
/ aiexplained Non-Hype, Free Newsletter: signaltonoise.beehiiv.com/ - วิทยาศาสตร์และเทคโนโลยี
Yet another amazing video! I really enjoyed your critical take on benchmarks like MMLU, this is much needed.
Thanks so much Sebastien, Phi-2 is an incredible model - have been testing it for many hours - congratulations to you and the team! And yes, am looking forward to new benchmarking standards for 2024. Thank you again for speaking yesterday.
Philip doing public work by fact-checking the MMLU WHILE creating all this content?? Impressive work, you're one-of-a-kind in the AI vulgarization field, congrats!
Thanks Diabloto, I am very LLM-curious
@@aiexplained-official major credits!!! Hope you sent a link to this to all those companies!
@@aiexplained-officialhey this MMLU flaws are crazy, could you share the doc of inaccuracies for others to go through?
@@sumanthbalaji1768I found a Medium post from August, "Errors in the MMLU: The Deep Learning Benchmark is Wrong Surprisingly Often," but that seems independent work by a Daniel Erenrich.
@@skierpage yes I went through that blog too, doesn't have this document of errors
You honestly need to publish a paper on the errors in the MMLU. This needs to be seen by academia.
100%
No paper needed, they should watch this video and immediately build a new test from the ground up!
They know lol
please please publish, but please prepare to be attacked for your honesty
I feel as though it is slightly ironic that the Ai's intelligence was held back not by their own way of learning, but by our inaccurate datasets.
It makes a lot of sense though. We tried to make it equally good at predicting experts AS WELL as predicting average Joes AND raging lunatics. Of course that task is much harder than just predicting experts.
I still can't believe the MMLU isn't being called out by people, at least. It's been so long since you found those problems, that I won't accept that people don't know about the issue enough to have it thrown out by every benchmark set using it.
Thank you again for your great work. Cheers.
Thanks rant. I thought so to and then up it pops with Gemini, front and centre
@@aiexplained-official what did the authors of "Measuring Massive Multitask Language Understanding," Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt, say when you contacted them?
Increase efficiency!
Well, at this rate this time next year we'll have a locally runnable AGI.
I'm more excited about how much better the largest cloud ones will be
Imagine that 😂, I'm calling mine hal
5 years from now we probably will
@@BoufonamongI'm calling mine Meg. Short for Megatron.
AG-phi?
I love that you're an informed AI enthusiast, yet you're not afraid to criticize -- and to do so forcefully -- where you see the need. It's a mark of integrity.
Thanks Clay
Having seen your previous work on the MMLU that graph that declared a .06 PERCENT breakthrough made me burst out laughing.
We need an MMLU 2 or something to that effect yesterday, and I'm starting to suspect the only reason we don't have it yet is that nobody wants their fancy benchmark numbers to go down, even if they would be more accurate.
Re: Phi-2, I am happy to see that synthetic data is getting more love, as opposed to the models that just use mass scrapes of any data that isn't tied down properly.
Thanks for looking into the benchmark data, they were too opaque up until now. Whenever a model scores impressively on one we should dig into it to know whether it really means it's good at X subject or if it's just good at making the same mistakes.
As a new Patreon member, I'm here to tell you how amazing AI Insiders is. Phillip's research is impeccable. The Insider info is priceless. Those of you who make your living with AI - do yourself a favor & budget the $30 each month to support Phillip. Everyone will eventually be making their living with AI; if not today, very soon. You will need quality, authoritative information upon which you can make important decisions. AI Insiders will provide you with AI news that is second to none. If you have not already, join. Completely worth the money.
Such high praise, thank you so much. If you like what's there, in 2024 you will be even more impressed!
The big big issue most people are currently overseeing is that all those benchmarks are in english. The data is in english. The models are heavily optimized for english. GPT3.5 and GPT4? Speaks about any language it has gotten some data for and also provides excellent results for tasks in those languages.
Great point
Yeah I am Swedish and will be truly scared of an automated workforce when these LLM:s speak and understand smaller more local languages fluently. GPT-4 is decent at it but not yet good enough for professional use
Very true. I haven't seen a model with less than 70b params yet that really impressed me with its performance in my language
though that is a very big problem i'd argue the larger problem is 'poisoned' models basically trained to tackle the benchmarks rather than being actual general-purpose models
Given that multi-modality seemed to boost performance I wonder if multilingual models would also be boosted.
For the record, you have the best thumbnails in the game.
Aw thanks Cal, often get criticised for them and hundreds of offers to pay for thumbnail services but I love them too. Minimalist.
thanks for having me as your first Insiders speaker Philip!
And thank you so much swyx. It was a great talk and laughed at the intro!
The pace of the race toward the edge of cliff quickens
Gemini Nano hit the punch line 🥊
Time to fly.
Hmm, but synthetic data is good for safety no?
@@aiexplained-official Maybe this model won't be a Shoggoth, but there are a lot of ways things may got wrong. E.g. because we will get extremely powerful systems without proper defence mechanisms against misuse, or things like instrumental convergence.
i think that smaller models giving out 100% accurate information to a general, bigger AI capable of understanding and finding anomalies in the process, or be critical of the result is the real AGI
i saw today "teslabot 2", im very much interested in seeing AI and robotics in everyday life
a lot of insights as always, thanks!
Maybe the way to go is finding ways to make models smaller and more efficient to the point that they could run on local devices instead of big datacenters rellying on clound and internet connection and higher latencies (Cloud would never work to make robots work properly)... Yesterday I was reading about liquid neural networks and how they can do the work with just a few neurons, it seems promising to shrink really large NNs into much smaller and faster ones especially for vision, videos, images and audio/speech recognition, for robotics LNNs can handle vision better than current neural networks and run fast even on small devices such as Raspberry Pi because it needs just a few neurons to do the same task as a really big NN based on other architectures do. LNN are very small and have plasticity to adapt to new situations without needing a new training process.
They should make Phi-3 a 7B model. If Phi-2 is a quarter of it then increasing by double should make it even better, and 7B models are runnable on 90% of computer hardware.
Their priority is probably phones.
yeah, i think it's especially important for decent models to be able to run on low-end phones so that LLM access isn't restricted to the first world @@berkertaskiran
@@boris6237 Oh wow, that's an incredibly important argument, had not thought about it like that and I really appreciate you sharing that perspective!
Don't quote me, but a 7 billion model would probably use about 6 GB of ram
@@QH96and iPhone 15 pro max only have 8gb ram . And IOS sistem aredy use 2gb of ram
I was impressed by the phi-2 code perf
Is there any effort to actually correct the MMLU? If not, why not? What would be required to get these corrected? I feel that this is a serious problem and it's disturbing that the MMLU is continuing to be used without correction.
I’ll be honest, I stopped watching your videos for a while because they caused me too much anxiety - but when I then look at some of the other things going on in the world, it is actually comforting to hear about AI. Congrats on your meteoric growth in 2023.
Thanks for coming back heinkle! Great to have you here
TL;DR If the data set trained for these 'gpts' would actually be accurate, we'd have even more impressive models overall. So not even changing the training method, just the data you can get way better models
I wonder if a large model with a big context window would be able to spot inconsistencies and mistakes in training data. I saw a documentary where a AI presented with logical inconsistencies went into a garbled "Does not compute" loop and eventually caught fire, so maybe it's too dangerous!
Idk, how can you determine if something is right or wrong if you learn the wrong thing
in the dataset. I think the best would be "smaller models" used by a bigger model.Where the smaller models are used to detect inconsistencies within the dataset.
You train the small models with 100% accurate data and you teach them to spot right/wrong answers, and that's their sole purpose, and they will find every mistake in any dataset. So a model for math one for chemistry one for biology,etc. Then the bigger model could access through api these mini models and get the results from them and recreate pdfs with "correct dataset".
I think it's safer that way, when you have a big models it's harder to "control" and know what he actually knows. And to make a model that have perfect data for code,math,physics,etc. It's basically the final product we want, but to obtain that we need to curate the data we have, and fastest way to do that is a smaller model. Then once all data is curated, we use that for a bigger model. I spammed q_q oops. u get the point.
@@skierpage
@@harnageaa symbolic AI tried to develop AI by teaching systems only the right answers, and it's utterly failed to keep up with transformers. One of the great things about LLMs is they can handle inconsistency and exceptions: "Water is wet" (ice), "Palestine is a state" (disputed), "An asteroid killed the dinosaurs" (generally accepted), etc. Learning everything includes ingesting megabytes of the "wrong" things; again, I want to know if an LLM can be aware of discrepancies while or after it trains.
The most value delivered per minute on TH-cam
Thanks so much jawadur!
Amazing as always! Whish regular media was half as good at divulging complicated topics, this channel is gold
Wow thanks pablo
With AI Insider, you really are creating a lot of content. I do hope it turns out you were AI all along!
Haha not quite, a hardworking human!
@@aiexplained-officialthe way you pronounced “Imagen” made me question if this was an AI voice as
well lol but I think you’ve left enough small clues to prove your humanity 😂
@@aiexplained-official thats what you think
@@aiexplained-official certainly hardworking! Thanks for all your explainers, they really help stay on top of the fast moving world of AI.
Amazing video, as always. Have you contacted any of the people in the field about the erroneous benchmarks? Do we know if anyone is working on it to create new ones or fix them? I can't believe they don't know or care about it but the problem is still there it seems.
Yes, and people are. There are better benchmarks coming out all the time, hence my surprise at this MMLU d-measuring
7:06 such a legend
Hi Philip
just wanted to say thank you for still pumpin out so many videos despite your patreon contend probably also being a ton of work
Thank you so much for keeping us informed!
Haha, no thank you for supporting on Insiders. It's what keeps the main channel going!
We have all seen the Boston Dynamics robots doing incredible things, but the scripting and trail and error involved to make those incredible videos is insane. And lets not forgot that the Atlas robot is huge. Are we actually meant to believe that Musk's Optimus robot is "as described"? AI powered, and physically capable of all the actions its shown doing?
This time next year
Anything Musk hypes up should be taken with a laaarge grain of sand
Billy Madison to the MMLU, "I choose: Business Ethics." 😉
As usual, a fascinating video with revealing insights that are seldom if ever found anywhere else. Great job mate and look forward to more
Thanks so much Dylan!
I absolutely love your videos! They deserve each and every view and like!
Thanks so much Dream!
Thank you so much for investigating problems with testing.
Great video, but why not mention Mixtral? Are you still experimenting with it?
First I think phi2 is more significant but also to cover it properly would be a lot more work, there's only so much time in the day!
@@aiexplained-official Fair point, can't wait to see your video on it.
Can’t wait to see Mixtral in the mix!
Sounds great for general chat conversation
Thanks! Fantastic content, as always. 🙏🏼
You know they are flexing their ability when they show hands :D
Haha
great vid as always, strapping in for next year's craziness
always a good day when phillip ai uploads
Phi-2 is a tiny model that could fit on a phone, but it outputs huge language models like Llama 2. (from the caption) outputs should be outperforms!
Thanks freek, missed that!
As always great video! Very informative! Many thanks to you!
Thanks so much, as always!
AIs are experiencing the real world of academic exams. I remember several times in college where textbooks were wrong, exam questions were ambiguous, or we were told to give outdated or blatantly wrong answers to pass tests and get good grades.
when will they learn? rubbish in rubbish out. Thanks for all your efforts to bring awareness to this issue of testing quality.
AGI coming to a mobile device near you in 2024 thanks to synthetic data. Is there any safety checking done on this data?
Well it's synthetic so shouldn't be as bad but I was still surprised that there was toxicity at all, maybe I shouldn't be
Better data, better models...makes sense.
You are beautiful sir. Thanks as always.
Wow thanks Aaron, that's cheered my spirits
I don't know what's more concerning, the fact that those models did so good at flawed test, or that they might be much more capable than we think.
Wow that was surprisingly sudden end to the video
The wonderful day came earlier!
Can’t appreciate your work enough! Thank you.❤
Needed to hear that, thank you
Great video again! Please do a video on synthetic data. I get that this will likely decrease toxicity but what else will it do. If, for example, Microsoft is building the synthetic data, does that mean that we are training the AI on Microsoft’s view of the world? One can imagine how this could be influenced by all sorts of commercial imperatives. Will synthetic data make models more and more similar to one another and perhaps less interesting?
I don't think less interesting if you ensure diversity - see original phi1 vid
Thank you! I hope you dont reduce the ammount of great content here on your channel. Its invalueable.
Hopefully this video is evidence not!
Amazing video as usual, thanks for the fast update
Thanks nacho!
Thanks for another informative video!
Thanks so much John, means a lot
Thank you for sharing your time and work Phillip, I responded to one of your tweets, by asking if you knew what is going on over at liquid ai, the new year is fine and by the looks of it your going to really busy, but if you get a chance I'm curious as that's where Joscha Bach is working at right now. Merry Christmas to you and your family and all the other family helping you with this great work, and a Happy New Year.
Merry Christmas Bill, will check it out, cool name at the very least
Great video just one question
Do you feel the AGI?
The others think you should no longer be on the board. It's not my decision, really.
Best yt channel
Great video, like always.
Thanks matus
Amazing content! Don’t get how you don’t have a million subs
Aw thanks Muhammed
Jesus crist. So many implications from this, and will result in massive improvements. Thank you so much for pointing this out as it will really slap AI into hyper drive.
if this was SDXL. i'd give the image a 9/10 the problems are:
-the eyes ( they are not pointing in the same direction)
-the ear (it is just weird)
-the lighting is wrong (the leafs are lit from behind the subject and the subject is lit from the front)
-her whole right side is a bit wonky
- the 1 strand of hair in the back is weird. 7:33
PS if you want to see the best SDXL models, use the ones over at cvitai and not the stablity ai's (the 1.0 model is still the best you can get from there). Just pick the "JuggernautXL" or " DreamshaperXL" as they are SotA for XL.
PSPS Other then the part about imagen-2 this was a very good video. Love your dedication to the craft of making ai news without the hype.
Thanks tomas, your professional eye caught much more than me, apologies
@@aiexplained-official i ain't a proffesional, but i use SD. these things are just what you train your eye for
really? Dreamshaper is SotA? I knew about Juggernaut but i remembered dreamshaper's earlier non-SDXL versions as kinda worse than some alternatives. Will have to try it out
WyvernMix was another that impressed me
@@maciejbala477 the XL version is at least close to the sota for trubo
Or at least it was late last week
Small models ftw! If I cant run it on my phone or self-host it (without really expensive GPUs) then 90% or use cases just don't work.
Models are flaky enough as they are. Add to that the unreliability of some companies' APIs, we need self hosted solutions we can fine tune. (Not to mention privacy issues)
Great video!! many thanks. On the topic of generated images of human faces: Look at the shirt collar (and ears and ear-rings if you can see them), instant give-away. The face is phenomenal...but textile manufacturing is apparently a harder problem.
Nice spot
This explains why I did so badly in my medical exams. The college has all the wrong answers :-)
True true
I can not thank you enough! I will definitely support you on patren when my finances allow it! THANK YOU FOR GUIDING US THROUGH ALL OF THIS, YOU ARE THE BEST!❤
Thanks so much, no worries on Patreon, your kindness here is enough!
Me thinks the Phi models were named after you. :-)
Haha, very kind of you to think it!
I think if we can find a formula, no matter how long & complex, that can be the 'Engine' for such a training, it will change the field.
0:00 “You my have thor”
Subtle t in there somewhere
It would be interesting to take a few LLMs and ask them to evaluate the questions: are they clear, are they ambiguous, do they make sense?
For the benchmark, it is common to include wrong or ambiguous questions so catch training leakage. It should not be possible to get a 100%.
By this time next year, I expect to see a continuous stream of AI Explained content generated by Phillip-2 or I'll be really disappointed.
Haha, I will be human-generated to the end
Apprciate the updates as always, I was wanting to look more into the MMLU since you mentioned people still using it and thought I'd go back and watch your video on it, but it's not in the description, might be a good idea to put it there since you played a bunch of it at the end here. I assume I'm not the only one that might want to look more at that part.
Anyways ty and have a good day :)
edit: also just a thought, has anyone compiled an exhaustive list of the issues in the MMLU test? And if so does anyone have a link to that list?
Hey dc, thought I put it in there somewhere! Can search mmlu broken benchmark too. And no, to the best of my knowledge this channel has shown the biggest repository of mistakes in that benchmark
Always the best videos!! Have you considered making a patreon tier where some of the funds go towards a google colabs backend for the patreons to use, depending on their subscription amount & time?? Given how little resources were used training Phi-2, it could be a good idea to let people experiment with the concepts shown in your videos, as well as more exotic variations in model architecture such as cyclic attention heads, sub networks etc..
Finally a new video!!!🎉🎉🎉
TOP FRICKEN NOTCH MY FRIEND, THANK YOU!!!
So kind KP!
Welcome HAL9000 and Skynet !
Is there a way to access the document you've compiled of the errors that you found in the MMLU Benchmark? i would like to source it for my bachelor thesis, if that's possible.
Feels like we're a part of those vaguely touched upon histories in AI movies 😅
Haha, nice way of putting the strangeness
I'll ask the question that everyone's curious about - how are you able to 1) access, 2) digest, 3) synthesize, and 4) produce so productively???
Will do a video on that someday! And don't forget the hours of content (researched, narrated and edited by me) for AI Insiders at the same time, plus sourcing and conducting interviews! And comment replying!
AGI must be helping you with the details :D @@aiexplained-official
Brilliant!
By my estimate it cost about $70k to train. However the real cost is preparing the data.
Is it rational to say that if Google and OpenAI are using the MMLU benchmarks in this way without acknowledging the benchmarks problems that they are behaving too naively to deserve public trust to try and solve the alignment problem?
It's so blatant once you point it out that I find it very disturbing no one else talks about it
The two issues seem unrelated. The numbers game to two decimal digits is stupid when the benchmarks are 1% flawed, and training to the test when the test is bad may degrade models' real-world abilities, but what does that have to do with alignment?
Hey Philip, dunno if it's me watching this at 6am but this video felt a little fast and stressful. Do you think you could integrate a 1 to 2-second pause before switching to a new topic to give us time to digest the information?
Thanks for the feedback David, will bear it in mind!
Agreed. We poor biological intelligences need a longer pause between concepts to integrate the information.
It's like we need a smidgen of noise between the signal
Soon we’ll have to get the LLMs to not only generate the next update’s training data, but to prove to us the labels are correct, because otherwise we are limited by what we think we know.
Tesla robot walking like "I just sharted"
Awesome video! You did a great job exposing MMLU and how shady it is. I agree that people should stop trusting it as a measure of capabilities. What about MoE and Mistral?
Thanks AICA, still investigating !
I'll speak only to what I know. That project sounds amazing, I wish I was into VC, I'd buy in! Tbh, most GenX weren't taught ANY entrepreneurship if we went the corporate/govt career. I'm sure you have even bigger plans, depending on what sticks. _Putting decimal places on data with ~? confidence intervals is how we manipulate ofc._
MMLU...WTF? (Merry Christmas/Happy Holidays)
Just wait until Jan 8th. No meetings on the last week of December or first week of Jan. All the AI researchers have extra time to work on their pet projects and keep the compute famrs running over the holiday; I expect to see some interesting results, it's always the most productive time of year.
Could you add timeline sections to your videos? I'd like to see an outline of what topics exactly are being covered at a quick glance
When I add timestamps YT doesn’t automatically segment the video, wondering what I am missing
Please make a cleaned (remove or fix questions) MMLU bench as a PR to Eleuthear's evaluation benchmark :)
Some fixes better than none...
You can't just change a benchmark that is already widely used. It would create confusion when different models are tested at different times. And produce results that aren't comparable to each other. It needs to be a new benchmark like "MMLU 2"
@@Houshalter I mean, arguably it's pretty worthless in its current state ... I suppose it could be its own bench, or v2 or 1.5 or whatever, but seems better to fix it somewhere than to just say it's bad, since it's gonna get used anyway...
comment for the algo. keep up the great videos :D
Thanks zocker!
I wonder what you think about Mixtral 8x7b model? And what about new MMMU (multimodal) benchmark? Is it good enough?
MMMU is so much better yes. And Mixtral I am still playing about with, and investigating with the help of experts. The future looks bright.
@@aiexplained-officialI tried it on deepinfra site, looks good for such a small model.
Can you also talk about the newly released Mixtral 8x7b?
will you also make youtube shorts for those who don't have time for all the details?
Maybe one day! Do you find the videos too long?
4:30 Not many people are talking about the flaws in these benchmarks, e.g. MMLU. Perhaps we need another video on this?
I've read, heard and watched a lot of content for Gemini and very few mentioned any issues with MMLU.
For once I think a more clickbaity title is needed.
Haha, more so than the original 'Broken Benchmark Smartgpt' one!
@@aiexplained-official Hehe, indeed. Perhaps it needs spelling out more, including words like "MMLU" and not "SmartGPT". BTW, how is SmartGPT going?
@@KolTregaskes more to come on that front in 2024...:)
7:33 The ear looks weird, the system messed up the earrings. Even so the system is getting better by the day.
Is it time for Philip to create a bench marking test for AI systems himself?
16:46 “type of question that depends which source you ask” I didn’t catch this point. Anyone can elaborate?
The errors with the MMLU makes me think a good test for AI should have trick questions - questions without actual answers or lacking the appropriate option - to test the AIs ability to recognise when it doesn't know or can't find the answer.
I think the video showed that GPT-4 would give a better answer than any of the garbled multiple choice answers. I think you could engineer a different test-taking prompt where you prompt the AI to pick the best multiple choice answer but also point out when there's a problem with the Q&A.
One problem is these technical reports are drowning in a sea of benchmark numbers, so I'm sure the person cranking out all the scores to two decimal digits has no time for nuance or evaluation.
@@skierpage
While it is useful to let it answer freely, in terms of serving people AI should be able to work within constraints. Otherwise it will likely become just an advertising tool, always telling you to buy the industry tool to get the job done.
In an example specefic to me, I do a lot of python programming on my phone and ChatGPT often gives coding examples for libraries that don't don't work on my phone. So it's handy if we can give it a constraint - asking it to solve the coding problem with a specefic library - since we may want the best solution we can do right now than the theoretical perfect solution.
@@BradleyZS Make up your mind. Do you want to constrain the AI to answer a multiple choice question, or point out that it's flawed? What should the AI do in response to a sleazy lawyer: "Have you stopped beating your wife? Answer yes or no!"
@@skierpage
The ideal would be if the AI could recognise the intent of such questions. That it could understand that a leading question is intended to ascribe undue guilt to it, or that a trick test question exists to test the AI's ability to react to an impossible task.
Such an ability, I believe, is crucial for the progression of AI beyond the simple LLM. An AI should be able to understand the desire of the user, and in the context whether it should give the best answer or admit the inability to answer.
They gotta do something about this holy its unnaceptable