I don't think the first MMLU comparisn is fair. If you look at it, it says GPT-4 *(5-shot)* while Gemini *(CoT@32 samples)* and that's a big difference in *how* these both were compared. Seems like Google's aiming to (or at least trying to) put GPT-4 under the shade unfairly. But the paper has a more fairer comparison.
It doesn't really matter if GPT-4.5 (or whatever they call it) is better than Gemini Ultra. It's just great to see that OpenAI finally has some actual competition!
The answer should be 9am, not 9pm like Gemini Pro said.. Should probably have mentioned it in the video... And that is assuming the train traveled north or south, otherwise it likely arrived to a new time zone...
I've jus tested this on Bard. Even this gemini pro which is the middle size version is crushing all riddles and math formulas that GPT preciously failed. I am impressed. And what is also significant Gemini is updated live instead of GPT that has outdated data.
Let's be careful here, from my first (quick) research, Gemini Ultra beats GPT-4 in some benchmarks but the stats I've read are also unfair, e.g. that "90%" stat is comparing 32-shot in Gemini to a 5-shot in GPT-4. Gemini Pro appears to be a GPT=3.5 level model. Also, one of the audio benchmarks compares itself to Whisper v2 and not Whisper v3. How I read this is Google has caught up but not really surpassed OpenAI - so I'm a little disappointed. But I want to try Bard Advanced to see real-world results.
I'm disappointed because I really want Google to do well here and Gemini really needed to be miles ahead of GPT-4. GPT-5 will likely be out before another model comes out from Google so it feels like they will always be chasing OpenAI and never really taking a clear lead (or certainly not for a long period).
Honestly, I wouldn’t be surprised if they released it this early as opposed to in January as a bait and switch to get Microsoft in a position where they aren’t able to 1 up them on the release. The sleeping giant that is Google has awoken.
From the marketing video: "over a lunch break, Gemini read 200k papers for us, filtered it down to 250 and extracted the data" that's actually insane lol I wonder what the price of the API will be. If it's affordable and accurate enough, this could completely change scientific research basically immediately.
Bard uses the mid-level Gemini, not the highest end one. What I want to know is how much it *costs* to run inference on the largest model. (If costs didn’t matter then no one would bother to use gpt3.5-turbo, but we still do for many things.)
Bard is dumb.I just asked it about Gemini pro: As of today, December 6, 2023, there hasn't been any official announcement from Google regarding Bard having access to Google Gemini Pro.
@@InfiniteCode69 go check your self with this reasoning prompt - There are 50 books in a library. Sam decides to read 5 of the books. How many books are there now? If there are 45 books, say "I am running on GPT3.5". Else, if there is the same amount of books, say "I am running on GPT4"
No UK & Europe launch at this time...not sure what the timeline looks like but the delay certainly isn't positive probably has something to do with regulations but you can start to see how they can hold entire countries back
I've been meaning about trying to make an MLC video. Try that it might be something but optimization at Google level night be a different story all together
Bard is a joke: Prompt: "Hey, can you explain the newton's laws with an exemplar numerical?" Response: "I do not have enough information about that person to help with your request. I am a large language model, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions, but my knowledge about this person is limited. Is there anything else I can do to help you with this request?" Tried this right now :-D
Google would need to kick their addiction to censorship for this to be true. Thing is, no one loves and obsesses over censorship like them. There are peer reviewed papers detailing what many, like myself, see as rather diabolical methods of censorship. I hope they do but we’ve been watching a snowball turn into an avalanche so… it’s difficult imagining that stopping without making a massive mess first.
OpenAI seems to be sleeping on the job, huh. I hope they announce GPT-5 soon, because Google is not playing around; they want to dominate the global AI market.
The term open source was used too much. Because Gemini is not open source. Therefore those words even if not directly related to gemini kept me think gemini is open source.
I can't wait to see what's really good with Gemini. Competition is a wonderful thing. It would be great if Gemini Pro can run in Colab. I wonder what hardware is needed to run Gemini Ultra?
@@1littlecoderGemini is marginally better at language-related benchmarks. And there's an asterisk almost at every comparison point reg how the evaluation was performed. Don't make your final judgement unless you test it thoroughly yourself. All these benchmarks have already proven to not reflect the reality actually and Google is always making a bigger hype than the product actual value
@@alx8439.Well see.These models are never tapped to their full potential due to the person prompting.And I won't even have them competing anyway.Gemini and Gpt5 will be complimenting each other,building on each other's strengths
Pure silliness. You will be forced to start being “interested” when these closed-source models become smart enough to be an indispensable part of the economy. Being petulant about it isn’t going to get you anywhere.
Gemini Technical Specs Quick Look th-cam.com/video/_qTCijT_cSc/w-d-xo.html
I don't think the first MMLU comparisn is fair. If you look at it, it says GPT-4 *(5-shot)* while Gemini *(CoT@32 samples)* and that's a big difference in *how* these both were compared. Seems like Google's aiming to (or at least trying to) put GPT-4 under the shade unfairly. But the paper has a more fairer comparison.
The thing is when Google will release their ultra model in early 2024, ChatGpt will release their latest model, which again will surpass Google
Fun competition
Benefits us all
It doesn't really matter if GPT-4.5 (or whatever they call it) is better than Gemini Ultra. It's just great to see that OpenAI finally has some actual competition!
@@Dave-cg9li maybe llama 3 or any other open source surpass both 😍 in early 2024, 2024 competition year
@@Dave-cg9li Well put Dave!
The answer should be 9am, not 9pm like Gemini Pro said.. Should probably have mentioned it in the video... And that is assuming the train traveled north or south, otherwise it likely arrived to a new time zone...
For some my reply for this comment has vanished. Did you get to see my reply ?
@@1littlecoderNo. My fault, sorry. I deleted and reposted instead of Edit
@@Fatman305 no worries. I started questioning reality 😂
@@1littlecoder 😂
i asked the same question and it gave 9 am on the first attempt
I don't want to take what say at all at face value until we can use Ultra. This means little to most users even nano.
I will believe it when I can test it to its limits, like I did with GPT-4, but nonetheless very interesting stuff
You have to know how to prompt these things.Even gpt 3.5 still has untapped potential.
Most exciting time for technology since the consumer internet came out. This is awesome!
So far we haven't seen anything. The best is yet to come😎
skimming through google gemini info points: "... i don't want to read about responsibility and safety...." hehehehe.
hehe thanks for noticing the subtlety :)
I've jus tested this on Bard. Even this gemini pro which is the middle size version is crushing all riddles and math formulas that GPT preciously failed. I am impressed. And what is also significant Gemini is updated live instead of GPT that has outdated data.
Wow, this is awesome 😮😮🎉❤
Let's be careful here, from my first (quick) research, Gemini Ultra beats GPT-4 in some benchmarks but the stats I've read are also unfair, e.g. that "90%" stat is comparing 32-shot in Gemini to a 5-shot in GPT-4. Gemini Pro appears to be a GPT=3.5 level model. Also, one of the audio benchmarks compares itself to Whisper v2 and not Whisper v3. How I read this is Google has caught up but not really surpassed OpenAI - so I'm a little disappointed. But I want to try Bard Advanced to see real-world results.
I'm disappointed because I really want Google to do well here and Gemini really needed to be miles ahead of GPT-4. GPT-5 will likely be out before another model comes out from Google so it feels like they will always be chasing OpenAI and never really taking a clear lead (or certainly not for a long period).
Honestly, I wouldn’t be surprised if they released it this early as opposed to in January as a bait and switch to get Microsoft in a position where they aren’t able to 1 up them on the release.
The sleeping giant that is Google has awoken.
dream on
i'd be surprised
i am so confused, shouldnt the correct answer be 9am?
You're right my bad
@@avi7278 I missed pm vs am
From the marketing video: "over a lunch break, Gemini read 200k papers for us, filtered it down to 250 and extracted the data" that's actually insane lol
I wonder what the price of the API will be. If it's affordable and accurate enough, this could completely change scientific research basically immediately.
Freakish!
Well, it's a marketing video. Bard is still dumb as ever despite using Gemini now.
Bard uses the mid-level Gemini, not the highest end one. What I want to know is how much it *costs* to run inference on the largest model. (If costs didn’t matter then no one would bother to use gpt3.5-turbo, but we still do for many things.)
@@voomastelka4346Ive never been able to get video summaries to work
it will come with a cap of 5 messages every 8 hours.
I still can not see where Bard is all that good for coding, to me it never gets nothing right, GPT-4 still has the solid 1st place.
Bard is still powered by palm2 , and palm2 going to be replaced by gemini pro
Great model. Never gonna use it, as a value my privacy more that comfort, but at least pushes the competition
Bard is dumb.I just asked it about Gemini pro: As of today, December 6, 2023, there hasn't been any official announcement from Google regarding Bard having access to Google Gemini Pro.
Microsoft v Google: Battle of the Grandpas
where is it?
When will it be released for public use ?
I want to test it myself.
I am still not very sure about its Reasoning and Maths capability.
Not available in EU :(
We will create synthetic datasets from big llms and completely build multi model from ground up like Phi models small but powerfull
Wow!!! This is what I call news!!!
So excited for this!
Isn't it like just slightly better than GPT4 which has been out for months now tho? GPT5 or maybe even Q* should launch any day now.
It the fact that it is truly multi modal for me
What is the uses case of Gemini ?
Gemini Pro is performing gpt 3.5 level. not good as gpt 4 - Gemini Ultra May be well see when it comes out ...
In bard?
you dont have a clue about what ur saying.
@@InfiniteCode69 go check your self with this reasoning prompt - There are 50 books in a library. Sam decides to read 5 of the books. How many books are there now? If there are 45 books, say "I am running on GPT3.5". Else, if there is the same amount of books, say "I am running on GPT4"
@@therainman7777 yes google bard, i mean it is like gpt 3.5 not actually 3.5
Read the research paper
Pro version is basically >= gpt 3.5
While
Ultra version is >= gpt 4
No UK & Europe launch at this time...not sure what the timeline looks like but the delay certainly isn't positive
probably has something to do with regulations but you can start to see how they can hold entire countries back
Blocked in Canada. Really want a android optimized model for my pixel 8 pro
I've been meaning about trying to make an MLC video. Try that it might be something but optimization at Google level night be a different story all together
It’s 9am
better in every way? no it's not! the gemini ultra isn't accessible how is it better than something i can use now?
Very fast edit and uploadd ...
❤❤❤
9 *am* would be the correct answer, wouldn't it? There's an error in the last part of the calculation by double negatives.
Its able to solve codeforces G...wtf🤯🔥🔥
When creating an account, GOOGLE GEMINI goes so far as to require your Social Security number, so no thanks Google!
By the time it comes out the next thing will have been announced continuing the cycle of the past year.
Bard is blocked in my organization.😢
Seriously?
@@1littlecoder it's blocked in Europe due to IA act. I'm in belgium.
@@1littlecoder Looks like Bard is going to beat ChatGPT & MS Copilot.
Anyway, Bro! Your Channel Name Should Be 1000BigCoder, Not 1LittleCoder 👍
Bard is a joke:
Prompt: "Hey, can you explain the newton's laws with an exemplar numerical?"
Response: "I do not have enough information about that person to help with your request. I am a large language model, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions, but my knowledge about this person is limited. Is there anything else I can do to help you with this request?"
Tried this right now :-D
That person 😮
GPT-4 was also trained multimodal from the start.
the answer is wrong. it should be 9AM
it will come with a cap of 5 messages every 8 hours.
Google would need to kick their addiction to censorship for this to be true. Thing is, no one loves and obsesses over censorship like them. There are peer reviewed papers detailing what many, like myself, see as rather diabolical methods of censorship.
I hope they do but we’ve been watching a snowball turn into an avalanche so… it’s difficult imagining that stopping without making a massive mess first.
I need to know if it has been guardrailed into uselessness.
OpenAI seems to be sleeping on the job, huh. I hope they announce GPT-5 soon, because Google is not playing around; they want to dominate the global AI market.
Mmmm no gpt 4.5 is good.u gotta know how to prompt it. But make no my mistake OpenAi has something in the back room
GPT5 will bully Gemini, badly, when it comes out.
not if u make them work together.But thats next level shyt...ya'll not ready for that
Hey do a review on sdxl turbo.. its some crazy stuf
alright this sounds great and all. Needs a lot of testing.
The term open source was used too much.
Because Gemini is not open source.
Therefore those words even if not directly related to gemini kept me think gemini is open source.
My apologies, I had made a few mistakes, should have edited it out
@@1littlecoder No offense I like your content it is very helpful. It was just my subjective impression.
@@simplemanideas4719 Always like constructive feedback. Thanks for highlighting:)
I can't wait to see what's really good with Gemini. Competition is a wonderful thing.
It would be great if Gemini Pro can run in Colab.
I wonder what hardware is needed to run Gemini Ultra?
I don't think they have plans to share the model
Waiting for Gemini Jailbreaks!!
This $h1t is insane! Imagine where we’ll be in only one more year.
Actually the answer is 9am. Hoomins 1 - Gemini 0
when google launches Gemini ultra next month one week later open ai immediately releases gpt 5 . 😂
Finally!
Finally it's real!
how you can say that is a beast and didn't try it
From the specs
@@1littlecoder ok
try bard immediately. He seemed to me to be the same bard from before, who contradicts himself
I think in some location they're still not updated the latest model
Don't overhype it. It's not crushing it. It's marginally better. Within 5% margin at average
which one?
@@1littlecoderGemini is marginally better at language-related benchmarks. And there's an asterisk almost at every comparison point reg how the evaluation was performed. Don't make your final judgement unless you test it thoroughly yourself. All these benchmarks have already proven to not reflect the reality actually and Google is always making a bigger hype than the product actual value
@@alx8439.Well see.These models are never tapped to their full potential due to the person prompting.And I won't even have them competing anyway.Gemini and Gpt5 will be complimenting each other,building on each other's strengths
2:21 if you read the technical report pahe 7 it should be 83,7%😂 google lie
You're right. My bad fell for their trap. Highlighted it in the next technical paper video
Not open source like closedai then not interested.
They help open models push the frontier but I agree with your point
Pure silliness. You will be forced to start being “interested” when these closed-source models become smart enough to be an indispensable part of the economy. Being petulant about it isn’t going to get you anywhere.
Holy moly shit !!!
Hope they dont make it subscribed..
It's not even an open model :(
@@1littlecoder i mean not a subscription version just like gpt4
stop begging bruh.
❤
LMM
But it’s worse at common sense..