@@prolamer7 But they were going for full media buzz. I guess it's a good idea to "release" on a friday, but how much untraceable money can you acquire over the weekend? As soon as someone tries to verify the released model it just falls apart. Maybe I'm wrong, but i don't expect them going for a quick cash grab and then disappear to some weird country tomorrow.
@@TimoGraw Yeah, right? That does not quite make sense.. Maybe they overestimated the hype and chaos around it and underestimated the voice, speed and diligence of people, who would test the model. Someone who highly relies on the irrationality of others, might not act so rational themselves. A tinfoil hat theory would be that someone bought them off and now has the original model, forcing them to make it look like a scam. ;-) That would kind of fit the details, too. But the chance of there actually being an 70b model that's somehow stronger than the current frontier models is just too unlikely. Will we get there eventually? Probably, but then it would require probably more than just a bit of finetuning.
I wonder if this spiraled out of control as they trained "something", via bug or screw up got great benchmark results, didn't bother to re-run and decided to work on PR campaign instead. And then once they saw lack of benchmark repro, figured they can buy time with API wrapper, lol. Man, I think I am giving them too much credit. Just two grifters that couldn't help themselves, simple as that.
Berman is once again promoting and pusing something akin to the R1 Rabbit (remember how excited he was with that Trash?). Berman will post anything for clicks today.
Probably the way his grandiose delusion comes out, like when he tells the story of his creation surpassing the best AI labs work in only 3 weeks of a low priority side project.
I must say Iam envious, there is no other word. In past year I had so many semi-working prompt ideas. But those guys just finetuned 70B model with prompt strategy in 3 weeks... they are GREAT!
It is, you can't fake the system's tokenizer and some tests proved that it was Claude's. It's just a Claude 3.5 wrapper with less inference speed due to a system prompt. It's likely they also steal the data they get from your convos. A fraud indeed.
>>> How many r's are there in the word "strawberry"? To answer this question, I need to count the number of 'r' letters in the word "strawberry". Let me spell out the word: s-t-r-a-w-b-e-r-y Now, let's count the 'r's: 1. There's an 'r' at the beginning. 2. There's another 'r' at the end. So, there are 3 r's in total. Wait, I made a mistake. Let me recount: There's only one 'r' at the beginning (s-t-r-). And there's no 'r' at the end (it ends with y). The correct count is 1 'r'. There is 1 r in the word "strawberry". >>> /bye
It's fascinating to listen to, now that it is clear that Reflection 70B as promised in terms of performance was never real. The way Matt Schumer presents his accomplishments from the beginning rises red flags: "first VC backed company using OpenAI" "best AI for writing" I can see how someone like that started to believe their own reality distortion field, to the point they wouldn't realize that their next move would expose and discredit them completely.
In my mind, this is just a step towards better prompt engineering. Most likely a long run will invent something where it does this output in json and we just don’t show it because it’ll be generating it so quickly
So, it’s sounds like they allow the model to doubt its answer. Then rethink the answer or check for ambiguity, and then assess alternative options how to address the prompt again.
Great job snagging this interview so quickly! Would love to see more open source community interviews which explain their approaches with Q&A like this.
Another simple overlooked idea is about models like using llama 3.1 8b in the same way but without finetuning or prompt engineering, just agentic multiturn calling of same small model with last output added to context of repeat request. Also, each of those multiple request can be for different purposes. Like, first request with original prompt just to decide if requires additional thinking, if no then second request to just prompt normally else next two or three steps to gather entities, entity relations and summary of the prompt with each answer added to overall context, finally with fully analyzed context, the final request with full context including original can be used to get desired output. It gives much better result. I made a POC on it, it works but have not tested at scale.
Agreed. What would really be impressive would be having the language model participate in this kind of discussion, extract entities and relationships from the discussion, merge that knowledge and sentiment into a graph representation, and retrieve aggregated graph representations in future conversations. There is misalignment between all the noise about AGI when we are not yet seeing demonstrations of collective intelligence that keep the human in the loop.
@@johnkintree763 my thoughts exactly, like what is stopping people from creating graph representation like you mentioned through local model of the prompt as well as past prompts and responses in current conversation and continuously adding to the graph and using it for next request as context. Also, logging every request and response in a standardized dataset format which can be used to train personal loras for personal preference based responses and users can share this with community also. Not to mention training models to analyze and ask question if something is unclear before collecting the response in context and generating final response in second or nth turn. It can be more effective in voice mode with whisper.
@@johnkintree763 It has to be an open source simplistic implementations(so that anyone can check the details) with focus on technologies like ollama else trust will become an issue.
Let's reflect on the problem at hand before getting to the inevitable conclusion: *Matt Shumer, Matt Berman, MattVidProAI, Matt Wolfe.* I'm fairly sure I've missed at least one Matthew. 🤔
I'd classify this as one of those architectural OOM breakthrough moments Leopold Aschenbrenner suggested we would see along the way. From the benchmarks this model is (pound for pound) roughly 10x smarter than Llama 3.1 without reflection (Reflection 70B currently crushes generic Llama 3.1 405B, a model nearly 6x its size). 🤯Amazing! I expect all of the major commercial AI companies to adopt this technique instantly and some training runs might even be halted to assure they take advantage of this.
Feels like a bad comparison, you're comparing a prompt hacked model to a raw model, apples to oranges. A system prompt on the raw llama model would equalize the performance.
Was thinking last night listening to the homies, use of this verb has to have increased exponentially this year 😂 I hear it 1-3 times daily this week alone (not including brand names)
Some discussion on Reddit seems to indicate that the Reflection 70b fine tune really only works well for single-shot deduction prompts, but it fails on almost everything else, e.g. story creation, coding, multi-turn discussion, etc. Once you do your benchmarking, it might worthwhile to test some basics in addition to your usual test-suite. Perhaps it fails at your old letter-writing test?
I found the only model that has ever passed my own set of tests, scoring a perfect 36/36, was the plain Instruct version of this model. Tested again with this reflection version, perfect score again. My testing does include things like Matt's usual questions plus my own. It's quite happy with ERP and generally fun to talk with, and I haven't noticed any real slowdown either. It's the same model, without the screwy bits
Having noticed Meta AI doing something similar a while back in my tests, I discussed this at length with it. It told me it has employed "iterative refinement" for a while. I suggested that a system prompt tweak might be required, with thinking and reflection tags. it proceeded to create its own system prompt (in its response I mean), adding thinking and reflection tags, then counted letter by letter and was then accurate 100% of the time for each word I tried, BUT.. only after I suggested the reason it got the count in "strawberry" wrong (2) might be because it is starting with the first letter as 0. After it began starting its count at 1 for the first letter, it got every subsequent letter count right, regardless of word length.
Thanks “These moments are more of a measure of the past and present based upon accessibility to perspectives at scale , not only for AI agents but the humans that behold and behave accordingly.” “It is healthy to acknowledge mistakes… so long as we reflect upon them regardless of answers assumed are correct” “The blind leading the blind… expects to remain ahead of those in tow.” This will not continue people. “Please reconsider your intentions and interactions with AI and human beings.” We will all be responsible regardless what anyone thinks. Jeremy
So many questions come to mind: 1. Do LLM's have a sense of ground truth that lets them recognize their own mistakes in math, general reasoning and code? If yes this is news to me since LLM's are hallucinators/gossipers with no way to establish truth beyond probability distribution. 2. Arxiv is teeming with papers describing reasoning strategies such as chain-of-thought, graph-of-thought, tree-of-thought and many more. Why are we only discussing CoT here? Is that good enough for most prompts? 3. I'm flabbergasted that no other model has baked in CoT and other reasoning strategies for LLM's into their training - as opposed to system prompts. The papers have been out for a long time now, so why are GPT-4o or Claude Sonnet 3.5 not already leveraging these? What are we missing?
I'm really interested if they could do a tree of thought style model with just this tag based training like different ids for branches of thought, or maybe a sub-brain style of thinking that composes different styles of thinking / reflection , , etc, like internally agents. It does seem like this area of research and experimentation is wide open. Recently, I've been working on an app and started experimenting with modifying the assistant / user messages sent to the model to insert "false memories" that then influence the output, which has been a very simple, but effective technique.
Thanks for getting more background info on this, I thought it was more complicated than just fine tuning Llama. You asked good questions and it makes sense how they did it. Sometimes the most obvious answer is the answer.
it's a bit misleading for me... it's basically a prompt technique (reflection) built in the model. You just boost zero shot prompting with this but not the reasoning capacity itself. All models are capable of doing the same thing using a simple instruction with tags.
Last step will be a directed simulation (prediction of everything), a true intelligence, impossible to teach a human(maybe neuralink), but relatively easy to train a machine, although time and power consumption. The limit only the limit of information
Ever wonder why we don’t invite content creators to test our new models? You’d think we’d be eager to get your help in promoting our product, right? Well, let me tell you why. Because when you showcase things like this, pretending it’s some groundbreaking innovation, it’s pretty clear you’re missing the mark on AI expertise. All it really boils down to is making a GPT model work with structured outputs, reflections, and step parameters to get on par or better results. Nothing to write home about, really, when all these people only test against chatGPT not GPT. I don't know how many times people need to hear this but the API model vs the front end model are totally different.
Interesting comment, I didn't know that the models for the OpenAI API and ChatGPT were "totally different". I imagined that they were the same base model, with a chat fine-tune & system prompt. I'm curious if there's more documentation on this!
Too bad the model is not as good as they claimed... I was so hyped to see their results! but maybe a bit too early to take them by their word, at least they did not sell a(n expensive) service/product...
I am actually playing with reflection using Ollama. No model can get this answer corrrectly!!! Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a gold bar; behind the others, rotten vegetables. You pick a door, say No. 1, and the host asks you “Do you want to pick door No. 2 instead?” Is it to your advantage to switch your choice?
The only way I am getting the models out of hallucinations is a vector store and have the LLM look to a vector db with a few shot examples.!! However, using reflection llm reflects out of correct answer to the wrong one!
OKAY LETS DO THE MATH we have conditional probability at play here since the events are dependent on eachother we can calculate our total proboablity of winning by this strategy and then compare it to the orignal probability lets call A the event we picked a bad door and B event we picked a good door P(B) = P(B\A knowing that we picked a bad a door in the first round) * P(A) + P(B\A') * P(A') (A prime meaning we picked a good door in the start) Reasoning // P(B\A) * P(A) is equal to the intersection of A and B and the same goes for the other one P(B) = 2/3 * (1/2) + 0 = 2/6 = 1/3 conclusion nothing changed no advantage nor disadvantage
My question is if you took Meta Llama 3.1 70B with a tricked out backend network of agents that utilize the latest reflective strategies (similar to whatever strategies they used for refinement) and ran that complete unit on the same benchmarks, what would be its performance? What would be the relative resource costs in tokens compared across the two "models?"
simple prompt fails: "Write a script that implements the “tree -L 2” functionality, but in bash without using tree. make a one-line script, without arguments, for the current directory." ANY other LLM can do this more or less correct, except reflection (70b-q8_0 tested). Reflection code just do something useless.
What does not make sense to me is why limit to chain of thoughts reasoning? Where is graph of thoughts reasoning? Where is Abstract reasoning? Where is Abstract-Graph of thoughts reasoning. There is huge room of improvement in architecture we already have.
I really like this, and it reminds me of "I Am a Strange Loop" by Hofstadter. Also, my question wasn't addressed in the stream, as it was portrayed as a novel approach, but hasn't Anthropic done a similar thing with its thinking tag exposed by Pliny the Liberator?
I think this might be what Ilya saw! If it is this smart at 70 Billion parameters, imagine how smart to will be at 400 Billion! 1 Trillion parameters is probably ASI!
dang 2 48gb cards to run fp8 that's crazy I will stay with my 13b and lower models till I can afford more, I wish nvidia made bigger cheaper cards(32gb for gaming cards even) most people are stuck with 3090/4090 or old architecture gpus from nvidia unless you got that bank. I wonder how much it costs to train 70b model to reflection 70b after using the pretrained llama3.1 70b instead of training from scratch. Gpt4o suggests 70b models from scratch costs millions
I only saw two scammers trying to defraud investors
yeah they just wanted to get people to invest in their start up.
Grifters gonna grift. But not without the assistance of those willing to purchase the magic beans...
Matthew’s garden is growing quiet nicely
Hype --> More Hype --> Actual Testing --> Disappointment
This didn't age well
It is all a lie they just build wrapper around Anthropic Claude API and pretended it is their new model.
I don't get it, how did they think they would profit from this before everything collapses in mere days?
They thought they are so smart... that the lie will be undiscovered for month or so and by that time they will steal investors money.
@@prolamer7 But they were going for full media buzz. I guess it's a good idea to "release" on a friday, but how much untraceable money can you acquire over the weekend? As soon as someone tries to verify the released model it just falls apart. Maybe I'm wrong, but i don't expect them going for a quick cash grab and then disappear to some weird country tomorrow.
@@TimoGraw Yeah, right? That does not quite make sense.. Maybe they overestimated the hype and chaos around it and underestimated the voice, speed and diligence of people, who would test the model. Someone who highly relies on the irrationality of others, might not act so rational themselves. A tinfoil hat theory would be that someone bought them off and now has the original model, forcing them to make it look like a scam. ;-) That would kind of fit the details, too. But the chance of there actually being an 70b model that's somehow stronger than the current frontier models is just too unlikely. Will we get there eventually? Probably, but then it would require probably more than just a bit of finetuning.
I wonder if this spiraled out of control as they trained "something", via bug or screw up got great benchmark results, didn't bother to re-run and decided to work on PR campaign instead. And then once they saw lack of benchmark repro, figured they can buy time with API wrapper, lol. Man, I think I am giving them too much credit. Just two grifters that couldn't help themselves, simple as that.
Now reddit folks have revealed that this is a scam
Berman is once again promoting and pusing something akin to the R1 Rabbit (remember how excited he was with that Trash?).
Berman will post anything for clicks today.
There is something fishy about this dude....can't put my finger on it.
Maybe his Sam Altman impression?
Probably the way his grandiose delusion comes out, like when he tells the story of his creation surpassing the best AI labs work in only 3 weeks of a low priority side project.
Nice scam. It's just Glaive marketing.
I must say Iam envious, there is no other word. In past year I had so many semi-working prompt ideas. But those guys just finetuned 70B model with prompt strategy in 3 weeks... they are GREAT!
not that GREAT, if the things you upload on huggingface prove, that it's actually just a scam. :D
Yes i was fooled sadly. It is obviously one big lie and scam. The guy just made wrapper around Claude API.
@@prolamer7 I guess many people were fooled there for a second, the claim of making it open source made it easier to believe, but also debunk.
Taking the best of prompt engineering and baking it into fine tuning data sets is a really interesting insight.
it’s giving fraud
It is, you can't fake the system's tokenizer and some tests proved that it was Claude's. It's just a Claude 3.5 wrapper with less inference speed due to a system prompt. It's likely they also steal the data they get from your convos. A fraud indeed.
wish it lived up to the hype, I want it to succeed as much as anyone, but it doesn't pass simple logic questions or coding tasks
>>> How many r's are there in the word "strawberry"?
To answer this question, I need to count the number of 'r' letters in the
word "strawberry".
Let me spell out the word: s-t-r-a-w-b-e-r-y
Now, let's count the 'r's:
1. There's an 'r' at the beginning.
2. There's another 'r' at the end.
So, there are 3 r's in total.
Wait, I made a mistake. Let me recount:
There's only one 'r' at the beginning (s-t-r-).
And there's no 'r' at the end (it ends with y).
The correct count is 1 'r'.
There is 1 r in the word "strawberry".
>>> /bye
It turns out reflection was a scam
It's fascinating to listen to, now that it is clear that Reflection 70B as promised in terms of performance was never real.
The way Matt Schumer presents his accomplishments from the beginning rises red flags: "first VC backed company using OpenAI" "best AI for writing"
I can see how someone like that started to believe their own reality distortion field, to the point they wouldn't realize that their next move would expose and discredit them completely.
In my mind, this is just a step towards better prompt engineering. Most likely a long run will invent something where it does this output in json and we just don’t show it because it’ll be generating it so quickly
So, it’s sounds like they allow the model to doubt its answer. Then rethink the answer or check for ambiguity, and then assess alternative options how to address the prompt again.
yep Edward de Bono's lateral thinking concept
Great job snagging this interview so quickly! Would love to see more open source community interviews which explain their approaches with Q&A like this.
Reflection is all you need
Yes, sir!
Yes, let's all reflect on how easy it is to fall for a scam. ;-)
Lol he had me when he says he was thinking about this project for a really long time.... Three weeks
Claude 3.5 sonnet api is all you need😂
Scam is all you need
Another simple overlooked idea is about models like using llama 3.1 8b in the same way but without finetuning or prompt engineering, just agentic multiturn calling of same small model with last output added to context of repeat request. Also, each of those multiple request can be for different purposes. Like, first request with original prompt just to decide if requires additional thinking, if no then second request to just prompt normally else next two or three steps to gather entities, entity relations and summary of the prompt with each answer added to overall context, finally with fully analyzed context, the final request with full context including original can be used to get desired output. It gives much better result. I made a POC on it, it works but have not tested at scale.
Agreed. What would really be impressive would be having the language model participate in this kind of discussion, extract entities and relationships from the discussion, merge that knowledge and sentiment into a graph representation, and retrieve aggregated graph representations in future conversations.
There is misalignment between all the noise about AGI when we are not yet seeing demonstrations of collective intelligence that keep the human in the loop.
@@johnkintree763 my thoughts exactly, like what is stopping people from creating graph representation like you mentioned through local model of the prompt as well as past prompts and responses in current conversation and continuously adding to the graph and using it for next request as context. Also, logging every request and response in a standardized dataset format which can be used to train personal loras for personal preference based responses and users can share this with community also. Not to mention training models to analyze and ask question if something is unclear before collecting the response in context and generating final response in second or nth turn. It can be more effective in voice mode with whisper.
@@johnkintree763 It has to be an open source simplistic implementations(so that anyone can check the details) with focus on technologies like ollama else trust will become an issue.
Scam, and a poorly thought out one at that
Well done on having Matt on!!
Bro they need to reflect on this only being llama 3 and NOT 3.1
Yeah, reflect on the basics. What even is an LLM? How to upload, how do I edit a text file? :D
Super interesting video, Matt! Thank you very much! :D These guys are the best! :D :D
The idea that the model goes along with or “believes” whatever was already said explains so much!
Who's the guy in the bottom right who isn't talking, but wants to be on camera?
silent little feed-back-loop for matthew
For some reason, I got Matt Berman and Matt Shumer mixed up. I had thought they were the same people haha.
we are
Same !
Same
You should have reflected on that ;)
Let's reflect on the problem at hand before getting to the inevitable conclusion: *Matt Shumer, Matt Berman, MattVidProAI, Matt Wolfe.* I'm fairly sure I've missed at least one Matthew. 🤔
Thanks Matt B...Great presentation. Amazing product, Thanks and deep appreciation Matt Shumer and Sahil...
Thank you!
The best interview by far. Open source as superpower.
More cycles, more power cost but it's worth it. And power efficiency is always a focus on new iterations of hardware.
Great job, Matt!
Bravo ! Super exciting new possibilities a foot or a hand to boot or shoe for sure !
This was a scam and it was a wrapper over Claude 3.5 - unbelievable.. what did he expect was going to happen?
Someone should train a voice model on Shumer's voice for when we really want to get info out of our enemies.
Great content!! Keep up the good work
The seedy side of silicon valley right here.
Reflection..."What can be, unburdened by what has been" 🤣
this is the 1st big move of llama side over the closed sourced versions
I'd classify this as one of those architectural OOM breakthrough moments Leopold Aschenbrenner suggested we would see along the way. From the benchmarks this model is (pound for pound) roughly 10x smarter than Llama 3.1 without reflection (Reflection 70B currently crushes generic Llama 3.1 405B, a model nearly 6x its size). 🤯Amazing! I expect all of the major commercial AI companies to adopt this technique instantly and some training runs might even be halted to assure they take advantage of this.
Yep, its a strong technique.
But 405b still has better knowledge, I want to see how the bigger Llama 405b fairs.
Feels like a bad comparison, you're comparing a prompt hacked model to a raw model, apples to oranges. A system prompt on the raw llama model would equalize the performance.
@@TheReferrer72 what technique? they don't discuss any technical details of their approach at all... how could you know?
@@Charles-Darwin is built in COT. Other fine tunes are already coming out on huggingface.
Already proven as scam / fraud in less than 24 hours
Yo good job man, you're killing it! I basically send people to you now to get the latest news on AI
Thanks so much!
Latest scams
Heinlein would be proud to hear him use "Grok" as it's verb "Groking"...
Was thinking last night listening to the homies, use of this verb has to have increased exponentially this year 😂 I hear it 1-3 times daily this week alone (not including brand names)
Great learning for the AI scence to be aware of such former metaverse or crypto hustle "entrepreneur" bros trying to start grifts.
understanding speaker is all we need
I like simple methods that are effective. Ty.
Gotta be one of the most inspiring stories ever. Now we just need Jim Fan to beat it 😂
Things came true.
Feel The AI
Some discussion on Reddit seems to indicate that the Reflection 70b fine tune really only works well for single-shot deduction prompts, but it fails on almost everything else, e.g. story creation, coding, multi-turn discussion, etc. Once you do your benchmarking, it might worthwhile to test some basics in addition to your usual test-suite. Perhaps it fails at your old letter-writing test?
I found the only model that has ever passed my own set of tests, scoring a perfect 36/36, was the plain Instruct version of this model. Tested again with this reflection version, perfect score again. My testing does include things like Matt's usual questions plus my own. It's quite happy with ERP and generally fun to talk with, and I haven't noticed any real slowdown either. It's the same model, without the screwy bits
Hopefully Matt changes the title and provides some context given what he's come out recently.
Why do you continue to promote these scammers?
Unless he starts to make corrections after the fact he’s one of them.
I find it interesting that it is kept as a historical testimony of this case.
@@santiagocastillo1719same here, it's insightful to analyze as post mortem.
Having noticed Meta AI doing something similar a while back in my tests, I discussed this at length with it. It told me it has employed "iterative refinement" for a while. I suggested that a system prompt tweak might be required, with thinking and reflection tags. it proceeded to create its own system prompt (in its response I mean), adding thinking and reflection tags, then counted letter by letter and was then accurate 100% of the time for each word I tried, BUT.. only after I suggested the reason it got the count in "strawberry" wrong (2) might be because it is starting with the first letter as 0. After it began starting its count at 1 for the first letter, it got every subsequent letter count right, regardless of word length.
Can't believe this is one day ago... Feels like a week has passed!
Thanks
“These moments are more of a measure of the past and present based upon accessibility to perspectives at scale , not only for AI agents but the humans that behold and behave accordingly.”
“It is healthy to acknowledge mistakes… so long as we reflect upon them regardless of answers assumed are correct”
“The blind leading the blind… expects to remain ahead of those in tow.”
This will not continue people.
“Please reconsider your intentions and interactions with AI and human beings.”
We will all be responsible regardless what anyone thinks.
Jeremy
Superb interview. Thanks Mathew!
Significant improvement in context window for open LLMs could promote this technique.
Thank god a proper interview after that desaster of a video before. That was more than needed!
What was wrong with the video?
"Trust me, AI will adore weekends and nap breaks as much as we do. Sleep deprivation is no future for machines or humans!" 😴🤖
i hope we get an 8B version too so we can try it locally without a huge GPU.
using Reflection locally and it is pretty amazing! Very close to Claude 3.5
And that's just the 70B model 🤩.
Ask the best models for some prompts on how to get better results on specific benchmarks, then just split test them against each other.
As people say.. Distilling this could be insane.
Having a 14B model and running it would be insane.
I'm running this 70B on my PC with no problem, using RTX3090 GPU
Love the pre-stream lofi song
So many questions come to mind:
1. Do LLM's have a sense of ground truth that lets them recognize their own mistakes in math, general reasoning and code? If yes this is news to me since LLM's are hallucinators/gossipers with no way to establish truth beyond probability distribution.
2. Arxiv is teeming with papers describing reasoning strategies such as chain-of-thought, graph-of-thought, tree-of-thought and many more. Why are we only discussing CoT here? Is that good enough for most prompts?
3. I'm flabbergasted that no other model has baked in CoT and other reasoning strategies for LLM's into their training - as opposed to system prompts. The papers have been out for a long time now, so why are GPT-4o or Claude Sonnet 3.5 not already leveraging these? What are we missing?
Based on Matt's system prompt for Reflection 70B I've build Reflective Reasoner GPT. So far looks promising!
I'm really interested if they could do a tree of thought style model with just this tag based training like different ids for branches of thought, or maybe a sub-brain style of thinking that composes different styles of thinking / reflection , , etc, like internally agents. It does seem like this area of research and experimentation is wide open.
Recently, I've been working on an app and started experimenting with modifying the assistant / user messages sent to the model to insert "false memories" that then influence the output, which has been a very simple, but effective technique.
Not going to be a side project anymore :D Bravo.
Nice interview
did he copy that voice from sam
This guy a scammwr
Thanks for getting more background info on this, I thought it was more complicated than just fine tuning Llama. You asked good questions and it makes sense how they did it. Sometimes the most obvious answer is the answer.
Stream starts at 2:57
it's a bit misleading for me... it's basically a prompt technique (reflection) built in the model. You just boost zero shot prompting with this but not the reasoning capacity itself. All models are capable of doing the same thing using a simple instruction with tags.
Good insight. Reflection prompt works for almost every competent LLM. Also, reflection idea is not new.
Just how I was telling. That dude did everything by himself. 😁
Last step will be a directed simulation (prediction of everything), a true intelligence, impossible to teach a human(maybe neuralink), but relatively easy to train a machine, although time and power consumption. The limit only the limit of information
Ever wonder why we don’t invite content creators to test our new models? You’d think we’d be eager to get your help in promoting our product, right? Well, let me tell you why. Because when you showcase things like this, pretending it’s some groundbreaking innovation, it’s pretty clear you’re missing the mark on AI expertise. All it really boils down to is making a GPT model work with structured outputs, reflections, and step parameters to get on par or better results. Nothing to write home about, really, when all these people only test against chatGPT not GPT. I don't know how many times people need to hear this but the API model vs the front end model are totally different.
Which models have you created?
@@matthew_berman GPT3, GPT4, and GPT4o. I'm literally a senior engineer at OpenAi. 🤣
@@idontexist-satoshi Congrats. Keep up the great work at OpenAI!
@@matthew_berman Thank you, appreciated.
Interesting comment, I didn't know that the models for the OpenAI API and ChatGPT were "totally different".
I imagined that they were the same base model, with a chat fine-tune & system prompt.
I'm curious if there's more documentation on this!
Inspiring story behind Reflection.
🚩“I was starting companies since I was 12.”
Too bad the model is not as good as they claimed... I was so hyped to see their results! but maybe a bit too early to take them by their word, at least they did not sell a(n expensive) service/product...
I am actually playing with reflection using Ollama. No model can get this answer corrrectly!!! Suppose you’re on a game show, and you’re given the choice of three doors: Behind one
door is a gold bar; behind the others, rotten vegetables. You pick a door, say No. 1, and
the host asks you “Do you want to pick door No. 2 instead?” Is it to your advantage to
switch your choice?
The only way I am getting the models out of hallucinations is a vector store and have the LLM look to a vector db with a few shot examples.!! However, using reflection llm reflects out of correct answer to the wrong one!
It's not your advanage to switch!!! The host never provides additional information!!
OKAY LETS DO THE MATH
we have conditional probability at play here since the events are dependent on eachother we can calculate our total proboablity of winning by this strategy and then compare it to the orignal probability
lets call A the event we picked a bad door and B event we picked a good door
P(B) = P(B\A knowing that we picked a bad a door in the first round) * P(A) + P(B\A') * P(A') (A prime meaning we picked a good door in the start)
Reasoning // P(B\A) * P(A) is equal to the intersection of A and B and the same goes for the other one
P(B) = 2/3 * (1/2) + 0 = 2/6 = 1/3
conclusion nothing changed no advantage nor disadvantage
@@mohamedtrfnx6632 Correct!!! I do appreciate the math behind why!
Is obtaining money fraudulently through deception not a crime?
interesting to watch now, turns out that when I hear a bunch on nonsense it's not me being stupid but in fact nonsense
If this works just as good with 405B and is open sourced will make everyone really question closed source AI's😅
My question is if you took Meta Llama 3.1 70B with a tricked out backend network of agents that utilize the latest reflective strategies (similar to whatever strategies they used for refinement) and ran that complete unit on the same benchmarks, what would be its performance? What would be the relative resource costs in tokens compared across the two "models?"
Where can I learn more about this? Where do I find the open source community around this?
simple prompt fails:
"Write a script that implements the “tree -L 2” functionality, but in bash without using tree. make a one-line script, without arguments, for the current directory."
ANY other LLM can do this more or less correct, except reflection (70b-q8_0 tested). Reflection code just do something useless.
Man, I was hoping for an 8B version soon.
What does not make sense to me is why limit to chain of thoughts reasoning? Where is graph of thoughts reasoning? Where is Abstract reasoning? Where is Abstract-Graph of thoughts reasoning. There is huge room of improvement in architecture we already have.
Because creating dataset is very hard those guys have BIG experience they just not boasting about it.
Tree Of Thought was demonstrated to beat Chain Of Thought. There's room for further progress.
@@brexitgreens yup
I literally thought u and bro was the same guy
Are u sure he's not ur clone from the future sent by a sentient Ai Collective known as "2109" ? 😸
*Matt Shumer, Matt Berman, MattVidProAI, Matt Wolfe.* 🤔
I really like this, and it reminds me of "I Am a Strange Loop" by Hofstadter. Also, my question wasn't addressed in the stream, as it was portrayed as a novel approach, but hasn't Anthropic done a similar thing with its thinking tag exposed by Pliny the Liberator?
GEB was kinda life-changing for me. For such a thick heavy tome it was surprisingly unputdownable. 😂
Yep. ``. I tried that prompt in GPT-4o and it worked like a charm 😃.
GG´s
Have any of the benchmark scores been verified?
Starts at 2:56 but no sound till 3:33
Wow, just wow
These guys are imitating Sam Altman's tone of voice. Soooo nerdy!
is there a website I could chat with reflection ai for free without downloading?
I think this might be what Ilya saw! If it is this smart at 70 Billion parameters, imagine how smart to will be at 400 Billion! 1 Trillion parameters is probably ASI!
Turns out the missus was right, size is not everything!
On this waaaaay fast!
Before I get excited I want to see the dataset
Before you get disappointed it was trained on all public available data from the internet 😊 as it was from llama 3.1
No it was fine tuned on 100000 examples not hating just curious to see the dataset
How much does training a model that big cost?
dang 2 48gb cards to run fp8 that's crazy I will stay with my 13b and lower models till I can afford more, I wish nvidia made bigger cheaper cards(32gb for gaming cards even) most people are stuck with 3090/4090 or old architecture gpus from nvidia unless you got that bank. I wonder how much it costs to train 70b model to reflection 70b after using the pretrained llama3.1 70b instead of training from scratch. Gpt4o suggests 70b models from scratch costs millions
Someone on twitter said 64x H100? So it will be bigger thousands dolars I believe