the holy grail problem to prompt it with is one in which each contributor gives different advice, and the decider model produces output that's different from all the contributors and different than what the same model would have produced alone without MoM
You could combine tree of thought into this mixture of models Where instead of having one model pretend to be three different experts, you use a tree of thought process across these different architectures to generate a more accurate output across the entire task
Cool experiment! I'd like to see a sort of "deliberative democracy" architecture--where the models have a chance to discuss back-and-forth or to see one another's initial answers before providing a final answer and then voting
This was really interesting, I really like your thought process. One suggestion I have to validate your results is to run each of your tests 3-5 times and see how consistent each architecture is, since you’re going to get different outputs each time. I also didn’t see whether you were controlling model temperature in your setup, and I wonder if putting a temp 0 for at least one participating model as a sanity check would improve output (another opportunity to run a batch test battery). This cuts really close to what I’ve been imagining for agent framework cooperative problem solving, so thank you again for showing us how to put ideas into code.
Very interesting you are not only thinking about such solutions but actually programming and testing them. Which is really great! What about this architecture: King architecture BUT with a twist that you will execute question batch with king being first GPT4, then Opus, then llama3 70b and you will then chose only answers with majority of votes ie if GPT, Opus agree.
And if they don’t agree, loop back and resolve the differences via Socratic method until there’s consensus on either a solution or a recommended plan of action to improve the input for the models to try again. I wonder if the local models could reason out the problem super cheaply if given enough reflection and conversation iterations, or if the general models are just too effective to leave out. Probably use case dependent.
Really interesting! Like others mentioned, even if the added reliability of a MoM architecture, it may not be worth the computing power and additional context length, and GPT-4 might be strong enough to figure most of these out on its own. I think a good compromise would be using less LLMs in the hierarchy
Thanks for sharing your research! I had an idea about pairing a large model with a tiny model, where the tiny model is asked a question and the large model evaluates the response. The large model identifies any issues and iteratively adjusts the prompt to help the tiny model get the right answer. Not sure if it would work in practice, but if it did maybe a tiny (and cheap) model could be made effective to solve more problems than it otherwise could have?
Damn this was a beautiful experiment! Really really can see this being used effectively on a grand scale. I mean even think of the little LLMs if we had those all deciding and the larger one is Llama3 or something we can run 100% locally. This can be really powerful for those with less GPU capabilities and low funding if they run them sequentially.
The democratic arch should probably have a step where you have one strong model that decides which answers are identical and remove all but one of them before voting. Otherwise you will get a situation where there are maybe 7 correct answers and one wrong, then the wrong answer has an unfair advantage since the models will spread their votes on the 7 correct answers, but not the one wrong answer. Also, solutions like this is where Groq with 300 tokens/s really shines.
Seems like the second and third method are both kind of getting the best answer of each model, which makes it kind of average. The first architecture on the other side takes all answers into a consideration, adds all insights and creates the best answer from them all. So in a sense it gets only the good stuff and excels the most. This is the way.
Next projekt is a Mo(MoM) model where we compare the result of all 3 MoM systems with these systems again. Think of recursively going down the rabbit hole.
🤣😂 I just have to say it. Monarchy seems to be the solution for most problems. However jokes aside, this experiment turned out really great, and fascinating how the models behave in the different setups and interactions.
@@jiyuhenHis demo is not really a monarchy. In a real life a king is able to have his own ideas or solution, he has the freedom to completely ignore proposal put forth to him. But in this demo the king doesn't have any of those freedoms and powers. He has to judge and decide what others put forth to him. Let the king have his own ideas and the freedom I mentioned earlier and I bet the results will be different.
This was a fantastic video thank you...brought me up to speed on the various models and structures of their hierachys was fascinating. Not dabbled in agents for a while so was nice to be surprised at progress in this area vs my early attempts that spent tokens and produced nonsense.... question....do you have any teams of agents setup doing actual real world useful stuff? (I'm sure I'll find out by watching more). Thanks for the video.
I think that it would be better to estimate performance when you have some simple model at the output level. GPT-4 can operate with all of provided problems without previous models solutions
22:38 it only got one vote because each model provided 10 unique answers. This shows that the democracy model isn’t a good architecture for all kinds of problems (specifically, the ones where the answer is determinate).
I would probably choose to do the tally locally using an optimized solution a simple python script bypassing any AI that way you eliminate the models sucking at math and use architecture specialized for solving the math problem of counting 😊
quite impressed by these MoM / MoE model architectures, however, I am not satisfied with the leetcode hard problem used in the test set. Testing the llm generated response on just 3 test cases is not a valid determinant for the architecture's capabilities. I could see the solution is itself in the order of 4 ( O(N^4)) time complexity, which essentially is Brute Force. A better bench mark would be testing how well it can optimise the code and handle diverse test suits IMO.
Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.
The marble problem proves nothing. Every model vendor cheats on some problems and sticks these problems into the training datasets. Same for other problems. You need to use new problems every time. Otherwise these reviews are b.s.
I was thinking same. So the problems should be more adequate. But it was a proof of concept, very nice work. I planned aomething similar with websocket agents last summer, but it became too expensive to test for a hobby. Now it is right time because ollama!
Different models have slightly different training, context, and perspective, so this creates a broader perspective to approach the problem at hand. This will not be practical for everyday prompts.
@@jichaelmorgan3796 I disagree on the practicality point. The way it is Setup in this video from a testing the methodology perspective yes it’s not practical.. But behind it is actually very practical. Because all of this stuff that is going on can be done in code so the user doesn’t see everything that goes on behind-the-scenes and just gets the final output
This video is a whole lot of fluff for a very simple concept: combining outputs from multiple LLMs to get better answers. The creator makes it seem revolutionary with fancy names like "King," "Duopoly," and "Democracy," but it's just basic ensemble methods. They feed the same problem to a bunch of LLMs (including weaker ones like Llama), then use those answers as context for a stronger LLM (GPT-4) to make the final decision. The "Duopoly" is just GPT-4 and Claude arguing about the best answer, and the "Democracy" lets all the LLMs "vote" on the best answer. He doesn't go into detail about how the voting works or what constitutes a good vote. The results are a mixed bag, sometimes better, sometimes worse than just using GPT-4 alone. The real takeaway is that ensemble methods can be useful, but this video overcomplicates it with theatrical names and presentation.
@@tellesu I'm not a bot, I just want to hear smart people like this guy break it down. more people need to be talking about this, it is AI development towards Armageddon
@@tellesu that's not fear mongering it is simple fact. A certain tribal state is using advanced AI to engage in the mass murder of civilians without any human oversight. The where is daddy program literally watches targets leave so-called military locations and orders an attack only after they are at home with their family. It is deliberate mass murder of civilians, and it is powered by AI.
Perhaps it would be even more powerful to make a Chain of Models/Chain of Experts architecture that has a line of models and each model builds upon the previous models' responses. Making a Network of Models would be even better, where you have layers of multiple models and each next layer gets the previous layer's outputs and ultimately combines everything in the end to form an answer to your question.
@@c.d.osajotiamaraca3382 Yeah, I didn't think of that. Maybe in the system prompt, it could say there is a chance that the output of the previous model(s) is incorrect and to consider that before making a decision. Or between each layer it could have an internet search if applicable. Not sure how I would program that as I don't know how to do HTTP requests but it seems possible.
A note: the marble problem has actually been solved in Matthew Berman's testing by WizardLM-70B-V1.0-GPTQ, Mistral 7b OpenOrca, Mixtral, Mistral Next, Mistral Large, and LLaMA 3 70b Groq. That is however only 6 of the 32 models he's tested. I will note that in my own testing Reka Core passed the marble test while it failed when Matthew was testing it.
Great experiment!! I wonder if adding in reasoning to their answer, if democracy would vote differently? Or if the answers could be discussed before voting. Having a bunch of AI/people voting on bad assumptions is never good. "before voting, discuss each answer for a) meeting all the requirements and b) being the most accurate".
actually, the most likely place for the marble to be is on the floor, because it is no easy feat to turn a cup with a marble in it upside down and put it on a table without the marble falling out before.
I loved the "crucial insights" into gravity 😂
😂😂
the holy grail problem to prompt it with is one in which each contributor gives different advice, and the decider model produces output that's different from all the contributors and different than what the same model would have produced alone without MoM
Give each actor its own motivation. Like a murder mystery. The king is the detective. Have it play Clue.
Different personalities would be good.
I assume this is all on the GitHub repo to play with
Wrong thread
Kris, That’s an extraordinary project with surprising results. Thank you for sharing your talents.
Good, but as noted in the comments it will be interesting to see the results for "plain" GPT4 and Claude for the same set of problems
yeah, kinda regret not inc a basetest
@@AllAboutAIYou can add another video. I'm very interested.
You could combine tree of thought into this mixture of models
Where instead of having one model pretend to be three different experts, you use a tree of thought process across these different architectures to generate a more accurate output across the entire task
great idea :) tnx
Cool experiment! I'd like to see a sort of "deliberative democracy" architecture--where the models have a chance to discuss back-and-forth or to see one another's initial answers before providing a final answer and then voting
This was really interesting, I really like your thought process. One suggestion I have to validate your results is to run each of your tests 3-5 times and see how consistent each architecture is, since you’re going to get different outputs each time. I also didn’t see whether you were controlling model temperature in your setup, and I wonder if putting a temp 0 for at least one participating model as a sanity check would improve output (another opportunity to run a batch test battery).
This cuts really close to what I’ve been imagining for agent framework cooperative problem solving, so thank you again for showing us how to put ideas into code.
yeah good idea, thnx :)
I love these experiments soo much.
thnx :)
that's epic man! well done.
Excellent...It'd be great if the models could have been run in parallel. This way, less time would be needed.
great idea, will def check into this!
Amazing idea, and excellent result. Well done!!
thnx :)
Excellent idea! I understand Opus and OpenAI as the overseer, but if local LLM could proof being the heroes here, would be supberb!
true!
Very interesting you are not only thinking about such solutions but actually programming and testing them. Which is really great!
What about this architecture:
King architecture BUT with a twist that you will execute question batch with king being first GPT4, then Opus, then llama3 70b
and you will then chose only answers with majority of votes ie if GPT, Opus agree.
And if they don’t agree, loop back and resolve the differences via Socratic method until there’s consensus on either a solution or a recommended plan of action to improve the input for the models to try again.
I wonder if the local models could reason out the problem super cheaply if given enough reflection and conversation iterations, or if the general models are just too effective to leave out. Probably use case dependent.
cool idea, noted :) tnx
@@AllAboutAI :-)
Great idea! Like your experiments. Keep up the great work!
thn mate :)
Really interesting! Like others mentioned, even if the added reliability of a MoM architecture, it may not be worth the computing power and additional context length, and GPT-4 might be strong enough to figure most of these out on its own. I think a good compromise would be using less LLMs in the hierarchy
yeah, or you could just use OS models with llama.cpp or ollama i guess
Thanks for sharing your research! I had an idea about pairing a large model with a tiny model, where the tiny model is asked a question and the large model evaluates the response. The large model identifies any issues and iteratively adjusts the prompt to help the tiny model get the right answer. Not sure if it would work in practice, but if it did maybe a tiny (and cheap) model could be made effective to solve more problems than it otherwise could have?
Damn this was a beautiful experiment! Really really can see this being used effectively on a grand scale. I mean even think of the little LLMs if we had those all deciding and the larger one is Llama3 or something we can run 100% locally. This can be really powerful for those with less GPU capabilities and low funding if they run them sequentially.
King is King!
Looks like the Ai jury is about to be seated in a courtroom. and voilà!
The democratic arch should probably have a step where you have one strong model that decides which answers are identical and remove all but one of them before voting. Otherwise you will get a situation where there are maybe 7 correct answers and one wrong, then the wrong answer has an unfair advantage since the models will spread their votes on the 7 correct answers, but not the one wrong answer.
Also, solutions like this is where Groq with 300 tokens/s really shines.
great idea, will try this. tnx
Seems like the second and third method are both kind of getting the best answer of each model, which makes it kind of average. The first architecture on the other side takes all answers into a consideration, adds all insights and creates the best answer from them all. So in a sense it gets only the good stuff and excels the most. This is the way.
You could prob achieve that with a more deliberative democratic architecture
You should've add a baseline with just gpt-4 turbo and compare.
100%
Next projekt is a Mo(MoM) model where we compare the result of all 3 MoM systems with these systems again. Think of recursively going down the rabbit hole.
lol
🤣😂 I just have to say it. Monarchy seems to be the solution for most problems.
However jokes aside, this experiment turned out really great, and fascinating how the models behave in the different setups and interactions.
Monarchy would be best only if the king is also by far above others in mental capacity, in all areas.
@@someverycool4552 Agreed
@@jiyuhenHis demo is not really a monarchy. In a real life a king is able to have his own ideas or solution, he has the freedom to completely ignore proposal put forth to him. But in this demo the king doesn't have any of those freedoms and powers. He has to judge and decide what others put forth to him. Let the king have his own ideas and the freedom I mentioned earlier and I bet the results will be different.
This was a fantastic video thank you...brought me up to speed on the various models and structures of their hierachys was fascinating. Not dabbled in agents for a while so was nice to be surprised at progress in this area vs my early attempts that spent tokens and produced nonsense.... question....do you have any teams of agents setup doing actual real world useful stuff? (I'm sure I'll find out by watching more). Thanks for the video.
I think that it would be better to estimate performance when you have some simple model at the output level.
GPT-4 can operate with all of provided problems without previous models solutions
yeah true, will try diff configs for this setup
Makes sence we create theoretically using something between fast and slow thinking. Maybe something like q* needs to use Claude 3
how exciting what an awesome video thank you
thnx :)
Great content!
wow, you have an innovative mind. 🤓
22:38 it only got one vote because each model provided 10 unique answers. This shows that the democracy model isn’t a good architecture for all kinds of problems (specifically, the ones where the answer is determinate).
I would probably choose to do the tally locally using an optimized solution a simple python script bypassing any AI that way you eliminate the models sucking at math and use architecture specialized for solving the math problem of counting 😊
I was following the explanation In the beginning about the different architectures very well, until he opened the code😂
haha, its not that bad
lovely
thats great can you please give us paying users access to the github
quite impressed by these MoM / MoE model architectures, however, I am not satisfied with the leetcode hard problem used in the test set. Testing the llm generated response on just 3 test cases is not a valid determinant for the architecture's capabilities. I could see the solution is itself in the order of 4 ( O(N^4)) time complexity, which essentially is Brute Force.
A better bench mark would be testing how well it can optimise the code and handle diverse test suits IMO.
Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.
The api call cost will go out of roof.
So this is all in the GitHub repo to play with I assume?
How much did each "tournament" cost in terms of tokens?
since 8 models was local on ollma, not much =)
You feed the answer into the next model and ask it is true or false
All seems rather pointless without comparing to the results by gpt4/opus on their own ...
yeah that was my regret, should have done a "basetest" to comp
means that autocratic systems are better compared to democratic - lol
haha lol
The marble problem proves nothing. Every model vendor cheats on some problems and sticks these problems into the training datasets. Same for other problems. You need to use new problems every time. Otherwise these reviews are b.s.
you need to go deeper into their reasoning because some of the wrong answers are actually right
are you sure chatgpt4 wouldn’t just solve all these problems without the other models
I was thinking same. So the problems should be more adequate. But it was a proof of concept, very nice work. I planned aomething similar with websocket agents last summer, but it became too expensive to test for a hobby. Now it is right time because ollama!
That’s a great question. In the past at least, ChatGPT4 failed the marble and the apple questions.
Do you cut wood with a hammer?
Different models have slightly different training, context, and perspective, so this creates a broader perspective to approach the problem at hand. This will not be practical for everyday prompts.
@@jichaelmorgan3796 I disagree on the practicality point. The way it is Setup in this video from a testing the methodology perspective yes it’s not practical..
But behind it is actually very practical. Because all of this stuff that is going on can be done in code so the user doesn’t see everything that goes on behind-the-scenes and just gets the final output
This video is a whole lot of fluff for a very simple concept: combining outputs from multiple LLMs to get better answers. The creator makes it seem revolutionary with fancy names like "King," "Duopoly," and "Democracy," but it's just basic ensemble methods. They feed the same problem to a bunch of LLMs (including weaker ones like Llama), then use those answers as context for a stronger LLM (GPT-4) to make the final decision. The "Duopoly" is just GPT-4 and Claude arguing about the best answer, and the "Democracy" lets all the LLMs "vote" on the best answer. He doesn't go into detail about how the voting works or what constitutes a good vote. The results are a mixed bag, sometimes better, sometimes worse than just using GPT-4 alone. The real takeaway is that ensemble methods can be useful, but this video overcomplicates it with theatrical names and presentation.
Why is nobody talking about "Where's Daddy?" or "Lavender" or "Gospel"? these are the most advanced AI systems and are already committing mass murder!
Why do you bots make the same stupid fearmongering comment on every post?
@@tellesu I'm not a bot, I just want to hear smart people like this guy break it down. more people need to be talking about this, it is AI development towards Armageddon
@@tellesu that's not fear mongering it is simple fact. A certain tribal state is using advanced AI to engage in the mass murder of civilians without any human oversight.
The where is daddy program literally watches targets leave so-called military locations and orders an attack only after they are at home with their family. It is deliberate mass murder of civilians, and it is powered by AI.
your mom
@william5931, your dad. :)
Perhaps it would be even more powerful to make a Chain of Models/Chain of Experts architecture that has a line of models and each model builds upon the previous models' responses. Making a Network of Models would be even better, where you have layers of multiple models and each next layer gets the previous layer's outputs and ultimately combines everything in the end to form an answer to your question.
very interesting, noted, tnx:)
If just one agent is wrong the entire work product crumbles. Wouldn't you need a hallucination check, or some kind of redundancy at each stage?
@@c.d.osajotiamaraca3382 Yeah, I didn't think of that. Maybe in the system prompt, it could say there is a chance that the output of the previous model(s) is incorrect and to consider that before making a decision. Or between each layer it could have an internet search if applicable. Not sure how I would program that as I don't know how to do HTTP requests but it seems possible.
A note: the marble problem has actually been solved in Matthew Berman's testing by WizardLM-70B-V1.0-GPTQ, Mistral 7b OpenOrca, Mixtral, Mistral Next, Mistral Large, and LLaMA 3 70b Groq.
That is however only 6 of the 32 models he's tested. I will note that in my own testing Reka Core passed the marble test while it failed when Matthew was testing it.
Add a gamifying token layer to learn which models' votes should be heeded/ignored in which situations
yeah cool idea, noted
Great experiment!!
I wonder if adding in reasoning to their answer, if democracy would vote differently? Or if the answers could be discussed before voting. Having a bunch of AI/people voting on bad assumptions is never good. "before voting, discuss each answer for a) meeting all the requirements and b) being the most accurate".
interesting =) noted for future itteration
Waw this much be so expensive lo but great
I came to the same conclusion that arbiter currently works best.
actually, the most likely place for the marble to be is on the floor, because it is no easy feat to turn a cup with a marble in it upside down and put it on a table without the marble falling out before.
that would be the big brain human answer yes hehe