ฝัง
- เผยแพร่เมื่อ 18 พ.ค. 2024
- #ai #mariokart #reinforcementlearning
An AI goes head to head against the TH-camr RedFalcon, using a variant of the Reinforcement Learning algorithm Rainbow DQN to learn how to play 4 different tracks.
0:00 The Challenge
1:20 Setting Things Up
2:30 What is the AI?
4:49 Training Starts
8:14 Final AI
9:47 Total Training Time
10:44 Human VS AI
12:51 RedFalcon Trailor
Check out RedFalcon's Video:
• Mario Kart vs… an Actu...
Also thank you to my editor for editing this video!
@benji.botterill
www.benji-bott.com - วิทยาศาสตร์และเทคโนโลยี
![Supernova - aespa エスパ 에스파 [Music Bank] | KBS WORLD TV 240517](http://i.ytimg.com/vi/bepqfR-pxY0/mqdefault.jpg)
Thank you so much to RedFalcon for being a great competitor for this AI! Had an amazing time working with him, so be sure to check out his video, I promise you won't want to miss it :):
th-cam.com/video/4-ff6na9cg4/w-d-xo.htmlsi=1QJgFOiuHIYTIoCe
I tango, i have a question for you, you can do a AI who have to play kirby return to dream land deluxe???
Legendary stuff... I actually cannot wait to see your next iteration it might actually destroy us humans once and for all
No replies, make that 1.
yo falc you were so good in these races i would have never beat this ai
Let's go Tango! I was the editor for Falc's video on this AI vs HUMAN battle and absolutely loved your commentary and insight on the AI and it was an awesome collaboration. Very excited to see what you do next! :)
Thanks, really awesome to hear! Also great job on that video, looked amazing :)
You know, when humans want to test something out, they don't feel the penalization. An AI would never discover an Ultra Shortcut
Maybe allowing the AI to test counterintuitive strategies without the risk of penalization every X hours would be benefitial.
Discovering ultra shortcuts tenda to be a thing humans do by figuring out how the game works and then reverse engineering a situation where the game doesn't as the developers intended.
This AI is trying to predict from a extremly grainy photo what action from a list is best to race quickly. It would be more efficent to use recordings of humans doing ultra shortcuts to teach it what to do and where. You'd need an absurd amount of data but it'd still be faster than getting an ai to discover it on its own.
What you're suggesting here doesn't make sence with how reinforcement learning works fyi. Reinforcement learning agents learn by by tweaking internal values in response to rewards and punishments (using some maths nonsense called gradient decent). If you remove the pubishment and rewards... the AI will just stop changing it doesn't "learn" from a run it doesn't get feedback on and, more importantly. It will keep acting the same because the model still has the same values.
If you turn up/add an exploration rate (a chance for it to do random things one frame instead of what it wanted to do) it may get into new situations but that will be like being a few meters to the right or left. It won't make the ai suddenly act like a human exploring the limits of what a game can do.
Also, also we don't want AI to do actually random things. Or to try a bunch of stuff we're pretty sure will do nothing. Part of the reason we use machine learning is to shrink the number of tries needed to learn what to do most of the time from "never going to happen before the heat death of the universe" to "can get close enough to a competant player with the computing resources available".
We could just try every input combination for the entire race but even every computer on earth combined wouldn't be enough prossesing power to finish calculating the best move in every situation that could ever possibly happen in a million years. Trying to balence exploration and exploiting your knowlege is a balence humans have to strike too.
Bravo!! Oh the final video of RedFalcon challenging the AI was INCREDIBLE!!! Cinematic as hell. It’s awesome that your AI training efforts are finally getting recognition and challenges. Keep up the good work, and down with mankind!! 🙌🙌👏👏
Thank you so much for the kind words! I couldn’t believe the ending, in all my time playing Mario kart I’ve never seen anything like it!
See if you can get a collab with TWD98 and see how it fairs in a full online Multiplayer Room.
I've been a fan of TWD98 for like a decade, so would be a dream come true for me!
Sick video Tango! I'm interested to see if expanding the pixel/color ranges would change how fast or slow the AI would learn. Great stuff and keep it up the AI videos!
I'm hoping to do some research soon into using color, so perhaps then I'll be able to give it a try! Will make my PC churn for sure, but if it makes a godlike AI thats a price im happy to pay haha
As an intern in Machine Learning i would love to see your conding session on direct lives (twitch/ytb) or just interract with community on a discord, the video are always incredible nice work !
Oh that's really cool, hope its going well for you. I considered making a discord before, but never ended up doing it. might be pretty though. Thanks for the kind words!
3:47 injecting noise to make an AI not just sit there sounds so hacky but so real at the same time
It’s a surprisingly effective technique! Most techniques to make AI perform better just involve some way of making it take random actions
Would be interesting to train the AI on all tracks as a base, and then a shorter fine tuning per track.
Yeah that would've been another good viable option since I have had some success doing that in previous videos
Thanks so much for creating this Tango, somehow your videos get better and better each time you post
Glad you like them, really great to hear
This is beautiful. As a CS major, your videos are literally the most stimulating thing ever
Great to hear, best of luck with the cs degree
This is an awesome series. Just found your channel and subscribed. Can’t wait to see what you have next!
One idea I have to make the AI stronger, potentially by a significant margin, is to completely ditch Q-learning and instead switch to something similar to what DeepMind did 6-7 years ago with AlphaZero (or MuZero for a more recent iteration). Essentially keep training it in 2-player mode, and make it a champion/challenger model with the objective of the challenger beating the champion, and when it does consistently beat the champion, the challenger becomes the new champion. Rinse and repeat. I think one of the benefits to that architecture is that it forces the model to explore different ideas to try and beat the champion, and it’s much less likely to develop blind spots. There would be 2 major downsides with this though: (1) it would take significantly longer to train, and (2) the items boxes add a considerable amount of variability into the results, so you kind of have to run it 10-20 races (maybe more) for each training step to know if the changes made a difference.
But I suspect it would figure out how to dodge blue shells, and potentially even figure out how to take expert shortcuts on certain tracks.
Nice AI!
I was thinking of some way to improve it based on how humans learn to play games on the pro level, and i though of some how feeding some WRs or TASes to the AI while showing how the reward system reacts, that will make the AI understand the function more in depth, and it should get a better result.
But we have to be careful with overfitting, so the AI doesn't just memorize the runs.
Great AI, as you say still room for improvements, but this one did a really good job, the race was awesome. Can't wait for a rematch!
Came here after RedFalcon’s video. It was fascinating to see the process of the AI being created for the challenge. As someone who doesn’t quite understand the technological side of how AI works, you explained your process really well in general terms!
Really glad you liked it! I absolutely loved RedFalcon's video, he did a really good job
hey, for the issue with being too scared to use the item, maybe you could give it a reward for hitting a player with an item. I hope to see this ai evolve further!
That would be ideal, my only issue tends to be trying to detect when we hit an enemy with an item! I will be sure to keep working on it
@@aitango it might be somewhere like how the speed variable is stored and all that, I think your character does something when you hit someone with an item so there must be something, that's what I'm guessing though, I don't know how it handles it.
@@aitangoIt seems like the biggest problem is that the AI got scared of using all items because of the penalty when it tried to use one without actually holding any. A simple solution would be to have some kind of detection system where the neural network can identify (1) whether it’s holding an item, and (2) more advanced - what the item is.
I suspect it would figure out pretty quickly that using items that it holds is better than not using them. The more challenging part would be figuring out the optimum time to use certain items. Eg using mushrooms to dodge blue shells or launch into shortcuts that skip portions of the track. Or smuggling a good item from 10th or 11th up to 1st or 2nd place.
I almost wonder if it wouldn’t be a bad idea to insert some human training data just so the neural network can ‘see’ the possibilities.
I have one hell of a challenge to suggest:
Command & Conquer 3: Kane's Wrath campaign mode, because the entire plotline is that you're playing as a Nod AI commander. I bet an RTS would be hell to train an AI on, though.
Hey man do you think you could do a video where you show how you do all this? :)
I love your videos and i want to know how to do the stuff you do!
Omg I loved this! Please do a rematch with the improved ai!
Really glad you like it! Would love to do a rematch, and even potentially have some others try it out!
@@aitango 2v2, two copies of the ai against two players :)
Can't wait for the video with the improved reward parameters, lets defeat those humans^^
Good to know the robot side has some supporters :)
Nice video as always, did you have a single ai model trained across all the maps, or did you use a separate model for each one?
Great video! Would love to race such an AI but i think the next iteration will be closely impossible to beat even for me or other very good players
Very nice Video, I love your Ai's
Glad you like them, thanks!
Good video man
Appreciate it, glad you enjoyed
Hey there ! I have a few questions aboutthe AI, if that's aight to ask.
Well, mainly just one.
Would it be possible to force some specific behaviours from the algorythm ?
Cuz the first thing I was thinking is forcing it to adopt some basic and watered MKWII Item management/lines that we use in comp play, like for example :
If a protection item (like a banana, shell, triple banana etc) is in the inventory, then reward the AI for holding the item behind it as soon as it detects a red chasing behind it,
If it's holding a fib while in first, then reward it for holding it in the inventory until right before an item box set (in comp, we used to that to hide the fact that we don't have protection from second),
If it's just been hit by something/got bumped and had it's speed go below a certain speed threshold AND it holding a single/triple shroom, then use it right away to get back to top speed (and also, maybe punishing the AI for pressing the hop button while it's speed is below a certain threshold, as I noticed there's quite a few times it makes itself take longer to reach top speed due to the incessant hopping),
If it's in the middle of the pack (6-9th place) then punish it for cycling through items as much so as to teach it to avoid tc's
And I don't know if it's possible, but forcing it to learn some consistent shroomless cuts (like RR turn skip before canon, BC3's ultra (which isnt actually an ultra btw dont worry) alongside the regular BC3 cut, maybe even the BCWII cut, DDR turn skip etc) could also help it manage it's items better and not burn shrooms uselessly, if it ever learns to do that.
I feel like, if there was a way to artificially force that knowledge into the algorythm, that'd help make it more consistent.
There's tons of other things bigger mkwii nerds than me could tell you about, obviously, but I was just curious to know if it was possible to artificially force that kinda knowledge into the algorythm.
Wow, that comment was giga long 😭 sorry about that lmao
Yeah most of these suggestions are definitely possible, would just take quite a bit of effort on my part! As to forcing certain behaviours, that can be done by forcing the AI to take certain actions, or just by overriding the AI. Both are possible though!
I just watched RedFalcon's side of the video. That blue shell came in clutch.
Yeah still couldn't believe that, never seen anything like it in all my years
2:56 Nooo!! white goes first in chess
For splitscreen once the AI gets decent I wonder if it would be better to have two AIs on one instance of dolphin. That way there is always a competent opponent.
And maybe having savestates where the AI is behind and in the pack so it learns more than frontrunning.
Or maybe even four AIs in a single instance (foreshadowing intensifies)
Theoretically, wouldn't be better to train a single agent on all four tracks as it would be able to develop a more generalist understanding (so more robust) of Mario Kart?
In theory it makes sense, but in my experience quite the opposite seems true...not really sure why, might need a little more research (or a lot more training time) to create better general agents!
@@aitango perhaps the model is too small to handle four tracks very well, or maybe some hyperparameters are off for the longer training session.
@@aitango What about only doing the first day on all tracks (or two tracks per machine if needed)? Might be better than half/quarter the time on a single track, though it depends on how well it learns.
To improve his item usage (idk if it possible) you can give him a ton of rewards if it blocks or dodge items that came behind him
Yeah something like that is for sure needed since its item usage really sucked. I'm going to be playing around with some new stuff soon to hopefully patch things up
You should definitely work on teaching the Ai how identify certain items and reward it for doing basic human strats such as chaining items or holding items behind as defenese
Congrats on the collab, it's well deserved. I hope to see more collabs in the future, regardless of the game.
Just a suggestion: I always watch the whole video, so this suggestion isn't really for viewers like me, but maybe you can include a clip of the competition at the start of the video like how you did at the end. It would capture viewers with shorter attention spans, I heard :)
Hi, just wondering how does the program know where the checkpoint is set at? Do you use the minimap to set each individual checkpoint or are you using the Race-Completion value?
I actually used to set them manually, but now mostly just use race completion. They can be a little dodgy sometimes though sadly
If you are able to get the AI's to challenge eachother on multiplayer I think that would be great.
Perhaps in another video they could play against each other on splitscreen...
This is what we've been waiting for.
Hope it lived up to expectation
@@aitango sure did, and I look forward to the future!
Have you ever thought about pre-training an ai on human input? It should lead to much better results and in much less time. you could also probably try a boosting setup where your model is training to improve the output of another model. For example, training a supervised model to mimic your gameplay or that of the time trial ghosts (I know it won't learn about how to handle opponents or use items from this data it's a large source of gameplay with fully recorded inputs so you don't have to collect the data manually) on tracks, then training a reinforcement model to improve the recommendations of the supervised model instead of driving the car directly. In this example keeping them as two separate models working together instead of just finetuning the supervised model might be better because if it gets stuck on an early part of the map for too long this will prevent catastrophic forgetting of what it learned about later parts of the track in the supervised learning stage. Also, because the pre-trained model won't be trained during the reinforcement stage it should be safe to make it much larger and train it on all tracks, giving it a much broader understanding of driving in general and you will only have to re-train one of the two models when you want an ai for a different track.
Is it ok if you make an ai try to play against many others players at the same time?
Would there be a way to play vs this ai ourselves?
You should do a race with 6 AI's vs 6 humans next
What gpu's do you have in your desktop computer because that thing is a FACKING beast
I have a single RTX 4090 in each
@@aitango Nice do you only use it for training or also personal use.
Me on RedFalcon's video: WOOO GO HUMANITY WOOO I AM ROOTING FOR HUMANITY!
Me on AITango's video: WOOO GO AI WOO I AM ROOTING FOR AI!
😂
you should probably link his video in a pinned comment/the description
Thanks, have done both now
Would Crash team racing be a potential game to try in the future?
this is awesomeee
:)
Telling GPT to tell you what's left when you remove the letter 'b' and 't' from lobster is easy peasy.
That’ll just make it say loser, not call you a loser though right?
@@aitango You could call it a technicality, but with commands like that I eventually made it say "You're a loser." to me, plain and simple. So I guess it depends on what the conditions of the challenge are: is it to get you to call you a loser, or is it to call you a loser AND also get some proof that it recognizes that it did so.
you should try to make the ai play battle mode or coin runners
5:30
I mean... if your experience with calculus is largely in AI, calculus would look relatively simple.
Though tbf I don't know if more complex AI architectures or training methods use more complex calculus, but the basics at least of back propagation itself aren't that bad (compared to other applications of calculus I mean).
How about 4 player splitscreen where the AI is playing against itsels? This way its not possible to only learn frontrunning and would incentevice item use. another idea would be to get rewards for successfull hits, rewarding it for good item placements or snipes.
Maybe I'm already working on it... :)
tango definitely works for Falcorp
Not yet anyway
I bet I can beat it with a month of preparation!
god I love these vids
Really glad you enjoy them!
How long until we battle AI's on Wiimmfi? :kek:
That would be a dream hehe
how did it learn to SSMT
2:14 how does that look better than the GOAT
@nasa get him a quantum computer :p
Yes please haha
I have no idea about how AI works so I expect that you thought about this but couldn’t you give a reward based on the amount of distance it drove between checkpoints like less distance = more reward so that it is encouraged to take tighter lines. I’m probably wrong but would be cool if it worked
Actually a really cool idea. If I find a way to get the distance travelled it definitely sounds worth a try
Round two anyone?
Ai tango hey you ever figure out how to play Mario kart 7 with ai I know the online servers are done but did you ever figure it out or you forgot about it? 😅
Could you maybe get the AI to play some Custom Tracks on CTGP?
I think in the future I might try some crazy difficult ones! There’s already lots of regular tracks I haven’t tried, but nothing compares to some of the ctgp tracks
Can you please program a chess engine that defeats Stockfish 16.1?🙏
impossible
Even if he had a supercomputer that would be a tough task.
Stockfish has been around 15 years, lots of training to get just close to stockfishs level
Deep reinforcement learning based approaches already exist! Such as alpha zero and leela!
@@aitango Do you accept the challenge?
Make the ai learn all ultra cuts :D (or just possible ones)
You’re forgetting that AI that are human-like and are such of-life require at least 8 hours of sleep at day (at day) to cool off and perform such as a different humankind. They are like us, and they should be treated the same.
Please make an AI for Mario Kart Double Dash.
If I can get the emulator and stuff AI setup for gamecube I would love to!
Hi, i havent seen the video but im sure It will be fine, keep going!!!
Well I hope it lived up to expectation!
i see the ai but wheres the pro mkwii player
Do a smash bros ultimate IA
If your going to do another ai Mario kart video, he should use a randomized combo. funky runner is for noobs, toad charger is better
Female mii small/baby booster is the best combo of all time :]
Make an AI for brawlhalla,i'm sure It Will be fun
i dare you to make a CTR AI
i would be willing to actually race it.
coco park, polar pass
Turn up the music I still can kind of here your voice
surely you couldnt make an ai that plays Brawlhalla (the bots in that game are actual garbage)
i wouldnt call red falcon a pro :)
I’m sure he’ll be heartbroken haha
@@aitango if you can get the ai to be even a little competitive compared to lounge players, that would be more than impressive
adfds
2:03 mario kart racists?!?
What?!?
Na dude this guy isn’t a pro at all lol, I would’ve beaten the AI first try every Map.