ฝัง
- เผยแพร่เมื่อ 1 มิ.ย. 2024
- #ai #mariobros #reinforcementlearning
AI uses a variant of the Reinforcement Learning algorithm Rainbow DQN to learn how to play a level of New Super Mario Bros within a day.
0:00 Intro
2:34 Coding
4:23 Training
9:10 Final AI
Github I need help on:
github.com/Felk/dolphin/issue...
Also thank you to my editor for editing this video!
www.benji-bott.com - เกม
You should make it so that collecting coins rewards it. Coins in Mario levels are often used to tell the player they're doing something right, so if the AI tried to get the coins it would probably learn faster
It may be possible that the AI is learning that coins are good naturally. That is, it might begin to "believe" that collecting coins is going to have a higher probability of receiving a higher reward.
@JoziGlitzh So from what I understood from the video, the AI is making decisions only based on the pixels that it “sees” - the pixels + the feedback from the reward function. It will try to maximize the rewards based on that (pretty much like a human).
I’m guessing, then, giving rewards for coins would make things harder as you would need to gather data from the game aside from the images.
If you think about it, deep down the AI probably does predicts rewards based on coin locations, as they mostly indicate the right path to ending the level.
@@TheGhostWinner I understand, maybe if the AI received colored images it could then detect coins, though, for it not to confuse anything yellow with a coin, it'd have to detect its shape too, which could be quite complex and it may still confuse other yellow things for coins, especially if they're round.
And yes, what you say is true. If it sees that the path it has to follow for it to be rewarded has coins, it would probably associate coins with playing the game correctly, however, that would take a lot of time, making it so just collecting a coin rewards it would speed up this process a lot, and as a consequence, it would probably learn quicker how to beat the level.
*proceeds to put it in a Mario Maker troll level where all coins lead to death*
@@TheGhostWinner How would retrieving the coin count from the game and using that as a reward function pose any problem?
As I understand the AI is already being rewarded for increasing its x position and punished for dying. Couldn't you just drop and replace that x position with the coint count?
i feel bad that you don't really get enough attention from people
because the content you are creating recently are good but also are getting better and better!
just keep up the good work and make those ai's shine brighter than ever!
Thanks! Good to hear you think its improving, hoping that if I stick at it for a while and the content is good, the channel will grow!
@@aitango For sure! well let's hope first that there is a way to get the shaking to work within dolphin!
@@aitangoit definitely will!
What’s really fascinating is being able to visualize the predictions, knowing it’s the same thing that your brain is doing in real time every time you pick up a controller.
I think one of the things that really stumped the AI could've been moving platforms. Not because they're hard, but because it can't see that they're moving.
If it only gets a still image at a time, and outputs a movement for what it should do based on that image, when the next image is given to it it's forgotten where the rotating platforms were in the previous frame, so it can't compare and see that they're rotating. This can lead to it landing on what it thinks is a walkable surface, only for that surface to then rotate out from under it and become a slope that it falls off.
That's a good point! I forgot to mention this in the video, but the AI isn't actually given just a single frame, but rather than last 4 (this is often called framestacking). Giving it the last 4 frames tries to rectify this problem by at least giving it some sense of motion
@@aitango What frame rate were these last 4 frames at?
That is an interesting point. I wonder if the pure reward for going right, excluding the penalty for death, is partly responsible for the slow training. For instance, I have noticed that it rarely chooses to do nothing when that might be beneficial on the rotating platforms. What it does instead is go right too much, then corrects by going left, then tries to go right again because all it learns is to go right. This process makes it more likely to die from failing at attempting a leftward correction.
My first thought is to introduce a penalty for going left, but that could also slow training. My second thought is to introduce a penalty for rapidly changing direction (i.e. moving right/left immediately after moving left/right) because that indicates you're correcting a suboptimal decision. You could make the penalty for changing direction decrease with the number of frames that the action "None" is taken in between changing direction. This could introduce problems with learning concepts like wall jumping, so further modifications would be needed, but the point is that "teaching" only to move right as much as possible might lengthen the training time for navigating moving platforms.
It would be insane to train the AI playing Mario Maker. Maybe being able to do Super expert endless or something.
That would be absolutely sick
that's legit a genius idea
I wonder if this would count as an adversial generated network (gan). One ai builds levels, one plays.
It would be cool to see 4 player AI.
there tiktok lives that sometimes show that
@Thisaintarealemail lol that would be insane
@@musicviolamusician4392TAS
Really impressive! Would love to see an AI against even harder Mario levels at some point!
I think if the he can figure out a way to use the shake function and the nunchuck through the forum in the video then that will most likely be a future video.
I will definitely be doing more Mario levels in the future! As mentioned, at current an annoying number of levels do require shaking, but at least for now I'll try and find some good ones that don't require the shake
AI v.s champion's road
@@aitangoI'd love to see it beat an entire game someday
I’m kinda curious how it’d tackle a tower level. I imagine it’s way harder to teach an AI that already likes just going right that “ _sometimes_ go up = good, _sometimes_ go right = good,” and I think that could be interesting.
This is so interesting! I wonder how long it would take for AI to beat the game.
Really glad you think so! I would be very curious too! I think some levels would require so changing of the rewards though, since some levels require you to go up and down rather than just right! I will have to try the AI on some harder levels though so I can give you an idea haha
oh hi mayro
it'd need to train on every level at random probably, since just training it on one dataset causes massive overfitting issues
just discovered this channel, don't know why this channel is so small because the content's legitimately pretty good
Thanks, glad you like it
It's a good thing you put that reminder at the end! The three popup animations and bell sounds were obnoxious enough to win me over, but I can't possibly be expected to remember for that long! There's three whole minutes between them!
I mean, I wasn't going to subscribe at all, but that nag really convinced me. I'd have completely forgotten about the buttons I can clearly see on my screen if I hadn't heard that bell!
This is really cool. By the looks of your next video, it seems that you’ve figured out the motion issue, but if you haven’t Dolphin has a pretty great way to emulate the motion within its own settings. Shaking, tilting, and pointing are all mappable to any keyboard or controller input. If the AI can input through either method, you should be able to circumvent the need for a gyrometer. Best of luck. These projects are really cool.
Really glad you like it!
I haven't actually figured out the motion issue yet, the Mario Kart video uses the gamecube controller since Mario Kart allows it. I've looked at Dolphin's settings, but sadly I'm using a modded version of Dolphin which allows Python code to interact with the game, but it doesn't have nunchuck/motion input yet :(
@@aitango I think dolphin's pipe input could help here it lets you use python / whatever language that can open files as an input method
it might be harder than the modded dolphin tho
Great work again. I'm loving your stuff. Thanks again.
It seems that the AI hasn't gained the experience necessary to associate damage with failure. It's made the connection for the koopas at the end, since the stun from the hit greatly increases the chance of falling into a pit... however, in other parts the AI walks through koopas when they don't directly affect the ability to clear the level. I think part of this is that the trial contains so few enemies as a whole that the idea of retaining a powerup to make handling enemies easier is lost. Since the AI likely never encounters enough enemies directly in any run to get any sense that getting hit is seen as a bad response (low impact on forecast rewards) it never makes a route which directly avoids them optimally.
I think stages that focus less on pits or contain more direct stage hazards (castle levels, for instance) would be interesting to see, as the AI would learn the value of Mario's powerups naturally as they give a great margin for error when moving through the stage. It would also be interesting to see if the AI learns how to acquire powerups to increase it chances of success.
I'm so excited to see another AI channel! Here's my subscription and like, you're doing great!
That's really great to hear, thank you so much! I'll try my best to keep making videos people want to see!
I'm going to let my kid play smb 24 hours straight, punching him every time he dies and giving him candies every time he gets to the end, he'll learn eventually❤
Real life reinforcement learning, I like it haha (minus the punching children of course)
Good job on this it was amazing to see how Mario Bros worked with the AI playing.
Thanks, I hope to do some harder levels in the future
I'm curious if any of the AI you've trained have been able to generalize their skills to other levels?
I guess the issue would be that any new items or platforms or enemies would be incredibly hard to deal with unless the AI had developed some good generic edge detection or something.
Could be interesting to see someday, even if its awful lol.
This AI probably wouldn't be able to generalize very well since it was only trained on one track, but I imagine if an AI was trained on a few at the same time it could start to generalize. I'm definitely looking to do a video on this in the future!
I would have suggested nsmb for the DS to start this out, as every button can be mapped more easily. It also helps that the ds is naturally low rez. If you were to do a game like sm64ds, I'd recommend a much bigger 'brain' and a much more complex reward structure. Probably something outside your scope.
I would love to see Google and Nintendo collaborate on training an AI to beat the ORIGINAL NES SMB1. A bit of an "I scratch your back, you scratch mine" scenario, where Google gets to show off its AI technology in an interesting proof-of-concept that nearly ANYONE would understand, and Nintendo could get some support from Google in their future tech problems as a favor for loaning their game rights (i.e. How do I make the fastest and most optimal secret area layouts to incentivize users?).
Honestly, one thing I really want to see is how well an ai trained on one level does in a different level
it would fail directly
@@Photo650D But how long would it take to learn a second level? Less than the first?
thought this video had millions of views for some reason, this is amazing! I'm going to subscribe
Thanks! I hope it gets millions of views haha
@@aitango How the hell are you this underrated. EXPLAIN YOURSELF
I'm not sure I can explain that one, but I'm hoping it changes soon!@@jaythecoderx4623
I hope so @@aitango
Your channel theme of training AI in classic video games has INSANE potential
Hi, I really like your channel and I'm impressed by the progress since the first video on ia for mario kart ! I was so inspired that I too wanted to get involved in reinforced learning for dolphin games. But I'm having some problems setting up the environment. You used the dolphin fork with integrated python, right? In the video, you don't run the script from dolphin but from a custom interpreter. Is it possible to find out how to do this? Thanks, and good luck with the youtube channel, with quality like this it's only a matter of time before it blows up!
Hey, its cool to see others trying to work on this kind of stuff! Sadly setting up the environment was a real pain and took months... The way I did it was by using a custom interpreter (which controlled the AI) to launch instances of dolphin with a script running from command line. I then used sockets to communicate between the main program and dolphin. The main program would run most of the AI code, and the dolphin side would just take actions, and get the state and reward. Was not easy though. Thanks for the good wishes though, I'm very much hoping it blows up at some point!
Very coool video, you deserve more attention :3
Do you consider teaching other people how to program something like this in future? I would be very interested in learning more about this ^^
Thank you very much! I have considered it before, I’m not sure it would get as much attention as videos like this, but I’d really like to help people get into AI so it’s something I’ve thought about a lot!
Bro this is amazing, thanks for sharing.
Thank you for watching!
subscribe button right as you explain reinforcement learning...well done :)
Welcome aboard, thank you!
I want to see AI beating Ganondorf in TOTK
It's interesting to see it damage boost on the first screw knowing the fire flower got restored upon reporting.
Im working on something like this, but im struggling with this one aspect.
How do you determine when Mario has moved right in the stage?
Is it just as simple as when hes pushing right reward him?
Or is there some other way you were able to determine Marios rightward progress across the stage?
I saw that you said you have a dolphin mod that allows you to read/write from memory. What are you using for this and how did you determine which memory adresses to look at?
Here’s a thought that you don’t have to do since it’d be a LOT of work but something I’d like to share! If you taught an ai all the levels of Super Mario Bros Wii, I wonder how long it would take. Comparing its time to a professional speed runner would be really interesting!
I find it interesting that the confidence in *all* possible actions goes up. The actual margome of difference between different action is actually pretty low most of the time. I wonder what would happen if instead of picking the highest action only you did a random action above a certain threshhold, or within a range of thd top one, to encourage the ai to be sure of its actions.
Astonishing. But this video raises two very interesting questions.
1. After training the AI on this particular level, will it beat another level? (at least, a simpler one)
2. How many levels should it learn to pass in order to beat the whole game (any level)?
And another, technical question. Which topology are you using? How many layers, which kinds?
Wait, so the AI needed 24 hours to beat one NSMBW level, so does it mean that technically to beat every NSMBW level, the ai would need (24x76) 1824 hours, so around 76 days?
So if it was a single AI, it would likely get faster as it could use what it learned on one level to help with another. Much of what the AI learns early on is things like detecting objects, edges and learning what the different actions do. That said, I have no idea how the AI would handle some of the levels on later worlds!
5:50 “Hopefully it would grow out of its phase of being mostly incompetent…”
Same, little AI… same.
you should give it a reward for finishing, and a penalty for losing your powerup
9:45 how the hell did that fireball go up here?
I like how it goes from fumbling to speedrunning strats before ever finishing the level
This video needs more views.
Imagine someone trains an AI to speedrun Super Mario 64
You should change the rewards to add Points to the rewards. Cause generally, the more points you get, the better you are doing. Plus is would make the levels look more cool as the AI collects as many Coins as it can, goes further to the end, and even the flag pole gets you points, so the AI would be heavily influenced to grab the flag. thus beating levels.
I love this video. AI is such a fascinating thing to look at in games.
...But also, what in the world did you just call a 0?
Also, I barely use C++, but I am willing to look into helping you figure out the issue!
Really glad you're enjoying the content! I can't even remember what I called a 0, this video was a while ago haha
Thanks, that really means a lot of you're willing to have a look around, I'd love to try and play other games that use the nunchuck!
Introducing Super Guide! Lets go Luigi 🔥🔥🔥🔥🔥
What would happen if for these games you set a big reward for ‚get to the end ´ i would be curious to see the kinds of pathways and solutions ai comes up with. From what I understand about that would take forever I guess ?
So currently the AI does get a pretty big reward for reaching the end, just to encourage it to do exactly what you say! I include the rewards throughout the level though, as you are right that without them it would take forever to train since the AI wouldn't be given much guidance
Great content!!! I just have one question, how you pass to the AI the x cordinates of mario, so it can get a reward for going right?
5:51 That was my parent's hope for me too. Sorry mom and dad.
I wonder how much its strategy would differ if you rewarded it for the game’s built-in score meter instead of moving right! It would go for coins, enemy kills, star coins, and the top of the flagpole instead of just focusing on moving forward. Could it 100% the level?
Would it be possible for the AI to play the whole game? Like maybe training it to speedrun or something.
One problem: boo mansions
After fully training the AI try to also run it on different but similar looking levels to see if it picks up on any general learning
Yeah that would be cool, I'm looking to do that on Mario Kart as well. I doubt after learning a single level it'll generalise too well, but I think it could if it trained it on a few simultaneously
very cool video! would it be possible to do an ai like this in stronghold (2)? Would love to have stronger ais. Probably a lot more complicated but technically still possible?
If i understand... the ia is trying to beat the level by beeing blind and deaf and the only way it have to know how good it does is by a score... no wonder it learn so slow... if there was a way to makes him recognise the platforms and the objects arround him or at least a way to makes a "virtual map" of what it have discovered (just like you remember the layout of your house and could replicate the path from your chair to the kichen while beeing into a big empty room)... it would probabily be far better in no time...i mean... if it is just a random number generator who is refuse to repeat the failed combinaisons... it would takes an eternity but would eventualy succede...
@AI Tango I am wondering how long it will take for the AI to discover that to progress in the "teleporting screw", it will be necessary to stop the temporary progress to the right, "shake", and jump near a range of appropriate timing.
Since the levels are deterministic, it is highly probable that by then end it was overtrained on this specific level, and that it would fail miserably at any other level.
Would it be possible to make the ai output 2 buttons at the same time when it wants to ?
That AI really got good. And indeed does look somewhat like a speedrun strat.
I love youre videos about ais and I wonder in what programming language you write. Is it python? if yes how do i write reinforced nerual networks?
I do write in Python! The Python library I use for neural networks is called PyTorch, which makes most of the really hard stuff a lot easier. If you want to learn more, there are lots of great resources on Deep Reinforcement learning out there. My favourite however is a course by the channel Deep Lizard
@@aitango thx o heard about it. My only steps i took where with tensorflow but this is a good tip
1:30
Me listening:
"Nutboy numbnut won?"
Why is THAT a reward?
This puts TAS on a whole new level
I know this is a big ask so i wont expect anytime soon, but could you make an ai that can get the sword and shield in ocarina of time? Im mainly just curious how hard it would be, how big the resolution the screen would have to be, and as well as it handling complex 3d environment.
Love the content! Please use different b-roll footage however! It's the same in every video
Really glad you enjoyed it! Yeah I'll keep that in mind
So how did you actually measure the features for the reward? U said 0.001 reward per pixel to the right but what were the details in being able to figure out how many pixels mario has moved?
It would be really interesting making an ai beat the whole of mario wii on it's own, you could reward coin collection, x value (even though some levels require backtracking) and score
i thought that the nunchuk was already fully supported and shaking could be done with the input script editor?
I’d love to see an AI try to 100% this game. It would take forever but the fact that it would inevitably figure it out and optimize it is amazing
It would take ages, but I'd love to see it slowly progress through all the levels too, learning and optimizing!
At a very certain point, the AI will always loose its ablility to shoot fireball.
Maybe you can try to decrease the reward when AI loses its ablilty to shoot fireball.
Plus, you can let AI to start from a small mario, too.
How exactly are you interfacing with the controls? All the controls in Dolphin can be remapped precisely and extensively. Why is it not possible for you to remap "shake wiimote" to any key on the keyboard, and then just have the AI send that key? I do this all the time to remap motion controls to standard buttons on a traditional game controller.
Ai beating the entire game?
Could you share what library you used? Ive never done ML but would love to try it
All of my AIs are written in PyTorch!
@@aitango thank you!
6:16 How did Mario's fireballs kill the red koopa on the yellow mushroom platform ? He didn't even jump !
im curious, what would happen if you took this exactly as it is, and put it in a different stage?
Maybe you should try making a Dolphin plugin for better emulator integration, giving it more access to the controls and possibly even allowing it to look at memory addresses and shit
The current modded version of Dolphin I'm using already allows me to read and write memory addresses! The mod is really good, all it needs is support for other controllers!
It would be interesting to see it's performance on a level it has not seen before. In theory it recognises enemies and gaps and knows it needs to jump over them. So how would it fair against a totally new environment given it's current knowledge
This AI lacks a flaw that 99.99% of players have. Greed. The lust for the coin counter and the point score to go up. Collect everything and kill everything. It didn’t even bother to get over 100 on the flagpole!!
24 hours: "Well I've let it run this long, might as well let it run some more"
254 years: "Wellll I've let it run this long, might as well let it run some more"
Unquestionable logic I know
I'm curious to know what would have happened if the AI had videos of the areas, as it appears to be learning step by step. 1st it learned that platforms are moving, then it learned that enemies were moving...and in the process it learned how to not die and everything of course... but... I'm thinking it could have skipped some learning with more info given. Then again I know nothing about coding and how important images are... I'm just here wondering how would low quality videos have worked
You know... If our brain manages to process moving images I'm sure AI can too!
Imagine if you played multiplayer with your ai
I'm currently trying to see if this is possible, because its something I'd love to do. Especially doing online so I could have an AI playing in a full room
@@aitangome when I’m lonely. Lol but seriously could you imagine playing smash bros online with your ai co-op against real players? Would be dope.. but wrong lol
Can the AI react to individual situations or is it more like memorizing the level?
Probably both, but it's definitely learning to react to individual situations.
If you plopped it into a different level, it should be able to apply its skills and what it learned in the previous level, so it could continue to deal with common things like bottomless pits or enemies.
Why does the AI throw out so many fireballs?
How does it calculate fireballs into the mix?
Does it have a reward for kills?
Does it remember that it shot say 100 fire balls and got X rewards?
And what if it got X rewards but only shot 90 fire balls.
I'm curious about fireballs cause you can get through the level not even using 1 fireball.
You should also encourage it to get a high score/coins or get powerups
Yeah that would probably be interesting! I think I like seeing it go full speedrunner mode though :)
Make 2 variants 1 is high score 2 is spedrun mode
If you do something like this in the original mario, I wonder if it could learn warp pipes or even glitches at enough time
What is my purpose?
You play mario.
4:24 you say "our AI" but the subtitles go "REI", heh
I've seen that on a couple of videos, I never seem to spot it when looking over the subtitles :(
I'm assuming this AI is blind and cannot actually see the entities?
Fr, idk what that skeleton dragon is as your desktop background, but that's so based. Is it like, elder drake?
I just put it as my wallpaper because I thought it was cool, but a friend later told me its actually from world of warcraft, even though I've never played it haha
@@aitango That's so badass lol /gen
Also, could I add you on league? (I'm half asking as a bit, because it would be dope to play a match with a youtuber gldkhlkshg)
@@tylerclark3870 I was asking the uploader lol. I saw the shortcut on their desktop. Who're you? /gen /nm
Not sure who the imposter was, but I also happen to have stopped playing league after I hit diamond haha in a very very similar way. Oh that makes sense you saw it on my desktop
@@aitango Yo that's totally valid! I hit plat and I'm starting to lose the drive to play dibsksbsksbsken
But is the ai good only on that level or also decent on any other ones
Your gameplay at the beginning remind me of those “this game is so hard” mobile ads I’m not gonna lie
Oh no what have I done haha
@@aitango Hey, don’t worry about it bro. I was just messing with you, the rest of the video absolutely makes up for it. Watched your Mario Kart video just the other day too, you got some good stuff going here.
5:50 is what my mum said about me
😂
Does the AI’s skill more or less reset every level, or will it eventually get to the point where it could sight-read levels if left to its devices for long enough?
I wonder if the same ai could be put into other levels. Or if it's too specialized for this level in particular. Would also be cool to give a big reward for getting star coins. Another thing that could be interesting is just make the reward identical to the in game score tracker and see what happens. Might find new high score tech. Lol
Ai decided after 20-24 hours of training Ima speedrun this YOLOOOO
i feel you this level was one of the hardest ones
Nice video,
So this is a mid game level would be interesting what the learning curve is like when it’s learned as intended from the beginning
I wonder if in the future they could use AI to create human limit TASes
I mean, the thumbnail has a picture of NSMB (for the DS), the video is about NSMBW and the title just says Mario Bros .-.
No but seriously, nice video I have to say, as an eternal NSMBW fan I find it quite awesome to see an AI mario
I know its from DS, but I just thought it was a cool thumbnail haha. Glad you liked the video anyway though!
My only suggestion is, after you're done training an AI, allow us to see how well it does on a different stage.
Because, I'm sure many people are interested in that. +1
this man boutta be the first victim in the robot uprising
Even using AI for the captions! nice
Thanks, glad you like them!
now maybe add a reward for star coins, 1-ups and power-ups. maybe even coins. Then it can get a 100% player.
*enables spin-jump*
Pretty cool video, but if you're going to burn in subtitles, you should double-check them first. I assume you meant to say "count to 8" instead of "counter 8", and "our AI" instead of "REI".